The Storage First Pattern

Last Updated: July 23, 2020

The Storage First Pattern allows you to reliably capture data from incoming API requests without needing a Lambda function to parse, process, transform and save the data. Under the right circumstances, this pattern can reduce latency, save money, and minimize bugs.

Interactive Reference Architecture

Click on the components or numbered steps below to explore how this architecture works.

The Storage First pattern is useful when your application doesn’t require a lot of data transformation on incoming API requests. Rather than attaching API Gateway to a Lambda function that has to parse, process, transform, and save data, we can bypass the Lambda function by using a “service integration” that will send the data directly to an AWS service, like SQS. This reduces the latency of our API calls, saves money by removing the need to run a processing Lambda function, and makes our application more reliable because we are not introducing additional code.

In our example above, we’re using an SQS queue and then processing data off of that using a Lambda function subscription. There are plenty of other services that can be written to directly including DynamoDB, Kinesis, and EventBridge. To the best of my knowledge, Eric Johnson from AWS coined the term “Storage First” to indicate that we want to ensure that we save a user’s raw data before we attempt to run any processing on it. That way, if downstream services or processing fails, we always have a copy of the original request. He explains the process in his post¬†Building a serverless URL shortener app without AWS Lambda.

The incoming data can be transformed and verified using VTL templates, but the more complexity you introduce, the more likely you are to create issues with edge cases. This is an incredibly useful pattern for high velocity workloads like webhooks and clickstream data because it provides low latency and high reliability. Additional processing can be done asynchronously, allowing you to add resiliency to your application if downstream systems are unavailable.

Deploy this Pattern

Below are the basic configurations for deploying this pattern using different frameworks and platforms. Additional configuration for your environment will be necessary. The source files and additional examples are available in the GitHub repo.

  • TO IMPLEMENT

Want to Contribute?

All patterns and sample code are free to use and available under the MIT License. If you'd like to contribute to these patterns, please submit PRs to the GitHub repo.

» Explore more Serverless Reference Architectures and Patterns

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.