All Posts

The Top 15 Serverless Chats Podcast Episodes (so far)

As of this writing, there are now over 115 episodes of Serverless Chats! It’s been an incredible journey and I’ve learned so much about serverless from all these amazing guests (and I hope you have too). Recently, with the addition of Rebecca Marshburn as my co-host, we’ve added even more perspective to these conversations, expanding well beyond the technical and diving deeper into how serverless affects people, processes, and productivity.

If you’re new to serverless (and Serverless Chats), it can be hard to get started. Finding the right content and information can be overwhelming. If you’re a serverless veteran, finding interesting conversations that go beyond Hello World examples can also be challenging. My hope is that Serverless Chats provides a good mix for both the skeptical beginner and the experienced serverless developer. But even so, with 115 episodes, where do you start?

Continue Reading…

Big Serverless Chats Announcement!

When I started the podcast in 2019, I wanted to invite others into the great off-the-cuff, offline conversations I was having with members of the serverless community. The chats I’d share after an event over drinks, in the hallway between conference sessions, or while mingling at a meetup, were thought-provoking and interesting, and offered fresh ideas, perspectives, and sometimes, even solutions that I could take back to my work and projects. After more than 100 episodes of inviting listeners and guests to these conversations, I am now extending a different sort of invitation for one of our serverless community members.

I am extremely excited to announce that Rebecca Marshburn will be joining Serverless Chats as a co-host!

Continue Reading…

New Year, New Job, Same Serverless Mission

I took a new job as the GM of Serverless Cloud at Serverless, Inc. I’m really excited about what I’m working on there, and here’s why…

Some Background

When I first discovered AWS Lambda six years ago, I became fascinated with serverless, or rather, I became fascinated with the potential of serverless. Having spent an inordinate amount of time during the first 15 years of my career racking, configuring, patching, maintaining, and (poorly) scaling servers, the notion of abstracting all of that away seemed too good to be true. I had already been using AWS for five years at that point, so even though the cloud had eliminated the need for physical hardware, all the complexity around building high-scale, highly-available, highly-distributed applications was still there. For me, serverless was the way to solve that problem. So I went all in. 

Continue Reading…

Aurora Serverless v2: The Good, the Better, and the Possibly Amazing

Three years ago at re:Invent 2017, AWS announced the original Amazon Aurora Serverless preview. I spent quite a bit of time with it, and when it went GA 9 months later, I published my thoughts in a post titled Aurora Serverless: The Good, the Bad and the Scalable.

If you read the post, you’ll see that I was excited and optimistic, even though there were a lot of missing features. And after several months of more experiments, I finally moved some production workloads onto it, and had quite a bit of success. Over the last 18 months, we’ve seen some improvements to the product (including support for PostgreSQL and the Data API), but there were still loads of problems with the scale up/down speeds, failover time, and lack of Aurora provisioned cluster features.

That all changed with the introduction of Amazon Aurora Serverless v2. I finally got access to the preview and spent a few hours trying to break it. My first impression? This thing might just be a silver bullet!

I know that’s a bold statement. 😉 But even though I’ve only been using it for a few hours, I’ve also read through the (minimal) docs, reviewed the pricing, and talked to one of the PMs to understand it the best I could. There clearly must be some caveats, but from what I’ve seen, Aurora Serverless v2 is very, very promising. Let’s take a closer look!

Update December 9, 2020: I’ve updated the post with some more information after having watched the “Amazon Aurora Serverless v2: Instant scaling for demanding workloads” presentation by Murali Brahmadesam (Director of Engineering, Aurora Databases and Storage) and Chayan Biswas (Principle Product Manager, Amazon Aurora). The new images are courtesy of their presentation.

Continue Reading…

Jeremy’s Guide to a Very Serverless re:Invent 2020

It’s AWS re:Invent time, and once again, developers, architects, business leaders, and everyone in between are faced with the daunting task of selecting from thousands of hours of re:Invent content. As usual, I will be focusing most of my time on serverless, so I’ve combed through the massive session catalog and picked out the ones that look the most interesting to me. If you’re looking to focus on serverless during this re:Invent, perhaps you’ll find my suggestions useful.

My picks are also available on the Cloud Pegboard re:Invent Tool (Thanks, Ken). Select “Jeremy Daly” from the “AWS Hero Picks” dropdown and you’ll see all my selections with the options to add them to your wishlist and export them to your calendar. I have about 60 sessions on my list, which I’ve categorized below. But I realize that no human is likely going to be able to watch them all, so I’ve also made a list of Sessions you can’t miss!

There are sure to be plenty of announcements throughout the three weeks of re:Invent, so be sure to subscribe to the Off-by-none newsletter for weekly recaps.

Continue Reading…

LAMBDA – A Serverless Musical

Join the Serverless Revolution!

Learning a new paradigm can be really difficult, especially something as revolutionary (and different) as serverless. Thanks to a little inspiration from fellow AWS Serverless Hero, Forrest Brazeal, I created this Hamilton parody to help teach people what serverless is all about and why it’s such an amazing way to build applications. Hopefully it inspires you as well. Enjoy!

Continue Reading…

The Storage First Pattern

The Storage First Pattern allows you to reliably capture data from incoming API requests without needing a Lambda function to parse, process, transform and save the data. Under the right circumstances, this pattern can reduce latency, save money, and minimize bugs.

Interactive Reference Architecture

Click on the components or numbered steps below to explore how this architecture works.

The Storage First pattern is useful when your application doesn’t require a lot of data transformation on incoming API requests. Rather than attaching API Gateway to a Lambda function that has to parse, process, transform, and save data, we can bypass the Lambda function by using a “service integration” that will send the data directly to an AWS service, like SQS. This reduces the latency of our API calls, saves money by removing the need to run a processing Lambda function, and makes our application more reliable because we are not introducing additional code.

In our example above, we’re using an SQS queue and then processing data off of that using a Lambda function subscription. There are plenty of other services that can be written to directly including DynamoDB, Kinesis, and EventBridge. To the best of my knowledge, Eric Johnson from AWS coined the term “Storage First” to indicate that we want to ensure that we save a user’s raw data before we attempt to run any processing on it. That way, if downstream services or processing fails, we always have a copy of the original request. He explains the process in his post Building a serverless URL shortener app without AWS Lambda.

The incoming data can be transformed and verified using VTL templates, but the more complexity you introduce, the more likely you are to create issues with edge cases. This is an incredibly useful pattern for high velocity workloads like webhooks and clickstream data because it provides low latency and high reliability. Additional processing can be done asynchronously, allowing you to add resiliency to your application if downstream systems are unavailable.

Deploy this Pattern

Below are the basic configurations for deploying this pattern using different frameworks and platforms. Additional configuration for your environment will be necessary. The source files and additional examples are available in the GitHub repo.


The Circuit Breaker

The Circuit Breaker pattern keeps track of the number of failed (or slow) API calls by using a cache to share the status across multiple Lambda functions. This allows you to perform load shedding when downstream services become unavailable.

Interactive Reference Architecture

Click on the components or numbered steps below to explore how this architecture works.

The Circuit Breaker pattern keeps track of the number of failed (or slow) API calls by using a cache to share the status across multiple Lambda functions. In this example, we’re using a DynamoDB table so that we can avoid using a VPC. If you were in a VPC already, ElastiCache would be a good alternative.

Here’s how it works. When the number of failures reaches a certain threshold, we “open” the circuit and send errors back to the calling client immediately without even trying to call the API. After a short period of time, we “half open” the circuit, sending just a few requests through to see if the API is finally responding correctly. All other requests receive an error. If the sample requests are successful, we “close” the circuit and start letting all traffic through. However, if some or all of those requests fail, the circuit stays “open”, and the process repeats with some algorithm for increasing the timeout between “half open” retry attempts.

This is an incredibly powerful (and cost saving) pattern for any type of synchronous request to an API or downstream system. You are accumulating charges whenever a Lambda function is running and waiting for another task to complete. Allowing your systems to self-identify issues like this, provide incremental backoff, and then self-heal when the service comes back online, adds a tremendous amount of resiliency to your applications.

Deploy this Pattern

Below are the basic configurations for deploying this pattern using different frameworks and platforms. Additional configuration for your environment will be necessary. The source files and additional examples are available in the GitHub repo.

🚀 Project Update:

Data API Client: v1.1 Released

Bug fixes and feature updates including support for native JavaScript dates (thanks @cklam2), support for non-specific database queries, and deprecation of the HTTP keepAlive workaround in favor of the native SDK support. Read More...

Announcing the Serverless Reference Architectures Project

Serverless gives us the power to focus on delivering value to our customers without worrying about the maintenance and operations of the underlying compute resources. Cloud providers (like AWS), also give us a huge number of managed services that we can stitch together to create incredibly powerful, and massively scalable serverless microservices.

Almost 2 years ago now, I wrote a post on Serverless Microservice Patterns for AWS that became a popular reference for newbies and serverless veterans alike. The capabilities of serverless have changed dramatically since then, opening up a ton of new patterns and possibilities. Today I’m announcing the Serverless Reference Architectures Project. This project is intended to capture, share, explore, and debate the patterns and practices being used in serverless production applications today.

Continue Reading…