Product Guy, Serverless Advocate & Startup Veteran

My name is Jeremy Daly. I appreciate the visit. 👍 I’ve been managing the development of complex web and mobile applications for businesses across the globe for over 20 years. I’m currently the Chief Technology Officer at AlertMe, but I always keep myself busy with several side projects and consulting clients.

I write a lot about serverless and I like to share thoughts and stories about programmingproduct managemententrepreneurship and productivity. Sometimes I’ll write reviews or have random thoughts that I need to get out of my head. I also like to post a how-to article every once in a while. Feel free to jump in to one of the categories above, view all my posts, or view my latest posts below.

If you’d like to get updates, please subscribe to my weekly newsletter, or follow me on TwitterGithub and Facebook.

My Latest Posts:

🚀 Project Update:

Serverless MySQL: v1.1.0 Released

Serverless MySQL v1.1.0 adds additional transaction support capabilities to allow users to retrieve interim query results for use with future queries. This is useful for getting the insertId from previous queries when performing transactions. Read More...

Serverless Security: Locking Down Your Apps with FunctionShield

I’ve written quite extensively about serverless security, and while you don’t need to be an expert on the matter, there are a number of common sense principles that every developer should know. Serverless infrastructures (specifically FaaS and managed services) certainly benefit from an increased security posture given that the cloud provider is handling things like software patching, network security, and to some extent, even DDoS mitigation. But at the end of the day, your application is only as secure as its weakest link, and with serverless, that pretty much always comes down to application layer security.

In this post we’re going to look at ways to mitigate some of these application layer security issues by using some simple strategies as well as a free tool called FunctionShield.

Audio Version:

Continue Reading…

Managing MySQL at Serverless Scale

“What? You can’t use MySQL with serverless functions, you’ll just exhaust all the connections as soon as it starts to scale! And what about zombie connections? Lambda doesn’t clean those up for you, meaning you’ll potentially have hundreds of sleeping threads blocking new connections and throwing errors. It can’t be done!”  ~ Naysayer

I really like DynamoDB and BigTable (even Cosmos DB is pretty cool), and for most of my serverless applications, they would be my first choice as a datastore. But I still have a love for relational databases, especially MySQL. It had always been my goto choice, perfect for building normalized data structures, enforcing declarative constants, providing referential integrity, and enabling ACID-compliant transactions. Plus the elegance of SQL (structured query language) makes organizing, retrieving and updating your data drop dead simple.

But now we have SERVERLESS. And Serverless functions (like AWS Lambda, Google Cloud Functions, and Azure Functions) scale almost infinitely by creating separate instances for each concurrent user. This is a MAJOR PROBLEM for RDBS solutions like MySQL, because available connections can be quickly maxed out by concurrent functions competing for access. Reusing database connections doesn’t help, and even the release of Aurora Serverless doesn’t solve the max_connections problem. Sure there are some tricks we can use to mitigate the problem, but ultimately, using MySQL with serverless is a massive headache.

Well, maybe not anymore. 😀 I’ve been dealing with MySQL scaling issues and serverless functions for years now, and I’ve finally incorporated all of my learning into a simple, easy to use NPM module that (I hope) will solve your Serverless MySQL problems.

Continue Reading…

Jeremy goes to AWS re:Invent 2018

It’s official! I’m going to AWS re:Invent 2018. 🙌

My goal from this trip is to learn, learn, learn… and then share, share, share.   There are over 30 sessions that talk about serverless, plus 40,000 other people there to meet and learn from! I’m so excited. 🙃

I know that many of you will be there, but for those of you who can’t be, I’ll do my best to share insights, tips, how-tos, best practices and more. I’ll even have a drink for you if you’d like 🍺 (no arm twisting necessary)!

Continue Reading…

Serverless Microservice Patterns for AWS

I’m a huge fan of building microservices with serverless systems. Serverless gives us the power to focus on just the code and our data without worrying about the maintenance and configuration of the underlying compute resources. Cloud providers (like AWS), also give us a huge number of managed services that we can stitch together to create incredibly powerful, and massively scalable serverless microservices.

I’ve read a lot of posts that mention serverless microservices, but they often don’t go into much detail. I feel like that can leave people confused and make it harder for them to implement their own solutions. Since I work with serverless microservices all the time, I figured I’d compile a list of design patterns and how to implement them in AWS. I came up with 19 of them, though I’m sure there are plenty more.

In this post we’ll look at all 19 in detail so that you can use them as templates to start designing your own serverless microservices.

Audio Version:

Continue Reading…

🚀 Project Update:

Lambda API: v0.8 Released

Lambda v0.8 is finally here and was well worth the wait! New features include allowing middleware to accept multiple handlers, new convenience methods for cache control and signing S3 URLs, and async/await support for the main function handler. And best of all, new LOGGING and SAMPLING support for you to add more observability into your APIs and web applications. Read More...

Aurora Serverless: The Good, the Bad and the Scalable

Amazon announced the General Availability of Aurora Serverless on August 9, 2018. I have been playing around with the preview of Aurora Serverless for a few months, and I must say that overall, I’m very impressed. There are A LOT of limitations with this first release, but I believe that Amazon will do what Amazon does best, and keep iterating until this thing is rock solid.

The announcement gives a great overview and the official User Guide is chock full of interesting and useful information, so I definitely suggest giving those a read. In this post, I want to dive a little bit deeper and discuss the pros and cons of Aurora Serverless. I also want to dig into some of the technical details, pricing comparisons, and look more closely at the limitations.

Audio Version

Continue Reading…

A Tale of Two Teams

Audio Version:

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness… ~ A Tale of Two Cities by Charles Dickens

There is a revolution happening in the tech world. An emerging paradigm that’s letting development teams focus on business value instead of technical orchestration. It is helping teams create and iterate faster, without worrying about the limits or configurations of an underlying infrastructure. It is enabling the emergence of new tools and services that foster greater developer freedom. Freedom to experiment. Freedom to do more with less. Freedom to immediately create value by publishing their work without the traditional barriers created by operational limits.

Continue Reading…

How To: Add Test Coverage to your Serverless Applications

Writing serverless functions brings developers closer and closer to the stack that runs their code. While this gives them a tremendous amount of freedom, it also adds additional responsibility. Serverless applications require developers to think more about security and optimizations, as well as perform other tasks that were traditionally assigned to operations teams. And of course, code quality and proper testing continue to be at the top of the list for production-level applications. In this post, we’ll look at how to add test coverage to our Node.js applications and how we can apply it to our Serverless framework projects. ⚡️

Continue Reading…

25 Serverless Peeps You Need To Follow

In my never ending quest to consume all things serverless, I often find myself scouring the Interwebs for new and interesting serverless articles, blog posts, videos, and podcasts. There are more and more people doing fascinating things with serverless every day, so finding content is becoming easier and easier. However, this increase in content comes with an increase in noise as well. Cutting through that noise isn’t always easy. 🙉

Great content with valuable insights

I personally love reading articles that introduce new use cases or optimizations for serverless. Stories about companies using serverless in production and how their architectures are set up are also extremely interesting.. I’ve been working in the serverless space for several years now, and have come across a number of people who produce and/or share really great content. I’ve put together a list of 25 people that I follow and enjoy their content regularly. Hopefully these people will help you learn to love serverless as much as I do. ❤️⚡️

Continue Reading…

How To: Tag Your Lambda Functions for Smarter Serverless Applications

As our serverless applications start to grow in complexity and scope, we often find ourselves publishing dozens if not hundreds of functions to handle our expanding workloads. It’s no secret that serverless development workflows have been a challenge for a lot of organizations. Some best practices are starting to emerge, but many development teams are simply mixing their existing workflows with frameworks like Serverless and AWS SAM to build, test and deploy their serverless applications.

Beyond workflows, another challenge serverless developers encounter as their applications expand, is simply trying to keep all of their functions organized. You may have several functions and resources as part of a microservice contained in their own git repo. Or you might simply put all your functions in a single repository for better common library sharing. Regardless of how code is organized locally, much of that is lost when all your functions end up in a big long list in the AWS Lambda console. In this post we’ll look at how we can use AWS’s resource tagging as a way to apply structure to our deployed functions. This not only give us more insight into our applications, but can be used to apply Cost-Allocation Tags to our billing reports as well. 👍

Continue Reading…

Thinking Serverless (Big and Small)

I’ve been reading and writing a lot of about serverless lately, and one of the things I realized, is that most articles talk about how SCALABLE serverless architectures are. This, of course, is one of the major benefits of using serverless to build your applications. The ability to scale to thousands of concurrent requests per second without needing to manage your own servers, is simply amazing. 🙌

However, not needing to manage any servers has other benefits beyond the capabilities to achieve web scale. Having on-demand compute space also make serverless the perfect candidate for smaller workloads. In this post, let’s discuss how we can utilize serverless to handle our “less than unicorn 🦄” services and the benefits this can bring.

Continue Reading…

Lambda Warmer: Optimize AWS Lambda Function Cold Starts

At a recent AWS Startup Day event in Boston, MA, Chris Munns, the Senior Developer Advocate for Serverless at AWS, discussed Lambda cold starts and how to mitigate them. According to Chris (although he acknowledge that it is a “hack”) using the CloudWatch Events “ping” method is really the only way to do it right now. He gave a number of really good tips to pre-warm your functions “correctly”:

  • Don’t ping more often than every 5 minutes
  • Invoke the function directly (i.e. don’t use API Gateway to invoke it)
  • Pass in a test payload that can be identified as such
  • Create handler logic that replies accordingly without running the whole function

Continue Reading…

15 Key Takeaways from the Serverless Talk at AWS Startup Day

I love learning about the capabilities of AWS Lambda functions, and typically consume any article or piece of documentation I come across on the subject. When I heard that Chris Munns, Senior Developer Advocate for Serverless at AWS, was going to be speaking at AWS Startup Day in Boston, I was excited. I was able to attend his talk, The Best Practices and Hard Lessons Learned of Serverless Applications, and it was well worth it.

Chris said during his talk that all of the information he presented is on the AWS Serverless site. However, there is A LOT of information out there, so it was nice to have him consolidate it down for us into a 45 minute talk. There was some really insightful information shared and lots of great questions. I was aware of many of the topics discussed, but there were several clarifications and explanations (especially around the inner workings of Lambda) that were really helpful. 👍

Continue Reading…

Mixing VPC and Non-VPC Lambda Functions for Higher Performing Microservices

I came across a post the in the Serverless forums that asked how to disable the VPC for a single function within a Serverless project. This got me thinking about how other people structure their serverless microservices, so I wanted to throw out some ideas. I often mix my Lambda functions between VPC and non-VPC depending on their use and data requirements. In this post, I’ll outline some ways you can structure your Lambda microservices to isolate services, make execution faster, and maybe even save you some money. ⚡️💰

Continue Reading…

5 Reasons Why Your Serverless Application Might Be A Security Risk

There has been a lot of buzz lately about serverless security. People are certainly talking about it more and sharing great articles on the topic, but many serverless developers (especially new ones) are still making the same critical mistakes. Every time a serverless function is deployed, its unique security challenges need to be addressed. Every time. I’ve researched and written extensively about serverless security (see Securing Serverless: A Newbie’s Guide). I’ve read countless articles on the subject. And while there is no shortage of information available, let’s be honest: developers are busy building applications, not pouring through hundreds of articles.

I know, it sounds boring, but I would encourage you to do your research on serverless security. Serverless applications are different than traditional, server-hosted applications. Much of the security responsibility falls on the developer, and not following best practices opens you (or your company) up to an attack. But I know you’re busy. I totally get it. So rather than forcing you to read a bunch of long articles 😴 or watch a plethora of videos 🙈, I’ve whittled it all down to the five biggest serverless security risks for you. Sure, there are a lot of other things to consider, but IMO, these are the most important ones. Nothing here hasn’t been said before. But If you do nothing more than follow these principles, your serverless applications will be much more secure. 🔒

Continue Reading…

Serverless Consumers with Lambda and SQS Triggers

On Wednesday, June 27, 2018, Amazon Web Services released SQS triggers for Lambda functions. Those of you who have been building serverless applications with AWS Lambda probably know how big of a deal this is. Until now, the AWS Simple Queue Service (SQS) was generally a pain to deal with for serverless applications. Communicating with SQS is simple and straightforward, but there was no way to automatically consume messages without implementing a series of hacks. In general, these hacks “worked” and were fairly manageable. However, as your services became more complex, dealing with concurrency and managing fan out made your applications brittle and error prone. SQS triggers solve all of these problems. 👊

Audio Version

Continue Reading…

Event Injection: A New Serverless Attack Vector

As more and more developers and companies adopt serverless architecture, the likelihood of hackers exploiting these applications increases dramatically. The shared security model of cloud providers extends much further with serverless offerings, but application security is still the developer’s responsibility. There has been a lot of hype about #NoOPS with serverless environments 🤥, which is simply not true 😡. Many traditional applications are frontended with WAFs (web application firewalls), RASPs (runtime application self-protection), EPPs (endpoint protection platforms) and WSGs (web security gateways) that inspect incoming and outgoing traffic. These extra layers of protection can save developers from themselves when making common programming mistakes that would otherwise leave their applications vulnerable. With serverless, these all go away. 😳

Continue Reading…

10 Things You Need To Know When Building Serverless Applications

I am a HUGE fan of serverless architectures. This new type of compute not only opens up more possibilities for developers, but can support highly-scalable, complex applications for a fraction of the cost compared to provisioning virtual servers. My first question when planning a new application is always, “Can I build this with serverless?” Spoiler alert, the answer is almost always YES!

I’ve been building serverless applications since the release of AWS Lambda in 2015, so answering the question above is pretty easy for me. However, a lot of people I talk to who are new to serverless often have many questions (and misconceptions). I want you to be successful, so below I’ve create a list of 10 things you need to know when building a serverless application. These are things I wish I knew when I started, so hopefully they’ll help you get up to speed a faster and start building some amazing applications.

Continue Reading…

🚀 Project Update:

Lambda API: v0.7 Released

v0.7 adds new features to control middleware execution based on path, plus additional parsing of the AWS Lambda context object. ESLint and coverage reports using Istanbul and Coveralls were also added to ensure code quality and adequate test coverage. Read More...

How To: Optimize the Serverless Optimizer Plugin

I’m sure you’re already well aware of how awesome the ⚡ Serverless Framework is for managing and deploying your serverless applications. And you’re probably aware that there are several great plugins available that make Serverless even better. But did you know that there was a plugin to optimize your functions and reduce the size of your deployment packages? Or are you already using this plugin to optimize your functions, but hate how it takes too long to optimize locally run functions? In this post I’ll share some quick tips to help you optimize your Serverless Optimizer experience.

Continue Reading…

Transducers: Supercharge your functional JavaScript

This is the first in a series of posts on functional programming in JavaScript. My goal is to make these ideas more accessible to all levels of programmers. Feedback about style, content, etc., would all be greatly appreciated.

One thing that perplexed me early on in my functional programming days was the concept of transducers. I spent a lot of time Googling and found some great articles that went deep into the theory and the underlying mechanics. However, the practical use of them still seemed a bit out of reach. In this post I’ll attempt to explain transducers in a more understandable way and hopefully give you the confidence to use them in your functional JavaScript. While this article attempts to make transducers more accessible, you will need to have some basic knowledge of functional programming in JavaScript. Specifically, you should know about function composition and iterator functions like .map(), .filter(), and most importantly, .reduce(). If you are unfamiliar with these concepts, go get a grasp on them first.

Continue Reading…

🚀 Project Update:

Lambda API: v0.6 Released

v0.6 is all about making the serverless developer's life easier! New support for both callback-style and async-await in route functions and middleware, new HTTP method routing features, and route debugging tools. Plus Etag support and automatic authorization parsing. Read More...

Solving the Cold Start Problem

Dear AWS Lambda Team,

I have a serious problem: I love AWS Lambda! In fact, I love it so much that I’ve pretty much gone all in on this whole #serverless thing. I use Lambda for almost everything now. I use it to build backend data processing pipelines, distribute long running tasks, and respond to API requests. Heck, I even built an Alexa app just for fun. I found myself building so many RESTful APIs using Lambda and API Gateway that I went ahead and created the open source Lambda API web framework to allow users to more efficiently route and respond to API Gateway requests.

Serverless technologies, like Lambda, have revolutionized how developers think about building applications. Abstracting away the underlying compute layer and replacing it with on-demand, near-infinitely scalable function containers is brilliant. As we would say out here in Boston, “you guys are wicked smaht.” But I think you missed something very important. In your efforts to conform to the “pay only for the compute time you consume” promise of serverless, you inadvertently handicapped the service. My biggest complaint, and the number one objection that I hear from most of the “serverless-is-not-ready-for-primetime” naysayers, are Cold Starts.

Continue Reading…

How To: Manage Serverless Environment Variables Per Stage

I often find myself creating four separate stages for each ⚡ Serverless Framework project I work on: dev, staging, prod, and local. Obviously the first three are meant to be deployed to the cloud, but the last one, local, is meant to run and test interactions with local resources. It’s also great to have an offline version (like when you’re on a plane ✈ or have terrible wifi somewhere). Plus, development is much faster because you’re not waiting for round trips to the server. 😉

A really great feature of Serverless is the ability to configure ENVIRONMENT variables in the serverless.yml file. This lets us store important global information like database names, service endpoints and more. We can even reference passwords securely using AWS’s Service Manager Parameter Store and decode encrypted secrets on deployment, keeping them safe from developers and source repositories alike. 😬 Just reference the variable with ${ssm:/myapp/my-secure-value~true} in your configuration file.

Continue Reading…

🚀 Project Update:

Lambda API: v0.5 Released

v0.5 takes advantage of AWS Lambda's recently released support for Node v8.10 and has removed its Bluebird promise dependency in favor of async/await. Lambda API is now faster and adds built-in CORS support, additional wildcard features, new HTTP header management methods and more. Read More...

How To: Stub “.promise()” in AWS-SDK Node.js

Since AWS released support for Node v8.10 in Lambda, I was able to refactor Lambda API to use async/await instead of Bluebird promises. The code is not only much cleaner now, but I was able to remove a lot of unnecessary overhead as well. As part of the refactoring, I decided to use AWS-SDK’s native promise implementation by appending .promise() to the end of an S3 getObject call. This works perfectly in production and the code is super compact and simple:

The issue came with stubbing the call using Sinon.js. With the old promise method, I was using promisifyAll() to wrap new AWS.S3() and then stubbing the getObjectAsync method. If you’re not familiar with stubbing AWS services, read my post: How To: Stub AWS Services in Lambda Functions using Serverless, Sinon.JS and Promises.

Continue Reading…

How To: Manage RDS Connections from AWS Lambda Serverless Functions

Someone asked a great question on my How To: Reuse Database Connections in AWS Lambda post about how to end the unused connections left over by expired Lambda functions:

I’m playing around with AWS lambda and connections to an RDS database and am finding that for the containers that are not reused the connection remains. I found before that sometimes the connections would just die eventually. I was wondering, is there some way to manage and/or end the connections without needing to wait for them to end on their own? The main issue I’m worried about is that these unused connections would remain for an excessive amount of time and prevent new connections that will actually be used from being made due to the limit on the number of connections.

🧟‍♂️ Zombie RDS connections leftover on container expiration can become a problem when you start to reach a high number of concurrent Lambda executions. My guess is that this is why AWS is launching Aurora Serverless, to deal with relational databases at scale. At the time of this writing it is still in preview mode.

Update September 2, 2018: I wrote an NPM module that manages MySQL connections for you in serverless environments. Check it out here.

Update August 9, 2018: Aurora Serverless is now Generally Available!

Overall, I’ve found that Lambda is pretty good about closing database connections when the container expires, but even if it does it reliably, it still doesn’t solve the MAX CONNECTIONS problem. Here are several strategies that I’ve used to deal with this issue.

Continue Reading…

Is Code Really Self-Documenting?

In my 20+ years of programming, I’ve encountered a near endless amount of opinions on everything from coding styles to programming paradigms to the great whitespace debate. Obviously, I have strong opinions on a number of these. But for me, the one that bothers me the most is this notion that “code is self-documenting.” 😾

I know what you’re probably thinking: “of course not all code is self-documenting, only well-written code is.” I don’t entirely disagree. I can generally look at someone else’s code and understand exactly WHAT it is doing. However, often it’s not obvious WHY they did it that way, or even why they did it in the first place. In my opinion, the programmer’s intent (the WHY) is just as important as the HOW when it comes to properly documenting software.

So whether you agree with me or not, let’s explore how to better document our software by writing cleaner code, following some general commenting etiquette, and commenting more effectively to make you and your team more productive. 👍

Continue Reading…

Securing Serverless: A Newbie’s Guide

So you’ve decided to build a serverless application. That’s awesome! May I be the first to welcome you to the future. 🤖 I bet you’ve done a lot of research. You’ve probably even deployed a few test functions to AWS Lambda or Google Cloud Functions and you’re ready to actually build something useful. You probably still have a bunch of unanswered questions, and that’s cool. We can still build some really great applications even if we only know the basics. However, when we start working with new things we typically make a bunch of dumb mistakes. While some are relatively innocuous, security mistakes can cause some serious damage.

I’ve been working with serverless applications since AWS launched Lambda in early 2015. Over the last few years I’ve developed many serverless applications covering a wide range of use cases. The most important thing I’ve learned: SECURE YOUR FUNCTIONS! I can tell you from personal experience, getting burned by an attack is no bueno. I’d hate to see it happen to you. 😢

To make sure it doesn’t happen to you, I’ve put together a list of 🔒Serverless Security Best Practices. This is not a comprehensive list, but it covers the things you ABSOLUTELY must do. I also give you some more things to think about as you continue on your serverless journey. 🚀

Continue Reading…

How To: Build a Serverless API with Serverless, AWS Lambda and Lambda API

AWS Lambda and AWS API Gateway have made creating serverless APIs extremely easy. Developers can simply create Lambda functions, configure an API Gateway, and start responding to RESTful endpoint calls. While this all seems pretty straightforward on the surface, there are plenty of pitfalls that can make working with these services frustrating.

There are, for example, lots of confusing and conflicting configurations in API Gateway.  Managing deployments and resources can be tricky, especially when publishing to multiple stages (e.g. dev, staging, prod, etc.). Even structuring your application code and dependencies can be difficult to wrap your head around when working with multiple functions.

In this post I’m going to show you how to setup and deploy a serverless API using the Serverless framework and Lambda API, a lightweight web framework for your serverless applications using AWS Lambda and API Gateway. We’ll create some sample routes, handle CORS, and discuss managing authentication. Let’s get started.

Continue Reading…

Off-by-none: Issue #3

They say that breaking up (an app) is hard to do

Hello, All! Welcome to Issue #3 of Off-by-none. I’m so happy that you’re here to talk about #serverless. ⚡️

Last week we talked about serverless security and how important it is to educate developers, especially now that they are much closer to the execution stack and typically lack the necessary security training. This week, I want to start a discussion about what to do as more and more complexity begins creeping into our serverless applications. This is a long discussion, so we’ll just skim the surface today, but over the next few weeks, I’d like to expand on this topic and see where it leads us.

Let’s dive in! 🏊‍♂️

When your monolith starts to get out of hand… 🔥

A very popular use case for serverless is to create microservices (or even nanoservices). And before you respond with the obvious, “well of course, who would create monoliths with serverless?” Let me just say, a lot of people create monoliths in serverless. While there is much discouragement about doing so, in some narrow use cases, it can make sense. However, for most serverless workloads, splitting up your services into small, composable units, is a much better choice.

I’ve written extensively about serverless microservice patterns for AWS, and I’m personally a huge fan of microservice architectures in general. But microservice architectures are complex, and require coordination and communication between services to operate efficiently at scale. We strive for low coupling and high cohesion in our services, constantly wrestling with our choices of synchronous versus asynchronous communication, eventual consistency, orchestration versus choreography, service boundaries and much more.

Serverless introduces us to new ways to interconnect these microservices. We can use the more traditional pub/sub or message bus approach with things like SNS. Plus we can still queue information with SQS and use competing consumers to process our messages. But serverless is also an event-driven architecture, which goes beyond the service-oriented architectures (SOAs) of the past. Our services can be triggered by streamed events from Kinesis or DynamoDB, react to things like file uploads, configuration changes, alarms and much much more. They can drive the caching and routing behavior of CDNs with dynamic and distributed worker functions. They can coordinate massively distributed jobs at almost any scale and still aggregate the results using state machines, durable functions, or other available patterns.

This is a new way to think about and design our applications, and it isn’t easy. My most recent project has left me scratching my head a number of times about the best way to choreograph the communication between services. I’m even rereading Sam Newman’s Building Microservices to try and make sense of some of these paradigm shifts while still applying the best practices that more traditional microservice architectures have taught us. There is a lot to take in, so don’t feel bad if some of this stuff perplexes you. 😕

There are several good articles and talks that I link to in today’s newsletter, so explore these to see how others are dealing with these services at scale. I’ve been working on a few pieces that attempt to boil all this down for you, but they’re not quite ready for primetime. I hope this gets you thinking about this topic, and please feel free to share with me other resources you’ve found on the subject.

When you want to learn from the experts… 👨‍🔬

ServerlessDays Portland was yesterday, and it looks like it was another successful event. (BTW, I’m on the planning committee for ServerlessDays Boston, so follow us on Twitter and stay tuned for announcements).

Obviously Chris Munns was there, and gave a talk about “Serverless and DevOps?” and was kind enough to put the slides online. I’m hoping that the event was filmed and we’ll see some of the videos pop up online soon. It’s always great to see talks by Charity Majors, Nitzan ShapiraErica WindischKelsey Hightower and others.

Speaking of videos of serverless talks, A Cloud Guru just dropped the mother load! A ton of ServerlessConf 2018 talks are now available for your viewing pleasure! Grab a pen and a notebook, some popcorn, and get ready to binge watch hours of brilliant serverless talks. 🍿😳📺

Also, don’t forget that there is an AWS webinar on September 19, 2018 at 9 AM PT about Serverless Application Debugging and Delivery. I think it will be interesting, so I’m planning on watching.

When you have a few extra minutes to read on the train… 🚂

Chris Armstrong over at Gorilla Stack wrote a great piece called Splitting your Serverless Framework API on AWS. The CloudFormation 200 resources limit has bitten many of us when we start to build and launch complex serverless applications. Chris outlines some strategies to break up your Serverless apps by using a “base stack” to create your API Gateway and then integrate child stacks as your app grows. 👍

If there’s one company that has gone all in on serverless, it’s Nordstrom. Last week they launched a new blog on Medium (with a really creative title 😉) called Tech at Nordstrom. In their first real post, Google Cloud Next 2018: Is this the year that Google embraces serverless?, Rob Gruhl talks about how Nordstrom invokes over 200 million serverless functions every day and outlines a great wishlist for Google Cloud. ⛅️

Finally, Raees Bhatti teaches us how to Use Cloudflare Workers + Serverless Framework to add reliability and uptime to your FaaS. This is a great follow up to Troy Hunt’s article we mentioned last week that allowed him to support 141M monthly queries of 517M records for just 2.6 cents per day. 💸

What happens when Ory Segal bumps into Simon Wardley at SFO? ✈️

💥 Pure magic. Ory and Simon spend just over 30 minutes Musing on Serverless and Application Security in this excellent “informal” interview. Simon Wardley is the creator (and namesake) of Wardley Maps, which helps you anticipate market and ecosystem developments so you know where to go and why. He is a huge proponent of serverless and makes a number of compelling arguments as to why it will be the next big thing. There is both an audio version and transcript of this interview. It is an excellent discussion that is packed with valuable information and definitely worth a listen/read.

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Andrea Passwater (@andreapasswater). Andrea is the Lead Content Strategist over at Serverless, Inc., creators of the amazing Serverless Framework. While Andrea describes herself as someone who “doesn’t code“, she has done a pretty good job stitching together some serverless applications to help automate her marketing tasks (see her Serverlessconf 2018 Lightning Talk). However, I didn’t choose Andrea just because she showed us how easy it is for “non-developers” to get started with serverless, but because of the valuable content she helps to create and disseminate.

“You probably aren’t just changing the way developers work. You could eventually start a mass automation movement that changes the way everyone works.” ~ Andrea Passwater

I’ve met thousands of developers over my career, and there are a select few that can present technical information in a way that can inspire others to change their mindset and explore new things. Andrea is not only coordinating this type of content (through the Serverless blog, [cron]incle newsletter, and Twitter), but is also doing a great job of distilling down technical content for the masses. Anything that helps lower the barrier to entry for serverless is a win in my book.

On a more personal note, I have two daughters in middle school, one who loves science, and the other who loves math. In a male-dominated tech industry, women like Andrea (who also started the #WomenWhoServerless meet ups) serve as role models to girls and young women all around the world who want to get into technology. You need to know that you’re making a positive impact, and that you have the undying gratitude of dads like me.  🙌

Final Thoughts 🤔

As I mentioned in the beginning, I just wanted to start this conversation about managing complexity in our serverless applications. As we begin to reach scale, there are a lot of things to start thinking about, especially when it comes to coordination of microservices. We’ll explore this in more detail in the weeks to come, but it never hurts to get a head start.

Also, it’s important to remember as we start to talk about serverless scale, that serverless has other use cases that apply to much smaller tasks as well. Think about the little things that Andrea Passwater automated to make her job a bit easier. Thinking Serverless (Big and Small) goes into some more detail about some of the smaller use cases as well.

I hope you enjoyed this issue of Off-by-none. Feedback is always greatly appreciated and helps me make this newsletter better each week. Contact me via Twitter, LinkedIn, Facebook, or email and let me know your thoughts, angry rants, or even how you’d like to contribute to Off-by-none.

Go build some great serverless apps. Hope to see you all next week!


Off-by-none: Issue #2

Serverless Security, yeah, that’s all I want from you

Hey, Everyone! Welcome to Issue #2 of Off-by-none. Thanks so much for being here. 🙌

Last week we talked about the goal of Off-by-none and introduced a new way to think about managing MySQL connections at serverless scale. This week, I want to focus on security, and how we can use cloud provider tools, common sense strategies, and other vendor solutions to lock down our serverless applications.

Here we go! 🚀

When you install a compromised third-party dependency… 🤦🏻‍♂️

Not too long ago, 54% of the NPM ecosystem was compromised by using a series of brute force attacks, leaked datasets, and fuzzing passwords. NPM has continued to add more protections to mitigate these types of risks, but nothing is foolproof. Sometimes, the developers using them are to blame for installing malware with similar names to popular packages. As with the NPM hack above, sometimes the compromised dependency could be way down the dependency chain. This could leave your application, user data, credentials, and even access keys, extremely vulnerable.

This isn’t meant to scare you away from serverless. In fact, the issues above aren’t even specific to serverless. However, the built-in scalability of FaaS is what can exacerbate an attack by spinning up thousands of functions to do a hacker’s bidding. So, when building serverless applications, we want to do whatever we can to minimize these types of risks.

There are some serverless security basics that developers should really know. These will go a long way to mitigating a number of popular attack vectors. Serverless also requires us to think about event injection since many of our functions will be processing events without protection from WAFs or WSGs. But beyond the basics, we still run the risk of compromised third-party packages leaking sensitive information when we run our functions.

I had conversation with Ory Segal, the CTO at PureSec about this and gave their free FunctionShield product a try. I wrote a posts called Serverless Security: Locking Down Your Apps with FunctionShield which tests it out and goes through the pros and cons. Perhaps some day the cloud providers will add additional security controls, but until then, I really think this is worth a look.

When you feel like geeking out with serverless… 🤓

Troy Hunt, creator of, wrote an excellent post about how it only costs him 2.6 cents per day to support 141M monthly queries of 517M records. Yeah, I didn’t quite believe it at first either, but read the post and see for yourself: Serverless to the Max: Doing Big Things for Small Dollars with Cloudflare Workers and Azure Functions. 💰

Sam Bashton over at threw some shade at Ben Kehoe with his How We Massively Reduced Our AWS Lambda Bill With Go post. He argues that single thread, single task functions are NOT always the best choice for serverless. Several people (including myself) weighed in on the Twitter debate. 😬

The New Stack had a chat with Nate Taggart from Stackery and discussed Serverless Analytics: Metrics, Collection and Visibility. Some interesting bits in there about the challenges with tracing, logging and observability in serverless applications. 🔭

And speaking of tracing, AWS X-Ray Now Supports Amazon API Gateway and New Sampling Rules API. This was a huge missing feature of X-Ray. I’ve haven’t tried it yet, but it is on my list of things to do. 📝

When Jeff Barr (aka The Wizard of AWS) decides to share the love… ❤️

Jeff Barr not only gave a ringing endorsement of Marcia Villalba’s new Getting Started with AWS SAM video series, but also tweeted a link (with one of his classic screenshots) to Lambda API. If you’re not familiar with Lambda API, it’s a lightweight web framework, built specifically for serverless applications. It’s similar to to ExpressJS and Fastify, just ridiculously smaller with ZERO dependencies.

I think we all know by now that minimizing dependencies helps to secure and speed up our serverless apps. But size matters too. The other day I had put together this infographic that shows the comparison between the node_modules directories of different web frameworks and their total package sizes:

These libraries are great for servers, but Lambda API is so small, it adds almost nothing to your cold start times. There is an important lesson here, especially when you follow the “single purpose” best practice.

When you realize there are people smarter than you… 🧠

There are a few webinars coming up that look extremely interesting. These are all AWS Online Tech Talks, so they are free to attend. I’ve been looking for webinars from other providers as well, but Amazon seems to be the leader here. If you know of any great webinars coming up, please let me know.

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Adnan Rahić (@adnanrahic). He’s a developer advocate at Dashbird as well as the author of Serverless JavaScript by Example. I chose Adnan this week because of the recent posts he’s been adding to his Medium blog and his profile. I personally love reading and sharing posts like his Crash course on Serverless with AWS — Triggering Lambda with SNS Messaging, because they provide practical and useful examples that others can follow and implement. It’s inspiring to see others using and having success with serverless, so posts like these help lower the barrier for adoption.

He’s also got another great post that looks at Containers vs. Serverless from a DevOps standpoint. Excellent read for those of you looking for an apples-to-apples comparison.

Final Thoughts 🤔

Security is an extremely important consideration when building our serverless applications. And since developers are now much closer to the execution environment, we must remember that with great power comes great responsibility. I think it’s imperative that the serverless community continues to focus on security best practices, and I hope all of us will continue to do our part.

I hope you enjoyed this issue of Off-by-none. Please send me your feedback so I can continue to make this newsletter better each week. Reach out via Twitter, LinkedIn, Facebook, or email and let me know your thoughts, criticisms, or even how you’d like to contribute to Off-by-none.

Go build some great (and secure) serverless apps. Hope to see you next week!

Take care,

Off-by-none: Issue #1

The awaiting is the hardest part

Welcome to the inaugural issue of Off-by-none! 🎉 I’m so happy that you’re here. 🙌 For those of you that have been email subscribers for a while, this will look a little different. I hope you’ll stick with me; I think this will be interesting. For those of you that are new, welcome, and thank you for joining!

I’m a constant learner, and I’ve found one of the best ways to learn is to take great notes, and then share what you’ve learned with others. This is the goal of Off-by-none. Not to be perfect, not get on a high horse 🐴 and tell you what you should and should not do, but to encourage collaboration and experimentation. Let’s push the limits of serverless. Then let’s take what we’ve discovered, and disseminate it to the masses, so we can learn from each other’s mistakes AND successes.

This newsletter has been awhile in the making. I’ve got all sorts of ideas, but this too will be a constant experiment. Your feedback, ideas, contributions, and encouragement will go a long way to making this what I truly hope it will be. Please reach out to me and share your thoughts and suggestions. They will be greatly appreciated. 🙇‍♂️

Let’s get started!

When you have thousands of concurrents user and limited MySQL connections… 🚀

Lots of people think RDBMS and Serverless don’t mix, and unless you use a series of hacks and tricks, then you’re likely to have scaling problems given the way serverless containers handle concurrent connections. In the past, I’ve often mitigated this issue by increasing the max_connections setting and gracefully handling errors. But now with Aurora Serverless, you can’t change that setting, which means LOTs of Too many connections errors.

I’ll use DynamoDB for the majority of my serverless datastore needs, but it can’t match MySQL’s ability to create normalized data structures, enforce declarative constants, provide referential integrity, and enable ACID-compliant transactions. Sometimes, you just need a relational database. And sometimes, serverless forces us to think about things differently. 🤔

Even though serverless functions are stateless, we can still reuse connections. Not only that, but status commands, like retrieving the number of max_connections and counting threads from the PROCESSLIST, are fairly inexpensive queries. I decided to run some tests to see how efficiently (and effectively) serverless functions could manage their own distributed database connections. Turns out, pretty darn well! 😜

I spent some time and turned all my learnings from the past several years (as well as my new experiments) into a new NPM module called Serverless MySQL. It manages connections for you, cleans up zombies to maintain total connection use thresholds, and even makes working with MySQL connections in Node.js super easy by adding async/await support and simple transaction workflows. I wrote a post about it called Managing MySQL at Serverless Scale that goes into a lot more detail.

You should still size your MySQL instance/cluster appropriately, but for one of my tests, I simulated 500 concurrent users per second with only 90 connections available. Not only did Lambda and MySQL not throw any errors, but the average response time was only 41 ms with the max being just over 3 seconds. Not too shabby. Bump your max_connections up to something more reasonable for that kind of load, and you should never have to gracefully handle connection limit errors again. 🤘🏻

When you’re looking for some light serverless reading… 📚

I’ve had plenty to say about cold starts in the past, but how much do they really matter, and how do they compare across the Big-3 cloud providers (AWS, Azure, and Google Cloud)? Mikhail Shilkov does a pretty darn good job answering those questions in his post Serverless: Cold Start War. Definitely worth the read. ❄️

Erica Windisch start writing again too, and you won’t be disappointed. Lessons from building a Serverless Data Pipeline with AWS Kinesis and Lambda is a relatively short read, but it is loaded with invaluable insights. If you’re thinking about using Kinesis in production, you should probably read this. 🤓

Looking for some serverless best practices? Paul Johnston (yes, the Paul Johnston, legendary cofounder of ServerlessDays), published a post with some “relatively accepted” best practices. There is a lot of debate over these, and Michael Hart and Tyler Love from Bustle jumped in with quite a difference of opinion. Both the article and the Twitter thread are good reads.  I have a feeling that this isn’t the last of this debate. 🍿

Finally, sometimes things are better late than never. Chris Munns, AWS Senior Serverless Developer Advocate, put out a post that contains a quarterly recap of all things AWS Serverless. AWS moves so fast that it’s often hard to keep up with all their innovations. This post outlines all the cool stuff AWS did with Serverless in Q2 2018. ⚡️

When getting through airport security is harder than implementing serverless security… ✈️

I’ve got some travel plans coming up! I’ll be heading to ServerlessDays NYC on October 30th and I will be going to AWS re:Invent this year! I’m hoping to meet a lot of you in person, but I’ll also be taking copious notes and will be sure to share what I learn with all of you. If you’ll be at either event, connect with me via Twitter, LinkedIn, Facebook, or email so we can meet up (and maybe grab a few 🍻).

When you join the “My post made it to the front page of Hacker News and my site crashed” club… 🏅

I’ve been very grateful for all the positive responses to my Serverless Microservice Patterns for AWS post. Lots of people found it very useful and have been using it as a reference, which is great. Someone did me the honor of posting the link to Hacker News, and on Sunday it made its way to #7! Well, my self-hosted WordPress blog running on a single EC2 instance couldn’t handle the traffic, and my site crashed. 💥🤦🏻‍♂️

But it turns out that I’m in good company. Yan Cui had the same thing happen to him earlier this year. He decided to make his blog serverless by using Shifter, a service that converts your WordPress blog into a static, serverless site. I really like this idea, but at the same time, I’m also really sick of WordPress in general. I’ve actually been thinking about this for quite some time and should have more thoughts to share soon.

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Slobodan Stojanović (@slobodan_). Not only did he write the book on Serverless Applications with Node.js, but he also has a number of great serverless posts (like here and here). But the reason I chose Slobodan this week is because of a recent talk he gave at Frontend Conference in Zurich. I wasn’t able to be there in person, but when I saw his slides, I was inspired. Convincing people to adopt serverless is a bit of a passion of mine, and the fact that Slobodan was introducing serverless to an entirely new audience (frontend developers) is a positive step towards more adoption. And, I’m also super jealous by how good his artwork is!

Final Thoughts

I hope you enjoyed this first issue of Off-by-none. It wouldn’t be possible without you and the contributions of a vibrant Serverless community. Please send me your feedback so I can continue to make this newsletter better each week. Reach out via Twitter, LinkedIn, Facebook, or email and let me know your thoughts, criticisms, or even how you’d like to contribute to Off-by-none.

Thanks again! And I hope to see you again next week! 🤞

Until then,