Off-by-none: Issue #15

It’s all about Layers…

Welcome to Issue #15 of Off-by-none. I’m glad that you could join us. 😀

Last week we recapped re:Invent and took a look at some of the excellent talks and AWS product releases. This week we’ll dig deeper into Lambda Layers and see how people are having a bit of fun with custom runtimes. Plus we’ve got more talks from re:Invent and plenty of other serverless tidbits for your mental ingestion.

Lots to get to, so let’s get started! 🚄

What to do when AWS gives people access to Custom Lambda Runtimes… ⚙️

AWS already took care of C++ and Rust for us, plus some launch partners have already added PHP and Cobol support as well. But it seems that the community is taking advantage of this new feature in a big way.

The team over at The Agile Monkeys added a Haskell runtime. Think about it, a purely functional programming language running pure functions on stateless serverless functions! Okay, maybe that’s a bit much, but if you’re a hardcore functional programmer, you may want to give this a look. 😎

Graham Krizek added Bash support, which is pretty darn cool. He even included executables like aws, scp, git, wget and a whole lot more. Think about all the interesting and powerful use cases this opens up. Just this git support alone adds a number of possibilities. 🤓

Data scientists rejoice! You can now run R on Lambda thanks to this tutorial by Philipp Schirmer. There might be some memory limitations, but overall this looks like a workable solution for all you number crunchers. 📊

There’s also this proof of concept for a Serverless Open Runtime for AWS Lambda. Definitely an interesting concept, especially the language agnostic middlewares piece. Could turn out to be a terrible idea, but definitely something to keep your eye on. 🤷‍♂️

When you want to know how to use AWS Lambda Layers… 🥞

You can certainly build Lambda Layers on your own, but several companies are now providing them as a way for you to easily instrument your code. Epsagon, PureSec, Thundra, DataDog, IOpipe, and more, have all built Layers that you can simply plug in to your existing Lambda functions without modifying your code. That’s pretty easy.

Of course, our friend Paul Johnston has some thoughts on Lambda Layers and Custom Runtimes, including initial thoughts on best practices.

If you’re looking to help influence the future of Lambda Layers, take a look at this RFC on how to handle permissions with LayerVersions in SAM. AWS always appreciates feedback from the community, so feel free to throw your hat in the ring and add your comments. 🎩

When you refuse to believe you’ve watched all the good re:Invent talks… 📺

Not all of these are available to watch, but there is still a ton of amazing re:Invent content out there that you probably missed, even if you were at re:Invent! Here are three more talks that I found to be super interesting.

Inside AWS: Technology Choices for Modern Applications (SRV305)
Tim Bray, a Senior Principal Engineer from AWS, talks us through how AWS dogfoods serverless to power many of their own services. Even API Gateway runs on Lambda. He notes that “capacity planning sucks” and that you should “use serverless whenever possible.” This talk is full of great advice, including ways to “minimize state hydration”, plus some helpful notes on the three integration patterns. Watch the talk

Reddit’s Serverless & Compute Infrastructure at Scale (STP18)
Anand Mariappan & Jesjit Birak from Reddit take us through their latest redesign process and the steps they took to avoid another incident like “the Digg Mass Exodus of 2010.” The overall process was helpful to understand, but their method for scaling their video ingestion system using serverless tech is really interesting. A great lesson for enterprises here, as they built this to run along side their existing monolith. Watch the talk

Close Loops & Opening Minds: How to Take Control of Systems, Big & Small (ARC337)
Colm MacCárthaigh, another Senior Principal Engineer from AWS, lays out ten patterns to use while building control planes for distributed systems. Since all of our serverless applications are distributed, this makes for a really useful guide when building our own applications. Colm dives a bit into control theory, but keeps the advice practical so that you can apply these techniques immediately. Watch the talk

When you’re still debating what database to use with your serverless app… ⚖️

If you plan on using DynamoDB, you may want to look at Alex DeBrie’s DynamoDB On-Demand: When, why and how to use it in your serverless applications. Plus, lots of your burning DynamoDB questions are answered in here.

If you still want to go the relational database route, check out A crash course on Serverless with AWS — Building APIs with Lambda and Aurora Serverless by Adnan Rahić. This is a great post to get you started, I just wish he didn’t use an MySQL ORM. 🤦🏻‍♂️

And speaking of MySQL, I released a new version of serverless-mysql that fixes an ENQUEUE issue. If you’re not familiar with it, this module helps you with Managing MySQL at Serverless Scale.

What to do when you need more serverless content… 🙏

Jon Vines gives us some ideas about Breaking Down the Serverless Monolith. It’s tempting to load up functions with a lot of capabilities as it keeps things “simple” and is familiar to most developers. Some good lessons learned are outlined in this post.

If you’re interested in learning some more best practices, take a look at Five Essential Principles for Developing Lambdas. I think most of these are pretty solid (especially single-purpose lambdas), plus there are some examples, which is quite helpful.

Another great thing about single-purpose functions is that they can be optimized for their specific job. Case in point, don’t overpay when waiting on remote API calls by using the appropriate memory configurations.

For you serverless security buffs, take a look at Ory Segal’s 6 Cloud Security Predictions for 2019. And if you want some hands-on experience, try going through this Serverless Security Workshop. 🔒

When you remember that Microsoft Azure has serverless functions too… ⚡️

Mikhail Shilkov is Making Sense of Azure Durable Functions for you with his new (very detailed) post. Though the title suggests this is all about Microsoft’s solution, there is quite a bit of background on microservices, event-driven applications, serverless function composition and more. Definitely worth the read if you’ve got 20 minutes or so to spare. 📖

Kate Baroni, a Software Architect at Microsoft Azure, shows us how an Azure Function can orchestrate a real-time, serverless, big data pipeline.  Plus, if you’re interested, there are some links to related posts that go into more detail. I love finding interesting use cases like this, but it’s curious to me that Azure is doing complex orchestrations within a single function (with no mention of Durable Functions). This has always been a big anti-pattern with AWS Lambda, but maybe not with Microsoft? 😕

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Simon Wardley (@swardley). Simon invented Wardley Maps, which changes the way we look at strategic planning. You can read all about how it works here (and I suggest you do). Beyond that, Simon is a huge proponent of serverless and has been predicting for quite some time that it is the future of computing. He has a number of brilliant talks about serverless (including ServerlessDays Hamburg and Serverlessconf San Francisco 2018), plus his Twitter feed often contains entertaining back-and-forth arguments as to why serverless adoption is inevitable (see this recent Twitter thread). I’m a big fan of Simon and appreciate the work he is doing to make the case for serverless.

Final Thoughts 🤔

Lambda Layers is exposing serverless computing to a number of new communities, and people have been rushing to add support for all kinds of runtimes and service integrations. A recent report by Gartner identified “serverless computing” as the number one key trend for 2019 and noted that “more than 20 percent of global enterprises will have deployed serverless computing technologies by 2020.”

We are still early in this journey, but as Simon Wardley says, “No more questions on serverless. It’s not an ‘if’ but ‘when’. Get on with learning.” This is sage advice, and what we’re encouraging with this newsletter.

I hope you’ve enjoyed this issue of Off-by-none. I love getting your feedback. It is always most welcome and much appreciated. Your suggestions help me make this newsletter better each week. Please feel free to contact me via Twitter, LinkedIn, Facebook, or email and let me know your thoughts, criticisms, and if you’d like to contribute to Off-by-none.

Go build some amazing serverless apps!

Take care,
Jeremy

Off-by-none: Issue #13

Live from AWS re:Invent…

Welcome to Issue #13 of Off-by-none. We’re coming to you LIVE from AWS re:Invent in Las Vegas!

Last week we looked at some clever use cases for Step Functions, revisited serverless microservices, and made the serverless case for startups. This week we rethink serverless+RDBMS, challenge the objections of laggards, protect ourselves from DoS and other attacks, and of course, look at some new AWS product launches.

So many amazing things to get to today, so let’s jump right in!

When you’re not sure if RDBMS and serverless mix… ☯

Many of us wished for RDS HTTP Endpoints, and the other day, AWS announced that you can now access your Amazon Aurora Serverless Database with the New Data API (Beta). No VPCs, no connection management, and automatic scaling with Aurora Serverless. Almost sounds too good to be true. 😳

And… it sort of is (for now). In Aurora Serverless Data API: A First Look, I share the results of a few experiments I ran as well as some of my initial thoughts on the implementation. TLDR; The latency is really bad and this isn’t ready for primetime. But like all things AWS, it’ll get much better before GA.

Is RDBMS in serverless applications even a good idea? Paul Johnston shares his thoughts on Serverless and Data Rigidity and argues that other technologies (like NoSQL) have removed the need for them. He’s not wrong, but there are still plenty of use cases that relational databases work well for. One thing we can definitely agree on: AVOID ORMs! 🙌

When you’re looking for some serverless inspiration… 💡

Serverless, Inc. is wrapping up #NoServerNovember with the re:Invent serverless virtual hackathon. Build a serverless app for a non-profit, feel good about yourself, and win some swag.

If you want to get a bit more complex, try building a chat application using AWS AppSync and Serverless.

Are you writing your code in Python? AWS SAM CLI just introduced the sam build Command that lets you easily package all your dependencies. Or you can learn How To Package External Code In AWS Lambda Using the Serverless Framework.

What to do when your boss won’t let you play with serverless… 👨🏻‍💻

James Beswick outlines five common objections to adopting serverless in his new post, Scared Serverless — How do you handle opposition from your IT group? Lots of ammunition in here if you find yourself needing to defend your (very wise) decision.

If they’re still not convinced, maybe this Twitter thread will help. Simon Wardley says, “The overwhelming output of most businesses is waste. Serverless is way larger than you think. More significant than cloud was.” It’s definitely worth the read (plus there’s maps).

When you realize you’re still responsible for securing your serverless application… 🔒

Avi Shulman from PureSec wrote a great post on Lambda DoS Mitigation Strategies. See how different invocation types and retry policies can be leveraged by attackers to wreak havoc on your serverless applications. Lots of practical tips in here including a number of best practices and tips to minimize your exposure.

Want to add even more security to your serverless app? Amazon API Gateway has added support for AWS WAF, which means no more creating regional endpoints and using your own CloudFront distribution. It still won’t prevent event injection, but it’s a good start.

And just when you think that npm audit will protect you from third-party package vulnerabilities, we discover another widely used open source software that contained a bitcoin-stealing backdoor. Luckily it only has 2 million weekly downloads. 🤦🏻‍♂️ A friendly reminder to minimize dependencies in your serverless applications.

What to expect when 50,000 AWS fans in Vegas are waiting for more product updates… 🚀

There’s only been one full day of re:Invent and AWS has already announced a number of products and services that are pushing serverless to a whole new level. I’ve heard a lot of whispers, so expect many more to come over the next few days. 🤘🏻

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Chris Munns (@chrismunns). Chris is a Principal Developer Advocate for Serverless at Amazon Web Services and a great resource for anyone working with (or interested in) serverless. He’s a regular speaker at events, an AWS blog contributor, a host on Serverless Bytes, and he also puts on the occasional webinar. Even though he works for AWS, he’s a huge advocate for serverless computing in general and will always jump into a good debate on Twitter. This week he’s not only giving a number of talks at re:Invent, but also finding some time to spend with members of the serverless community.

Final Thoughts 🤔

The buzz around serverless at re:Invent is absolutely amazing. Every session I’ve attended so far has been bursting with people that are either already using it in production, or are hoping to start. I know we are in a bit of bubble here, but it’s clear that AWS is continuing to make massive investments in serverless technologies and wants to continue to be the market leader. Exciting times ahead.

I hope you’ve enjoyed this issue of Off-by-none. Your feedback and suggestions are always welcome and much appreciated. It helps me make this newsletter better each week. Please feel free to contact me via Twitter, LinkedIn, Facebook, or email and let me know your thoughts, criticisms, and if you’d like to contribute to Off-by-none.

Go build some amazing serverless apps and enjoy the rest of re:Invent! ⚡️

I’ll be here all week😉
Jeremy

P.S. If you liked this newsletter, please share with your friends and coworkers. I’d really appreciate it. Thanks!

Aurora Serverless Data API: A First Look

On Tuesday, November 20, 2018, AWS announced the release of the new Aurora Serverless Data API. This has been a long awaited feature and has been at the top of many a person’s #awswishlist. As you can imagine, there was quite a bit of fanfare over this on Twitter.

Obviously, I too was excited. The prospect of not needing to use VPCs with Lambda functions to access an RDS database is pretty compelling. Think about all those cold start savings. Plus, connection management with serverless and RDBMS has been quite tricky. I even wrote an NPM package to help deal with the max_connections issue and the inevitable zombies 🧟‍♂️ roaming around your RDS cluster. So AWS’s RDS via HTTP seems like the perfect solution, right? Well, not so fast. 😞

Continue Reading…

Off-by-none: Issue #11

After this, there is no turning back

Welcome to Issue #11 of Off-by-none. I’m happy that you’re here! 🙌

Last week we recapped ServerlessNYC and talked quite a bit about serverless adoption. This week we’re going to point out some more resources for those getting started, as well as offer up plenty of options if you’re looking to take the red pill and go down the serverless rabbit hole. 🐇

Here we go! 🕺

What to read when you want to amp up your serverless knowledge… 🔈

Danilo Poccia has written a free ebook, Agile Development for Serverless Platforms. This book is over 100 pages and has a great section on architectural patterns. There is plenty to learn from this free resource and it is well worth a look. 📖

The team over at Financial Engines wrote a guide to help us with managing disaster recovery with DynamoDB. AWS DynamoDB: Backup and Restore Strategies looks at both Point-in-Time Recovery and On-Demand Backups. Lots of useful information here including configuration and pricing. 👨🏻‍💻

Finally, Thundra published a great piece that shows us how to Debug AWS Lambda Node.js Functions in Production Without Code Change. I really like the idea of automated instrumentation as it cuts down the burden on developers and keeps your code a bit cleaner. It can also ensure we don’t lock ourselves in to a specific software vendor. 📈

When you want to get started with serverless… 🏋️‍♂️

There have been a lot of new “Getting Started with Serverless” posts this week. I really like that more people are starting to create this type of content. The more that’s out there, the more likely someone is to come across it and get to that serverless “aha” moment. If you’re new to serverless, here are a few posts to get you started:

And don’t forget that the #NoServerNovember Challenge (hosted by Serverless, Inc.) is still going on. These challenges will give you something interesting to work on and let you go beyond the standard “Hello World” tutorial.

When you’re not ready to give up RDBS with serverless… 🤓

In our inaugural issue we introduced the serverless-mysql package with my Managing MySQL at Serverless Scale post. David Zhang (@Zigzhang) has taken this even further and created a five part series to help others get started. In his first post, Serverless & RDBS (Part 1) — Set up AWS RDS Aurora and Lambda with serverless, David lays out some background, then gives you full examples to get you up and running.

He’s also published Part 2 (Set up EC2 instance to securely connect to your Aurora DB) and Part 3 (Set up database migrations with umzug) with the final two parts (Set up continuous deployment to migrate database with CircleCI and Set up local development environment with serverless-offline and Docker) coming soon. These are sure to be helpful guides for anyone looking to build serverless apps with RDBS backends.

Of course, re:Invent is right around the corner, so let’s hope we get HTTP endpoints for RDS! 😬

When you feel like there are a lot of conferences… ✈️

Speaking of re:Invent, it is less than two weeks away! 🎉 This is the first year that I’m attending so I’ve been looking for tips like this and this. I’m excited for some of the sessions I’m attending and will be at several events as well. If we haven’t connected already, please contact me so we can meet up.

In other conference news, Serverless Computing London is happening right now and it is chockfull of great speakers. Follow their Twitter feed to see some snippets from the event. Some of the slide decks have been posted as well, so check those out. I was looking at Timirah James’ Function Composition in a Serverless World talk, good stuff. Hopefully we’ll see the videos posted soon. ⚡️

Also, ServerlessDays BOSTON finally has a date! The event is scheduled for March 12, 2019 at the Microsoft New England Research & Development Center. More information about our call for papers and sponsorship opportunities is coming soon. 🎊

When you realize that AWS has no plans to slow down their serverless innovations… 🚀

AWS has released several new features recently that could have a profound impact on our serverless applications. Some of these are pretty exciting. Now just imagine what they are going to announce at re:Invent! Here are just a few of the recent updates:

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Alex Casalboni (@alex_casalboni). Alex is an AWS Technical Evangelist, Serverless champion, co-organizer of ServerlessDays Milan and the serverless meetup there, contributor to serverless open source projects, and a regular conference speaker spreading the serverless gospel. He also helps coordinate ServerlessDays conferences around the word, including helping me and the Boston team. Thanks for all you do, Alex!

Final Thoughts 🤔

As much as I still worry that serverless adoption will be slower than I had hoped, the amount of innovation and new faces in the community is really encouraging. I’m already aware of a few announcements planned for re:Invent, but I also know that there will be a ton more. Other cloud providers are also pushing serverless innovations, and I expect Google and Azure to be announcing new things soon as well.

Serverless still has a long way to go, but all of these new tools, platforms, cloud provider features, conferences, and enthusiasm from the community, is helping to expose this paradigm to a much larger audience. I’m going to continue to write and promote it as much as I can, because there is little doubt in my mind that this is the future of application development.

I hope you’ve enjoyed this issue of Off-by-none. Feedback and suggestions are always welcome and appreciated. It helps me make this newsletter better each week. Please feel free to contact me via Twitter, LinkedIn, Facebook, or email and let me know your thoughts, criticisms, and if you’d like to contribute to Off-by-none.

Go build some great serverless apps and spread the word. 📣

See you next week,
Jeremy

P.S. If you liked this newsletter, please share with your friends and coworkers. I’d really appreciate it! 😉

Off-by-none: Issue #5

It’s nobody’s fault but mine

Welcome to Issue #5 of Off-by-none. I really appreciate you being here! 🙌

Last week we took a more in-depth look at how we can use traditional microservice patterns and adapt them to our serverless applications. An Introduction to Serverless Microservices was the first in a series of posts to demonstrate the power of serverless microservices and how they can be implemented.

There are a lot of topics to cover, but I consistently get questions about how to handle errors in distributed systems. This is especially significant with serverless architectures since functions are ephemeral and observability is less than ideal. I’m working on a new post on this topic that I’ll share soon.

In the meantime… ⏰

When you’re looking for a good serverless read… 📖

The Case of the Broken Lambda by Vicki Boykis is an interesting read that turns debugging a Lambda function into a modern Sherlock Holmes mystery. The issue described is very specific, but there are some valid lessons in here.

Speaking of debugging, Investigating spikes in AWS Lambda function concurrency by Ian Carlson, Principal Solutions Architect at AWS, is another great read. This post gives you some useful strategies for tracking down issues caused by errant code or upstream scaling that can exacerbate concurrency issues.

Finally, Adnan Rahić published A crash course on Serverless with AWS — Image resize on-the-fly with Lambda and S3. Interesting use of Docker to package binaries for a Lambda function. This is a great example of how the Lambda runtime environment can support much more complex use cases.

When the idea of “serverless” relational databases piques your interest… 🕵️‍♂️

Aurora Serverless has been generally available for almost 2 months now. I was really excited when it came out and published Aurora Serverless: The Good, the Bad and the Scalable which goes into quite a bit of detail regarding cost and performance. I notices that AWS is hosting some webinars this week that might be interesting to those who want to explore this technology in more depth. You can register here for a live session or to get the replay when it’s available.

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Paul Johnston (@PaulDJohnston). Paul was one of the co-founders of ServerlessDays (formerly Jeff), a senior developer advocate for serverless at AWS, and a serial CTO that has embraced serverless in all his recent endeavors. I really like the way Paul thinks about serverless, and while his occasional blog post is always worth the read, I’ve found myself spending a lot of time lately watching some of his conference talks. Like this one and this one and definitely this one.

One of the things I like so much about Paul is that he’s opinionated. I don’t always agree with some of his opinions 😉, but I definitely like how he makes his case for serverless and how it should be done. With best practices still emerging in serverless, I think it’s important for new developers to have some sort of framework to start with. Agree or disagree, Paul continues to put his ideas out there.

Final Thoughts 🤔

I’ve spent a lot of time this past week continuing to work on our serverless microservices at AlertMe. I’ve been immersed in this for the last few months, and through that process, some of my thinking has evolved a bit. I’ve discovered so many nuances, complexities, and workarounds, that I’ve had to stop a few times to build tools (like serverless-mysql) to encapsulate some of them. While these are obviously helpful, I feel like some of this tooling is starting to bog down the simplicity that is (or was) serverless. My boilerplates are getting a bit out of hand, which is something to think about.

I hope you enjoyed this issue of Off-by-none. Please send me feedback and suggestions to make this newsletter better. Contact me via Twitter, LinkedIn, Facebook, or email and let me know your thoughts, criticisms, or even how you’d like to contribute to Off-by-none.

Go build some great serverless apps. Hope to see you all next week!

Take care,
Jeremy

🚀 Project Update:

Serverless MySQL: v1.1.0 Released

Serverless MySQL v1.1.0 adds additional transaction support capabilities to allow users to retrieve interim query results for use with future queries. This is useful for getting the insertId from previous queries when performing transactions. Read More...

Managing MySQL at Serverless Scale

“What? You can’t use MySQL with serverless functions, you’ll just exhaust all the connections as soon as it starts to scale! And what about zombie connections? Lambda doesn’t clean those up for you, meaning you’ll potentially have hundreds of sleeping threads blocking new connections and throwing errors. It can’t be done!”  ~ Naysayer

I really like DynamoDB and BigTable (even Cosmos DB is pretty cool), and for most of my serverless applications, they would be my first choice as a datastore. But I still have a love for relational databases, especially MySQL. It had always been my goto choice, perfect for building normalized data structures, enforcing declarative constants, providing referential integrity, and enabling ACID-compliant transactions. Plus the elegance of SQL (structured query language) makes organizing, retrieving and updating your data drop dead simple.

But now we have SERVERLESS. And Serverless functions (like AWS Lambda, Google Cloud Functions, and Azure Functions) scale almost infinitely by creating separate instances for each concurrent user. This is a MAJOR PROBLEM for RDBS solutions like MySQL, because available connections can be quickly maxed out by concurrent functions competing for access. Reusing database connections doesn’t help, and even the release of Aurora Serverless doesn’t solve the max_connections problem. Sure there are some tricks we can use to mitigate the problem, but ultimately, using MySQL with serverless is a massive headache.

Well, maybe not anymore. 😀 I’ve been dealing with MySQL scaling issues and serverless functions for years now, and I’ve finally incorporated all of my learning into a simple, easy to use NPM module that (I hope) will solve your Serverless MySQL problems.

Continue Reading…