Last week I was deploying an app to AWS Lambda using Serverless and I accidentally misspelled a stage name. This resulted in deploying an entire CloudFormation stack with the wrong name, which wasted time and resources. I immediately realized that I spelled it wrong, so I was able to undeploy the stack quickly. I fully admit that misspelling the name was my fault, but since Serverless can deploy complex CloudFormation stacks that provision users, databases, SNS topics, etc., it seemed like there should be a better way to avoid this simple mistake.
Programming
Node.js Modules I Can’t Live Without
I’ve been writing and maintaining so many Node.js scripts lately that I’ve lost count. Node modules allow you to extend scripts to do just about anything you need. There are several Node.js modules that I find myself using over and over again in nearly every project I work on. Here are a few of my favorites that I’m sure will make writing your Node.js scripts easier and more bulletproof. All of these modules are well-documented and supported.
How To: Reuse Database Connections in AWS Lambda
Update 9/2/2018: I wrote an NPM module that manages MySQL connections for you in serverless environments. Check it out here.
I work with AWS Lambda quite a bit. The ability to use this Functions-as-a-Service (FaaS) has dramatically reduced the complexity and hardware needs of the apps I work on. This is what’s known as a “Serverless” architecture since we do not need to provision any servers in order to run these functions. FaaS is great for a number of use cases (like processing images) because it will scale immediately and near infinitely when there are spikes in traffic. There’s no longer the need to run several underutilized processing servers just waiting for someone to request a large job.
AWS Lambda is event-driven, so it’s also possible to have it respond to API requests through AWS’s API Gateway. However, since Lambda is stateless, you’ll most likely need to query a persistent datastore in order for it to do anything exciting. Setting up a new database connection is relatively expensive. In my experience it typically takes more than 200ms. If we have to reconnect to the database every time we run our Lambda functions (especially if we’re responding to an API request) then we are already adding over 200ms to the total response time. Add that to your queries and whatever additional processing you need to perform and it becomes unusable under normal circumstance. Luckily, Lambda lets us “freeze” and then “thaw” these types of connections.
Update 4/5/2018: After running some new tests, it appears that “warm” functions now average anywhere between 4 and 20ms to connect to RDS instances in the same VPC. Cold starts still average greater than 100ms. Lambda does handle setting up DB connections really well under heavy load, but I still favor connection reuse as it cuts several milliseconds off your execution time.
Better Data Management in Applications
Posted in Programming
Interacting with a datastore is generally an integral part of any web application. Without access to data, we typically can’t do anything useful or particularly interesting. The problem for developers is that every application brings its own specific set of data needs along with a unique data architecture and schema. While data technologies have SDKs for writing interactions across different programming languages, developers are likely required to write a lot of boilerplate code just to glue all the pieces together. Add in caching, data validation, sharding, and other complexities, and managing access to data quickly starts monopolizing your development team’s time. Time that could be better spent adding value for your customers to your application.
In this post I’ll discuss some techniques and technologies to make managing data between our apps and datastores more efficient, consistent and scalable.
The Beauty of Javascript Composition
I’ve been heavily into functional programming with Javascript for quite some time now. Every new line of code I write takes advantage of ES6’s shorthand syntax and functional programming techniques. When updating existing code, I’ll generally use the opportunity to refactor it to a more functional style. But perhaps the greatest benefit is function composition, the process of combining two or more functions to produce a new function.
Function composition lets us combine multiple functions into steps that transform our data as it flows through them. It’s like an assembly line where each step alters the data in some way. Technically you don’t need to use functional code to create a composable function, but when you do, the result is clean, elegant, easily reasoned, and beautiful code.
Hack, Refactor, Repeat
I’ve been writing a lot of code lately. And I mean A LOT. Between AlertMe, some side projects, and an open source project I’m working on, I’ve been pumping out a ton of code. Luckily, it’s mostly been using the same programming language (except AlertMe’s Symfony app), so I’ve been able to really focus on producing some solid output. The problem comes down to my constant struggle between perfectionism and actually shipping something. In this post I’ll discuss some strategies to ship code faster, without compromising quality.
Test-Driven Development
Posted in Programming on June 15, 2016
There are a lot of methodologies available for development. I’ve done everything from Agile to Waterfall, either iterating as we go, or planning a really large project all the way through upfront. I’ve found pros and cons to all the different methodologies, but in the end, I prefer to take an iterative approach. The problem with iterating quickly, or as Mark Zuckerberg would say, “Move fast and break something”, is that you do, in fact, break things. In smaller codebases this might not be that big of a deal, but when dealing with larger scale production systems, breaking something is simply not tolerable. A good solution I’ve found is to implement a “Test First” development methodology where you building unit testing first, and then write the code to pass the tests.
The Curious Case of PHP Template Engines
Update: Since this post was written, serverless technology has become my go to choice for most projects. I still like and use PHP, but I would most likely not build another major project with it. 😬
I’ve been writing PHP code since 1998, so it feels very familiar to me. I built my last major web project in PHP, and if I had to do it again, I’d make the same choice. So whenever a task comes up that could be automated with some quick code, I typically default to PHP. Recently I found myself battling with some reporting templates I inherited that were coded as a mix of Python, SQL, HTML, and Javascript. The finalized reports need to be converted to PDFs, which means the PDF converter needs to be able to process Javascript. This has proven to be a problem since the prior developer was unable to find a good solution. So now I’m left with a highly inefficient, manual process.