Serverless Security: Locking Down Your Apps with FunctionShield

I've written quite extensively about serverless security, and while you don't need to be an expert on the matter, there are a number of common sense principles that every developer should know. Serverless infrastructures (specifically FaaS and managed services) certainly benefit from an increased security posture given that the cloud provider is handling things like software patching, network security, and to some extent, even DDoS mitigation. But at the end of the day, your application is only as secure as its weakest link, and with serverless, that pretty much always comes down to application layer security.

In this post we're going to look at ways to mitigate some of these application layer security issues by using some simple strategies as well as a free tool called FunctionShield.

Audio Version:

Update September 27, 2018: FunctionShield now supports Java in addition to Node.js and Python.

Update October 18, 2018: FunctionShield now also lets you disable read access to the function's handler and  prevent source code leakage.

Who's responsible for serverless security?

While there is nothing inherently new that's specific to serverless when it comes to application security practices, there is a shift in terms of where security responsibility ultimately lies. Obviously, developers should implement proper programming practices by sanitizing all their inputs, handling errors correctly, and protecting sensitive user data with proper programmatic access controls. But with serverless, access and error logs are no longer automatically generated for you, which now puts this into the hands of the developer.

Even more scary, is how close serverless takes the developer to the execution environment. I've been using IAM roles since they were invented, and I still find myself consulting the docs from time to time to be sure I'm doing it correctly. Putting that responsibility on a developer often results in * privileges that could give attackers carte blanche access to your AWS environment.

We also train developers to be lazy (this isn't a criticism), meaning that we don't want them reinventing the wheel every time they write a new program. They should be focusing on solving business problems, not trying to figure out ways to manage MySQL connections or handle and respond to API Gateway requests. 😉 This means we encourage the use of third-party dependencies. Dependencies that are easily compromised and can leak access keys, execute remote calls, and even be used to mine cryptocurrency.

These events may all seem unlikely, until they happen to your organization. And the more responsibility we put on developers to mitigate these issues, the more likely the lack of security training will rear its ugly head. In my experience, most junior developers (and sometimes even "experienced" ones) will respond with, "Umm, what's SQL injection?" 🤦🏻‍♂️

Emerging tools and best practices 🔦

I'm not trying to scare anyone off. I'm obviously a huge proponent of serverless and I definitely think you should give it a try if you haven't already. But like most things shiny and new, it takes some time to work out the kinks and develop a set of best practices.

The community has been working hard to develop tools like the Serverless Framework to help us deploy and configure our serverless applications more easily. AWS has developed their own open source Serverless Application Model (SAM), which is tightly coupled with CloudFormation to configure and deploy complex applications. There are even several companies building tools that focus on observability within our serverless applications, extending the basic insights that cloud providers are offering.

However, there has been little focus on security, which is what intrigues me about a company called PureSec. While I have no official relationship with them, I have chatted with their team a number of times. As a result, I have gotten a much better understanding of what they do and why it's needed. I have yet to try their primary serverless security offering, but I did get a chance to play around with their free FunctionShield product. This is obviously a subset of what their full service can do, but it is quick and easy to implement, and provides some really great default protections.

What is FunctionShield and why do I need it?

FunctionShield is a security library (currently available for Node, Python and Java) that you package with your Lambda functions. Without using monkey-patching (which is a big deal that I'll explain in a minute), it gives you the ability to:

1. Disable outbound internet connectivity (except for AWS resources) from the serverless runtime environment
2. Disable read/write on the /tmp/ directory
3. Disable child process execution
4. Disable read access to the function's handler and  prevent source code leakage (new as of October 18, 2018)

I reached out to Ory Segal, CTO at PureSec, and asked him about these use cases and why they were so important. "We've heard of many customers that place their Lambda functions inside a VPC and position a NAT to block it from communicating outbound. Needless to say, that's the wrong way to do things," he said. "We've also met customers that asked us to block writing to the /tmp directory so that developers will not use it. This ask was because they don't know whether this data will eventually leak during other executions between users."

I think he is right on here. First of all, don't put your Lambda functions in a VPC if you don't have to. And second, the power that comes with global variable reuse and /tmp directory access is a double-edged sword. Reusing database connections and initialized packages is extremely effective and can gain you tremendous gains in speed, but you also run the risk of exposing sensitive user data if you don't use them correctly.

He continued, "We were looking to give confidence back to developers - allow them to control what the runtime can and can't do, and most importantly, provide a way to monitor any kind of outbound connectivity, or child process execution."

Again, I completely agree. Lambda functions are actually mini-Linux servers that have capabilities well beyond simple function execution. A compromised third-party dependency, coupled with loose IAM permissions, could wreak havoc on your AWS environment. Now with FunctionShield, Ory says, "This is the first time developers can monitor 3rd party open source libraries in serverless environments."

This all made perfect sense to me, so the next step was to give it a try and see if I could outsmart the security experts at PureSec (spoiler alert: I couldn't 🙅‍♂️).

Testing FunctionShield

I had to sign up for a security token here, which does require you to provide an email address, your name, and a company name. I typically don't like to give out information freely like this, but a) they're giving you a FREE tool, and b) they are a security company, so your info is most likely safe. 🔒 You'll get an email with your security token.

I installed FunctionShield as a node dependency with a simple NPM install command (but it's also available for Python as well):

defaultCopy code
npm i @puresec/function-shield

Then I required it in one of my functions, added the sample code provided, and pasted in my token.

javascriptCopy code
const FunctionShield = require('@puresec/function-shield')

FunctionShield.configure(
{
    policy: {
        outbound_connectivity: "block",
        read_write_tmp: "block",
        create_child_process: "block"
    },
    token: 'amVyZW15QGplc...8gbY+alv'
 })

An important thing to note here is that FunctionShield does not "wrap" your code. It requires ZERO instrumentation beyond simply requiring it and providing the configuration. I promise I'll tell you why this is so powerful in just a minute.

Then I added a remote API call with the native https module and ran my function:

javascriptCopy code
const https = require('https')

https.get('https://jsonplaceholder.typicode.com/users', (resp) => {
 ...
})

I got a "getaddrinfo Unknown system error -999 jsonplaceholder.typicode.com:443" error and the following in my CloudWatch logs:

So far so good. So then I tried writing to the /tmp directory:

javascriptCopy code
const fs = require("fs")

fs.writeFileSync("/tmp/test.txt", "data");

Denied again. This time I got the error, "Unknown system error -999: Unknown system error -999, open '/tmp/test.txt'" with the following CloudWatch log entry:

Now for the child process execution:

javascriptCopy code
const child_process = require("child_process")

child_process.execSync("ls -la")

I got a "spawn Unknown system error -999" error and the corresponding CloudWatch entry:

With just the basic configuration, FunctionShield blocked me from doing a number of things that could get me in trouble if I left them unchecked. Ory told me,  "We wanted something that is dead simple, so that any developer will find it simple to deploy. You can essentially embed FunctionShield in all of your functions, for free, automatically during build time."

It is pretty darn simple to add. And since it requires no instrumentation, you could just include it in a standard boilerplate function wrapper along with other initialization code and libraries.

Okay, so out of the box it seems simple enough, but I wanted to get an idea of how well it played with other AWS services.

Digging a little deeper

According to PureSec, FunctionShield will "disable outbound internet connectivity (except for AWS resources)." I tried grabbing a file from S3, no problems there. Invoked another Lambda function with the AWS SDK, all good. Finally, I tried sending a message to an SNS topic, and it worked perfectly. Next I tried adding X-Ray.

Uh oh. It looks like X-Ray needs to write to that /tmp directory. I brought this to Ory's attention and he was already aware of it and is working on a patch. Given how sophisticated their software is, I'm sure the patch is more complex then simply allowing the /tmp/.aws-xray directory to be writable.

Update September 9, 2018: The X-Ray issue has been fixed in v1.0.4.

So I changed the read_write_tmp config to "alert" instead of "block", and then got the following:

Well that's kinda cool. PureSec probably would have rathered that I didn't mention this X-Ray bug, but I am because it actually got me thinking about something. What if I DO need to write to the /tmp directory, or make outbound HTTP calls? Is blocking an all or nothing proposition? I asked Ory if I could whitelist specific URLs. He said that advanced functionality like that (blocking by IP , domain or CIDRs) was available with the PureSec Serverless Security Platform, which is their paid service. I totally respect that. We can't get everything for free, but a guy can certainly try! 😬

Tracing, blocking, and FINALLY, monkey patching 🐒

Now that X-Ray was working, I enabled some tracing on the https module to see how FunctionShield would affect my reports.

Exactly what I was hoping for. My traces still show the attempted call with the error caused by FunctionShield blocking it.

Then I decided to switch things up a bit. I used the request node module instead of the built-in https one. Sure, request uses http and https under the hood, but AWS X-Ray isn't smart enough to trace those requests. So for example, this won't work:

javascriptCopy code
const request = AWSXRay.captureHTTPs(require('request'))

In fact, request in this example, will be undefined. I don't necessarily believe this is a huge deal with X-Ray, you can still trace these requests, it just takes some extra work and instrumentation. This is because X-Ray is monkey-patching the https client. There are two major security issues with this. First, monkey-patching only affects the current instance and can easily be monkey-patched over to remove whatever safeguards may have been put in place. And second, there are multiple ways to execute remote calls, so a rogue third-party module could completely circumvent something like https altogether by simply using a child processing to curl some remote resource.

But as I mentioned in the beginning, FunctionShield DOES NOT monkey-patch!  Instead, it intercepts the actual underlying connection. This is a HUGE deal because this will guarantee that any outgoing connection will need to be inspected by FunctionShield. I ran a simple test with request and FunctionShield blocked it just like it did with https, and of course, I got my CloudWatch Logs entry:

Taking it to the next level 🔈

Now that we know how effective FunctionShield can be to block outgoing connectivity, disable /tmp directory reads and writes, and to stop child processes in their tracks, let's look at some creative ways we can use this to our advantage.

First, what if we do need to make some outgoing HTTP calls? We might not be able to whitelist domains, but we can still tell FunctionShield to "alert" us whenever an outbound call is made. Then we can create a metric filter with a filter pattern like this:

defaultCopy code
{ $.policy = "outbound_connectivity" && $.details.host != "my-okay-domain.com" }

Now we can set a CloudWatch Alarm to notify us whenever an HTTP call is made to something other than "my-okay-domain.com".

The same is true with /tmp directory reads and writes. We might not be able to block the bad ones if they are sometimes needed, but we certainly can know about them using the above technique. This would give us the ability to quickly assess the vulnerability and patch it before it does too much damage.

The Pros and Cons

Before we wrap up, I do want to address a few pros and cons about FunctionShield. It is free, but it is not open source. You'll see that when you look at the files in your node_modules directory, that they are compiled using node-gyp. This means that you can't really see what is going on behind the curtain. There is a lot of intellectual property behind FunctionShield, and I totally respect that. I just think it's just important to be aware of the modules you're using.

The PureSec team is also upfront about the fact that FunctionShield will periodically send metrics to them if you don't disable analytics. Disabling analytics is as simple as adding the following to your policy:

javascriptCopy code
disable_analytics: true

I was curious what type of analytics were actually being sent, so Ory provided me with the following sample:

javascriptCopy code
{
  "version":"1.0.2",
  "email":"you@example.com",
  "aws_execution_env":"AWS_Lambda_nodejs6.10",
  "aws_lambda_function_name_sha256":"8390b7b21071...af05dfe6"
}

As you can see, there is very little information here and certainly not enough to gain any real insights. (btw, the email is just the one you provided when you signed up for the token). I asked Ory why there was a need for this at all, and he again was upfront and honest, "It's impossible to learn anything else from the data. The only reason we added analytics was to see if the library is being adopted."

I think that is extremely fair. Given the time, energy and money that has gone into building the technology at PureSec, some simple adoption feedback is a small price to pay for access to the power of FunctionShield. But even so, you can disable it if you want to.

Final Thoughts

I was very impressed by FunctionShield, and I can't really think of a compelling reason NOT to use it. I'm obviously a huge supporter of open source software, but with open source comes a number of security risks. FunctionShield might actually be the Ying to open source's Yang. A number of dangerous and destructive exploits that are sure to creep into third-party packages are mitigated simply by including this small package within your functions.

I think Ory hit the nail on the head when he said, "Some developers are afraid to adopt serverless since they cannot control the runtime environment in which the function is running in. This means that they cannot monitor what the function is doing, whether or not it is leaking data, for example, because they are using untrusted open source packages."

I don't want developers to be afraid. But they NEED to be aware of the risks, especially now that serverless has put this extra level of responsibility on them. For me, FunctionShield goes a long way towards adding that extra layer of protection to applications that rely heavily on the security practices of a third-party. We can't control or have the insight we want into someone else's security practices, but we can certainly be more vigilant with our own. Adding FunctionShield would be a wise first step.