Developing Serverless Applications Locally with the “serverless-cloudside-plugin”

Developing and testing serverless applications locally can be a challenge. Even with tools like SAM and the Serverless Framework, you often end up mocking your cloud resources, or resorting to tricks (like using pseudo-variables) to build ARNs and service endpoint URLs manually. While these workarounds may have the desired result, they also complicate our configuration files with (potentially brittle) user-constructed strings, which duplicates information already available to CloudFormation.

This is a common problem for me and other serverless developers I know. So I decided to come up with a solution.

Introducing the serverless-cloudside-plugin

This new Serverless Framework plugin allows you to use AWS CloudFormation intrinsic functions (such as !Ref and !GetAtt) to reference cloud resources during local development. When used in your environment variables, these values are replaced with the same identifiers used when deployed to the cloud!

You’ll have access to the correct resource identifiers when invoking your functions locally, using the serverless-offline plugin, or using a compatible test runner plugin that uses the serverless invoke test command. You can now keep your serverless.yml files free from pseudo variables and other concatenated strings by simply use the built-in CloudFormation features.

Installing the plugin

Using the Serverless plugin manager:

Install the module using npm:

Add serverless-cloudside-plugin to the plugin list of your serverless.yml file:

Using the serverless-cloudside-plugin

When executing your function locally, the serverless-cloudside-plugin will replace any environment variable that contains either a !Ref or a !GetAtt that references a CloudFormation resource within your serverless.yml file.

In the example below, we are creating an SQS Queue named myQueue and referencing it (using a CloudFormation intrinsic function) in an environment variable named QUEUE.

If we deploy this to the cloud, our !Ref myQueue will be replaced with a QueueUrl (e.g. https://sqs.us-east-1.amazonaws.com/1234567890/sample-service-dev-myQueue). We can then use that when invoking the AWS SDK and working with our queue. However, if we were to invoke this function locally using sls invoke local -f myFunction, our QUEUE environment variable would return [object Object] instead of our QueueUrl. This is because the Serverless Framework is actually replacing our !Ref with: { "Ref": "myQueue" }.

There are workarounds to this, typically involving using pseudo variables to construct our own URL. But this method is error prone and requires us to hardcode formats for the different service types. Using the serverless-cloudside-plugin, you can now use the simple reference format above, and always retrieve the correct PhysicalResourceId for the resource.

Invoking a function locally

Once the plugin is installed, you will have a new invoke option named invoke cloudside. Simply run this command with a function and it will resolve all of your cloud variables and then execute the standard invoke local command.

PLEASE NOTE that in order for resources to be referenced, you must deploy your service to the cloud at least initially. References to non-deployed resources will be populated with “RESOURCE NOT DEPLOYED”.

All invoke local parameters are supported such as --stage and --path, as well as the new --docker flag that lets you run your function locally in a Docker container. This mimics the Lambda environment much more closely than your local machine.

By default, the plugin will reference resources from your current CloudFormation stack (including your “stage” if it is part of your stack name). You can change the cloudside stage by using the --cloudStage option and supplying the stage name that you’d like to use. For example, if you are developing in your dev stage locally, but want to use a DynamoDB table that is deployed to the test stage, you can do the following:

This will populate any ${opt:stage} references with dev, but your !Ref values will use the ones from your teststage.

You might also want to pull values from an entirely different CloudFormation stack. You can do this by using the --stackName option and supplying the complete stack name. For example:

Using with the serverless-offline plugin

The serverless-offline plugin is a great tool for testing your serverless APIs locally, but it has the same problem referencing CloudFormation resources. The serverless-cloudside-plugin lets you run serverless-offline with all of your cloud variables correctly replaced.

The above command will start the API Gateway emulator and allow you to test your functions locally. The --cloudStageand --stackName options are supported as well as all of the serverless-offline options.

Using with a test runner plugin

You can use this plugin with other test runner plugins such as serverless-mocha-plugin. This will make it easier to run integration tests (including in your CI/CD systems) before deploying. Simply run the following when invoking your tests:

This plugin extends the invoke test command, so any test runner plugin that uses that format should work correctly. All plugin options should remain available.

Available Functions

This plugin currently supports the !Ref function that returns the PhysicalResourceId from CloudFormation. For most resources, this is the value you will need to interact with the corresponding service in the AWS SDK (e.g. QueueUrl for SQS, TopicArn for SNS, etc.).

There is also initial (and limited) support for using !GetAtt to retrieve an ARN. For example, you may use !GetAtt myQueue.Arn to retrieve the ARN for myQueue. The plugin generates the ARN based on the service type. For supported types, it will return a properly formatted ARN. For others, it will replace the value with “FUNCTION NOT SUPPORTED”. In most cases, it should be possible to support generating an ARN for a resource, but the format will need to be added to the plugin.

Where do we go from here?

This initial version of the plugin solves a big problem that I have when developing locally, but there is plenty more that could be done. Feedback, bug reports, feature requests, and project contributions are always welcome and greatly appreciated. I hope this plugin helps make your local serverless development experience better.

NPM: npmjs.com/package/serverless-cloudside-plugin
GitHub: github.com/jeremydaly/serverless-cloudside-plugin

Tags: , , , , ,


Did you like this post? 👍  Do you want more? 🙌  Follow me on Twitter or check out some of the projects I’m working on.

6 thoughts on “Developing Serverless Applications Locally with the “serverless-cloudside-plugin””

  1. What are your thoughts on avoiding local dev entirely? I fluctuate between liking the convenience and hating it – usually when I get hit by a local environment quirk that sets me back a few hours, which ruins the times saving of fast iterations with offline vs sls deploy’ing each change.

    I lean towards high observability (eg with honeycomb) and dev’ing in cloud as soon as possible.

    1. Hi Steve,

      I’m sort of with you on that. I really like developing applications locally because of the speed of the feedback loop. However, once you start wiring up different cloud services, local development can get tricky. I often build a number of mocks to support the remote calls, and then test my code using a test runner (as opposed to something like sls invoke local -f someFunction -p event.json). This obviously takes more time to set up, but I also end up with a much more solid application.

      Integration testing an individual function from your local machine or a CI/CD system is also (somewhat) trivial if you generate the correct resource identifiers (SQS QueueUrl, for example) which is one of the main reason behind this plugin. However, once you have multiple functions working in concert, either as a choreographed workflow with queues/SNS Topic/DynamoDB streams/etc., or orchestrated as Step Functions, the local testing piece doesn’t really help. Having an observability tool (at least in my experience) has been the best way to understand what’s actually happening.

      Thanks for the comment,
      Jeremy

  2. Hey Jeremy!

    First, nice plugin! I’m definitely using this one.

    Second, what about Fn::ImportValue support? I split my stack in several serverless.yml files, and specifically one with all the resources that outputs and exports the ones needed by the others serverless.yml, so I reference this resources on the others like: ELASTIC: Fn::ImportValue: ${self:custom.stage}:infra:ServiceEndpoint

    Any plans on adding support for the exported members?

    Also there are any pattern for dealing with multiples serverless.yml that points to the same dns locally? I was using a docker with reverse proxy and running a sls offline command each one in a port and using the reverse proxy on nginx to point to only one port on docker.. but running 10 sls offline simultaneously in a docker shits the performance.. also cannot use asynchronous events like sqs. I tried localstack but with multiple serverless.yml its tricky.

    Thanks!

    1. Hi Pedro,

      If you’re using Fn::ImportValue to pull outputs from other stacks, you can just use the cf: variable reference that the Serverless framework already supports (https://serverless.com/framework/docs/providers/aws/guide/variables/#reference-cloudformation-outputs). If that’s not what you’re looking for, let me know.

      I’ve never tried to run more than a few APIs at once locally, and even then, I had them running on different ports. But I can certainly see a use case for that. I’m wondering if you could create a separate serverless.yml file and have that import your functions/envs/resources/etc. from your other serverless.yml files using JavaScript variable export support (https://serverless.com/framework/docs/providers/aws/guide/variables/#reference-variables-in-javascript-files)? Might be an interesting project that could be highly reusable.

      Hope that helps,
      Jeremy

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.