Since AWS released support for Node v8.10 in Lambda, I was able to refactor Lambda API to use async/await instead of Bluebird promises. The code is not only much cleaner now, but I was able to remove a lot of unnecessary overhead as well. As part of the refactoring, I decided to use AWS-SDK’s native promise implementation by appending .promise() to the end of an S3 getObject call. This works perfectly in production and the code is super compact and simple:
|
1 |
let data = await S3.getObject(params).promise() |
The issue came with stubbing the call using Sinon.js. With the old promise method, I was using promisifyAll() to wrap new AWS.S3() and then stubbing the getObjectAsync method. If you’re not familiar with stubbing AWS services, read my post: How To: Stub AWS Services in Lambda Functions using Serverless, Sinon.JS and Promises.
The old way looked like this (condensed for readability):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
const AWS = require('aws-sdk') // AWS SDK const Promise = require('bluebird') // Promise library const sinon = require('sinon') // Sinon.js library // Promisify S3 const S3 = Promise.promisifyAll(new AWS.S3()) // Stub the 'async' version let stub = sinon.stub(S3,'getObjectAsync') stub.withArgs({Bucket: 'my-test-bucket', Key: 'test'}).resolves({ AcceptRanges: 'bytes', LastModified: new Date('2018-04-25T13:32:58.000Z'), ContentLength: 23, ETag: '"ae771fbbba6a74eeeb77754355831713"', ContentType: 'text/plain', Metadata: {}, Body: Buffer.from('Test file\n') }) S3.getObjectAsync({Bucket: 'my-test-bucket', Key: 'test'}).then(data => { // data from stub }) |
Any test calls to S3.getObjectAsync that use the specified arguments resolves a promise and returns the data. Getting rid of promises seemed straightforward:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
const AWS = require('aws-sdk') // AWS SDK const sinon = require('sinon') // Sinon.js library // Init S3 const S3 = new AWS.S3() // Stub getObject let stub = sinon.stub(S3,'getObject') stub.withArgs({Bucket: 'my-test-bucket', Key: 'test'}).resolves({ AcceptRanges: 'bytes', LastModified: new Date('2018-04-25T13:32:58.000Z'), ContentLength: 23, ETag: '"ae771fbbba6a74eeeb77754355831713"', ContentType: 'text/plain', Metadata: {}, Body: Buffer.from('Test file\n') }) let data = await S3.getObject({Bucket: 'my-test-bucket', Key: 'test'}).promise() |
But then I obviously got the error: “S3.getObject(…).promise is not a function.”
As I always do when I encounter something like this, I Googled it. I found a GitHub issue in the aws-sdk-js repo that sort of gave a solution: https://github.com/aws/aws-sdk-js/issues/1973. The problem is that stubbing (or mocking) services should be entirely contained within your tests. You should avoid adding anything to your production code, which this solution would have required.
The Answer
After a bit of experimentation I finally realized that Sinon.js completely rewires stubbed functions, essentially eliminating any underlying prototype methods. In this case, the .promise() method no longer existed. Lucky for us, we can return anything we want from Sinon.js stubs, including (🥁 drumroll please) functions!
If we examine how our code is calling the getObject method, we see that .promise() is chained to it. This means that getObject has to return a promise method in order for the code to execute.
|
1 |
let data = await S3.getObject(params).promise() |
We can return an object from our stub (rather than resolve it) using the .returns() method provided with stubs:
|
1 2 3 |
stub.withArgs({Bucket: 'my-test-bucket', Key: 'test/test.txt'}).returns({ promise: () => {} }) |
Now getObject returns a promise() method and the code works. The only problem is that our stub isn’t returning any data. This is easily fixed by returning the data from inside the promise() method:
|
1 2 3 4 5 6 7 8 9 10 11 |
stub.withArgs({Bucket: 'my-test-bucket', Key: 'test/test.txt'}).returns({ promise: () => { return { AcceptRanges: 'bytes', LastModified: new Date('2018-04-25T13:32:58.000Z'), ContentLength: 23, ETag: '"ae771fbbba6a74eeeb77754355831713"', ContentType: 'text/plain', Metadata: {}, Body: Buffer.from('Test file\n') }} }) |
Voilà! Now your stub works correctly and we didn’t have to alter any of our production code to make it work.
You might have noticed that this “promise” doesn’t actually return a promise. For our implementation with await it doesn’t matter, because it will assume a value as a resolved promise. However, if you needed it to return a promise (like if you were still using promise chains), you could simply wrap your data in a Promise.resolve() and that should work.
Tags: aws, aws lambda, aws-sdk, nodejs, programming, sinon.js
Did you like this post? 👍 Do you want more? 🙌 Follow me on Twitter or check out some of the projects I’m working on.
Thanks a lot, this whole series on stubbing AWS methods was really useful!
Hi Jeremy,
Thanks for the great blog series. Learning a lot from reading it. Had a question for you. Why use bluebird when we have native support for promises from node 6 ?
Hi Gaurav,
Great question. I typically don’t use Bluebird anymore since Lambda now supports Node 8.10 and we can use
async/awaitinstead. However, in the examples above, I used Bluebird because of itspromisifyAll()function. This will take a library and (assuming it follows the standard(error, response)callback style) create duplicate methods that return promises instead of callbacks. In the case above where we didconst S3 = Promise.promisifyAll(new AWS.S3()), we now get a method calledgetObjectAsync()(in addition to the standardgetObjectmethod) that will return a promise rather than using a standard callback.Otherwise we’d have to write our own functions that wrapped the asynchronous calls in a promise. Bluebird does that automatically for us. However, if you were just using promises and didn’t need to promisify another library, native promises would be just fine. No need for the extra dependency.
Hope that helps,
Jeremy
I think you can just use resolves instead of returns
stub.withArgs({Bucket: 'my-test-bucket', Key: 'test/test.txt'}).resolves({
AcceptRanges: 'bytes',
LastModified: new Date('2018-04-25T13:32:58.000Z'),
ContentLength: 23,
ETag: '"ae771fbbba6a74eeeb77754355831713"',
ContentType: 'text/plain',
Metadata: {},
Body: Buffer.from('Test file\n')
});
Hi Andrew,
Not if you are using the
.promise()method in your calls. The stub won’t resolve that, which is why you need toreturnan object with apromisemethod.– Jeremy
Hi Jeremy,
Thank you for the great info! I have tried your code out in the past and worked with flying colors. However, I am trying it out again but I am for some reason I get: “InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.” Do you know if there have been any changes recently that may be causing this?
Hi Kelvin,
It sounds like your stubs are not catching the call to AWS. Feel free to drop a sample of your code in so I can review.
– Jeremy
Thanks for your help. Great article, I was using chained promised after await method and was unable to figure out where was I lacking. The solution was is the last paragraph of the article!
Glad it helped!
Could you provide a quick example of stubbing out the event hooks? I’m doing some extra work in the “.on(‘success’)” hook followed by .promise(). I’m trying to do some work using the complete response object that has the request Id in it, but I am having a heck of a time building a simon stub to fake a request ID.
Hi ,
I tried with your example , but somehow the mock object is not getting called the results are being fetched from S3 . , can you help please?
Can you post your code? You need to mock the AWS object after you’ve initialized it, otherwise it gets overwritten.
Good post!
This is how I usually do:
const getObjectsStub = {
Body: Buffer.from(JSON.stringify({ data: jsonAuditlog }))
};
sut.s3.getObject = sinon.stub().returns({ promise: () => getObjectsStub });
There is also a
promisfityin Node 8 (https://nodejs.org/dist/latest-v8.x/docs/api/util.html#util_util_promisify_original) that can be used for one single method.You can try this:
promise: async () => ({...props here...})(async will force the return value to be wrapped in a Promise for free!)
I tried it out, where i have the exact same scenario(instead i am stubbing headObject), however its throwing me:
Type ‘() => Promise’ is not assignable to type ‘() => Promise<PromiseResult>’
I’m facing the same issue.
Hi Jeremy – Any suggestion for below snip to implements sinon. Appropriated your response.
//https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Request.html
function getObject(bucket, s3Key, callback) {
var s3Params = {
Bucket: bucket,
Key: s3Key
};
var request = s3.getObject(s3Params);
request.on(‘complete’, function (response) {
//TODO
});
request.send((err, data) => {
if (err) {
console.log(“Error while downloaing file ” + err);
callback(err);
} else {
let filePath = “/tmp/file_” + new Date().getTime() + “.txt”;
fs.writeFileSync(filePath, data.Body);
console.log(“File has been downloaded :” + filePath);
callback(null, filePath);
}
});
}
I tried using your code, but its calling AWS instead of using the stub; Not sure what am I missing here, appreciate your help here. Below is my code and test js.
app.js
—————————
let AWS = require(‘aws-sdk’);
const _ = require(‘lodash’)
let s3 = new AWS.S3();
let response;
exports.fetchScriptFromS3 = async function (bucket, objName) {
console.log(“Fetching script from S3”, s3);
let params = {
Bucket: bucket,
Key: objName
}
console.log(‘Params: ‘, params);
let data = await s3.getObject(params).promise();
console.log(“Done”);
return data;
}
testhandler.js
——————————————-
let app = require(‘../../app.js’);
const chai = require(‘chai’);
const sinon = require(‘sinon’);
const script = require(‘./simple.json’)
const expect = chai.expect;
describe(‘app Handler Test()’, function () {
afterEach(function () {
sinon.restore();
})
it(‘fetchScriptFromS3 – returns successful repsonse’, async function (done) {
let bucketName = ‘bucketName’;
let objName = ‘objName’;
var s3 = {
getObject: {}
}
var AWS = {
S3: function () {
return s3;
}
}
app.AWS = AWS
//var s3 = new AWS.S3();
var stub = sinon.stub(s3, ‘getObject’);
stub.withArgs({Bucket: bucketName, Key: objName}).returns({
promise: () => { return {
AcceptRanges: ‘bytes’,
LastModified: new Date(‘2018-04-25T13:32:58.000Z’),
ContentLength: 23,
ETag: ‘”ae771fbbba6a74eeeb77754355831713″‘,
ContentType: ‘text/plain’,
Metadata: {},
Body: script
}}
})
console.log(‘S3 : ‘,app.s3);
const result = await app.fetchScriptFromS3(bucketName, objName);
expect(result).to.be.an(‘object’);
expect(result.Body).to.be.an(‘string’);
done();
});
});
Dude this is awesome! I was banging my head on my desk trying to figure out how to properly stub this sdk with the Promise. You rock, thank you!