If you code Batch in Go for example, the main method has it right there in the return value: void. Meaning… there is no return value.

How, then, do you use something like a noop in your mostly Functional Programming style Step Function that assumes you’re composing infrastructure like you’d compose functions? Basically tap. If you’re not familiar, tap is a quick way to debug pure function chains.

If you’ve ever heard of the identity function, it basically gives you back what you give it, like this in Python:

def identity(oh_yeah):
  return oh_yeah

Or this in JavaScript:

const identity = ohYeah =>
  ohYeah

That seems like a ridiculous function, no doubt. But watch what happens when you combine it with composed functions, say, parsing some JSON.

def parse(str):
  return json_parse(str) \
  >> parse_people \
  >> filter_humans \
  >> format_names

Parse composes 4 functions together. If they’re pure, how do you “see” inside that pipeline? You “tap” into it. We can take our identity function, and convert to a tap function like this:

def tap(oh_yeah):
  print("oh_yeah:", oh_yeah)
  return oh_yeah

Then we can “tap into” the pipeline.

def parse(str):
  return json_parse(str) \
  >> tap \
  >> parse_people \
  >> tap \
  >> filter_humans \
  >> tap \
  >> format_names

Sick, ya? Now the normal way to do that in Step Functions is via a Pass state:

"Tap": {
  "Type": "Pass",
  "Next": "NextStep"
}

You can then see the output in the Step Function console, or tweak the inputs/outputs if you wish.

… but that’s for Lambdas, Step Functions, SQS, and SNS steps that not only return a result like a pure function, but do so using your JSON. In JavaScript, that’s pretty easy to merge properties:

const addYoSelf = json =>
  ({ …json, name: "Jesse was here" })

Batch, and even Step Functions, don’t always work like this. You typically have to define Parameters manually depending upon the input… meaning you lose your JSON tree.

… UNLESS it’s just 1 branch. You can create siblings that are collected in a single spot using the Parallel task.

If you think Promise.all like in JavaScript or gather in Python, multiple results, but a single, Array filled result. This allows you to keep context at least in 1 spot using a Pass, and you don’t care about the rest.

a = stuff
identity = stuff => stuff
addSelf = stuff => ({… stuff, name: 'Jesse'})
destroy = stuff => undefined
end = a => Promise.all([identity(a), addSelf(a), destroy(a)])

Notice how end will result in an Array like:
[stuff, stuff + name Jesse, undefined]

Undefined is often what we get with noops that return no value. BUT, we need the first part to continue our Step Function with “context”. Using a Parallel + Pass, you can keep that context.

Until Batch gets the .waitForTaskToken feature (I mean… ECS has it…), you can wrap her in a Parallel to keep context. Just watch those catches….

tl;dr; JSON use Parallel and in next state go:

"OutputPath": "$[0]"

BTW, I know you can use ResultPath and just attach to your existing JSON, but AWS Batch is HUGE, and we’re already getting super close to our JSON size limit with our immense Array size I refuse to read from S3. You could just export and then rip off from a Pass state, but you’ll get a size exception before that happens sadly. Twice now I’ve got an exception when it completes the Batch because the JSON is too large, and using this technique has worked every time.

And Step Function JSON for above (supply your own Lambda):

{
  "Comment": "A Hello World example of the Amazon States Language using Pass states",
  "StartAt": "Both",
  "States": {
    "Both": {
      "Type": "Parallel",
      "Next": "Keep JSON",
      "Branches": [
        {
          "StartAt": "Good Function Returns Value",
          "States": {
            "Good Function Returns Value": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:us-east-1:0000000:function:jesse-identity",
              "Parameters": {
                "uno": "yup"
              },
              "ResultPath": "$.datLambdaResult",
              "End": true
            }
          }
        },
        {
          "StartAt": "Bad Noop",
          "States": {
            "Bad Noop": {
              "Type": "Pass",
              "Result": "World",
              "End": true
            }
          }
        }
      ]
    },
    "Keep JSON": {
      "Type": "Pass",
      "OutputPath": "$[0]",
      "End": true
    }
  }
}

Friends have asked that I report a from the trenches update on my thoughts on Lambdas. I fell in love about a year ago, so how do I feel now? I’ll cover the pro’s and con’s I faced, what I learned, and what I still don’t know.

Why

If you read my previous article on what Lambdas are, and how you build an automation workflow around them, you’ll see why I like them. In short, a front end developer with a bit of back-end Node experience can use continuous deployment for working REST services (or non) quickly, without worry of maintenance, or sudden spikes in traffic. I’m a coder, and with the AWS Node and Python SDK , I can code my way onto the cloud vs. learning Unix, various weirdness with Terraform/Chef/Ansible/Docker, etc.

I have Amazon manage my infrastructure so I can focus on coding. I love it.

Different Environments

If you utilize multiple AWS accounts and/or environments, the gotchas are basically your permissions. Specifically, IAM roles and what permissions those IAM roles have. If you follow the whole dev/qa/staging/prod etc. life cycle of testing your code in each environment and then “moving” to a higher environment once the code passes all testing, you’ll find you do the same thing here. In my experience, you’re just testing two things: Can my Lambda use it’s permissions the same (log to CloudWatch, access a particular S3 bucket, etc) as well as ensuring connectivity that you could of fat fingered somewhere (i.e. the subnet/security group in the Terraform that’s creating your Lambda isn’t correct for the prod environment)? Beyond that, everything seems to operate the same between environments which I loved.

Lesson: As long as you avoid configuration drift, you’re fine.

No API Gateway?

Lambdas are built for burst traffic or the occasional code run for simple services. However, it does NOT require an API Gateway. Meaning, you do not need a URL to have your code run. It truly is a “function”, or a bunch of code. How that code is run doesn’t have to be a URL being put in the web browser.

AWS is flexible. Lambdas are actually functions, not actual “API’s”. Meaning, anything can trigger them; they ARE functions. Files put on an S3 bucket (hard drive), some log messages in CloudWatch that match a string, and even periodic cron jobs managed by AWS. I can personally attest that during the S3 outage last year, when it came back up, not 1 of my messages over a 2 hour period was lost in my CloudWatch cro

More important, though, is that Lambdas can invoke other Lambdas.

Lesson: Even with AWS, you can code yourself out of any negative situation. :: flexes bicep :: Take a look the triggers that AWS provides for Lambdas. You truly can react to a variety of events.

Lambda’s Calling Lambda’s

Lambda’s can invoke other Lambda functions through lambda.invoke. Given the fact that each Lambda (Python or JavaScript) with a reasonable amount of code runs less than 100 to 200 ms is awesome… and doesn’t cost me much since I’m not running a high TPS in my personal projects, this is great.

Another subtle benefit is that you can call them in either a request / response for a more REST type of feel, or a fire and forget for longer running processes. While I haven’t been able to have the need yet, I loved playing with the step functions for when you have a bunch of microservices you’re trying to coordinate in a concurrent fashion.

Errors and Logging

Be aware that some connection errors aren’t verbose. This may be for security reasons. 99% of my “Lambda can’t connect to some thing” all had to do with a misconfigured security group, subnet, or IAM permission. That’s good to know since those things can be easily automated using Terraform or things like the SDK.

For logging, while Lambdas typically log to CloudWatch by default, make sure you log to a single log stream so you can utilize the console or SDK to more easily find logs in a time window. Searching various log streams with dates in the titles over time to find a particular log is harder if it’s not in a the same stream name.

Time Limit

Years ago, Lambdas were limited to 1 minute. Then they got 5 minutes. 5 minutes is still not enough once you have so much power at your finger tips, heh.

If you run into scenarios where 5 minutes isn’t enough, you have 2 options:

1. Use step functions. This can be challenging for some developers because it inverses the responsibility of those doing the work to poll for jobs vs. you being responsible to tell them. This helps solve concurrency issues, too.
2. Use an audit log + CloudWatch cron job. Either use a NoSQL database like Dynamo or an S3 bucket folder. Your Lambda’s will do their work, then audit that they “started and stopped” with a UUID. This could be a timestamped data entry in Dynamo, or a file in S3 bucket. You then have that long running process make an entry with the same UUID/correlation ID in the same audit log (Dynamo or S3). Finally, you have a CloudWatch cron job launch a Lambda every minute or so to check on the status. When the Lambda finds a stopped job audit entry that matches a start, it can mark the job as successful, error’d, or timed out based on your criteria. This CloudWatch cron jobs are beast, yes, they can be trusted.

Deployment

Deployment is huge contention amongst many developers. I’m a huge proponent of automating everything with the AWS SDK. Others like to use Terrform/Chef/Ansible, etc. Others like to upload ZIP files through the console.

Whatever method you do, I still encourage the use of existing integration testing solutions. AWS’s update process is super easy; you’re literally uploading code to the Lambda directly or an S3 bucket, and “poof”, you’re Lambda is updated. You can use the environment & versioning if you wish. I personally didn’t like that, though, and instead liked using a green/blue deployment model where you have 2 Lambdas with a blue/green suffix in the name. When you upload new code, you simply switch the trigger as the last step once all your integration tests have been run. If you screw up, no big deal, it’s near instant to switch it back without uploading new code.

Things like ping and health checks are still extremely important, as are dry runs. Ping’s are simply a way of ensuring your Lambda function is there. If it’s API Gateway, that’s a URL you can hit to ensure it works. If it’s an S3 bucket trigger, you can drop a particular filename in a particular place, and your Lambda can respond either there or in CloudWatch. For health checks, this is the most important; it ensures your Lambda can talk to all the required services it needs and report back. This goes hand in hand with a dry run to ensure it can talk to those things, but doesn’t affect them; like making accidental database entries. Instead, it’s just ensuring your connectivity (i.e. IAM role permissions, subnet/security groups, VPC’s) are all correct.

Where Do We Go From Here?

A guy reached out to me on Slack asking for help in Python. He’s never coded in his life, about my age (38), and manages databases that are recently moved from an on-prem situation to his AWS. He wanted to basically have an alarm (an AWS log event) trigger a Lambda, and if it’s cause was the database was running out of hard drive, to allocate 50 megs more. I gave him a crash course in Python and what I had learned of AWS and Lambdas, and he was up and running in about a month in prod with a 30 line Python Lambda.

That’s so awesome.

The people using AWS Batch and other short lived EC2’s for minutes to seconds worth of work are moving some of that to Lambdas for price reasons. Various Lambdas are in the background, facilitating various logging, monitoring, or reactionary/reactive roles to help functionality in Ops or for developers. It’s become “just another tool in the toolbox”, not some new, shiny thing to not be trusted by the cynics.

We’re still at the beginning of serverless. It is growing, and more and more people from a variety of skillets want to use them. That’s telling. If you’re not doing a lot of server-side development and just frontend websites, I encourage you to take a look at hosting your files on S3 and simply putting CloudWatch in front of it. AWS has a good wizard for this.

In the old days, you’d write code and allow another team called Operations (or OPs for short) to deploy it to various servers for testing, and eventually production. Quality Assurance teams would be testing your code from a few days to a few weeks ago on another server.

Developer tooling, infrastructure as a service, and shorter development cycles have changed all that. The Amazon practice of “you build it, you own it” has started to filter out to other companies as an adopted practice. Teams are now expected to build, deploy, and maintain their own software.

Today, I wanted to cover what I’ve learned about automated deployments around AWS. You’ll learn why you don’t need Ansible, Chef, or even Serverless, and instead can use AWS API’s to do everything you need.

Code

The source code for this article is on Github.

Contents

• Infrastructure As Code
• Infrastructure As An API
• Destroy, Build, and Test Your Lambda
• Step 0: Authenticate
• Step 1: Code Setup
• Step 2: Mad Creds
• Step 3: TDD
• Step 4: List Lambda Functions
• Step 5: Create Lambda Function
• Step 6: Delete Your Lambda Function
• Step 7: Making Testable Code by Testing It
• Step 8: Build a Command-Line Tool
• Step 9: Multiple Environments & Green Blue Deployment
• Step 10: Integration Test
• Conclusions

Infrastructure As Code

Servers As Pets

In the old days, you treated servers as a part of the family. You named them, constantly monitored their health, and were extremely careful when updating code or applying security patches to them.

As time went on, hardware got cheaper, and green blue deployments came about. You still had physical servers at your place of work or nearby, but you had 2 instead of 1 production boxes that you switched between.

Disposable Infrastructure

The cloud being cheap changed all that. Eventually servers became so cheap and easy to make, you just destroyed and created new ones at the first sign of trouble.

Infrastructure == Code

The Infrastructure As Code movement, in short, is that all your servers and API’s and builds… are code. Not JSON, not YAML, not even XML (lol, wtf). If you want to update a server, deploy new code, or update a library version, you simply change the code and check into version control.

As a coder, the reasons I like this are:

1. I’m a coder, I write code, and that makes it something I can more easily understand.
2. Source code is put into version control. Our releases are the same thing as our code versioning.
3. Version control has a record of all changes.
4. Team can peer review server/deployment changes increasing quality.
5. I can unit & integration test my infrastructure code more easily because why? It’s code.

Infrastructure as an API

You can build and destroy a lot of things using the AWS console, the web site you use to access your AWS account. However, everything on AWS is an API. While Lambdas are only JavaScript, Java, C#, and Python, you have more language choices, both official and non, to manage your AWS infrastructure such as Shell and Haskell. We automate these processes using code around these API’s AWS gives us.

As a JavaScript Developer, it’s perfect for me because my local Node development environment is in JavaScript, my front-end Angular is written in JavaScript, my REST API’s are written in JavaScript, and my infrastructure deployment code is now written in JavaScript. Some of the Jenkins job creation/deletion/staring is in JavaScript. Daddy has Thor’s hammer Mjölnir, some FullStack gauntlets of power, and everything in software is suddenly an orcish nail.

Immutable Infrastructure

When disposable infrastructure came into fashion, there was this concept of destroy and build. Rather than debug and nurse a particular server back to health, it was faster and more predictable to just destroy it and build a new one. As this process matured, it eventually became known as “immutable infrastructure”.

Just like how nowadays you try to avoid mutable state in your code, you treat your servers and API’s the same way. Since they are now a result of your code creating it, this is easier.

Mutable State Lives On

However, there are some statefull things you can’t really remove. URL’s for one thing. While API Gateway will give a brand new, unique URL each time you create a REST API, the URL to access your website doesn’t change. Mine is jessewarden.com. If I were to destroy my website’s server and code to update it to the latest code on a new server, I’d still want to use the same URL.

In programming, we use constants for these types of things, and that’s no different here. For statefull things that do not change, or perhaps are updated later, you’ll create variables for these.

Don’t Update. Instead Destroy, Then Create a New One

Everything else, assume it’s immutable. If you update your Lambda code in Gitub, don’t update it in AWS manually. Destroy and then create a new one with the latest code which takes seconds.

Treat your infrastructure like you would your state in Redux:

const newLambda = Object.assign({}, currentLambda, {updated: latestGitCommit});

Destroy, Build, and Test Your Lambda

While larger applications are made up of many API’s, databases, storages, and front ends, we’ll pick a small piece that should be able to be built & deployed independently. Let’s destroy, build, and test our newly built Lambda. Afterwards we’ll update some code in our Lambda to show you how you update over time.

Before we start, you should know that AWS Lambda has a nice alias and versioning functionality. I don’t use any of it and I don’t consider it useful. Same with API Gateway stage deployments. Once you start automating things and synching with your Git Feature Branch or Gitflow workflow, the Lambdas, API Gateways, and Buckets you create are then result of your code versioning. I’m only 3 months in so perhaps I’ll change my mind next year.

Versions are custom names such as “v1.0.0”, while “$LATEST” which you can’t delete is basically equivalent to Git’s “master”. They can be treated like Git tags. Aliases are names such as “dev”, “qa”, and “prod”. They don’t have to be environment based, but it helps because you can say “dev is pointing to $LATEST” and “qa is pointing to git tag 1.2.7”.

Also note, none of this has any rollback functionality. If you screw up, that should be considered OK. The whole point of using code to destroy and build is so you can have confidence that “rolling back” is simply switching to a stable Git tag and rebuilding your stack from scratch which takes seconds to minutes.

Easier said then done. For 2 weeks my API Gateway destroy functions were destroying the first api gateway it found that matched a substring. Whoops. #WithGreatPowerComesGreatResponsibility

Crash Course Into AWS Terminology

If you already know these, skip to Step 0.

AWS: Amazon Web Services. Amazon provides hard drives in the cloud, virtual servers, and all kinds of other amazon tools to build applications in the cloud.

AWS Console: The website you use to access your AWS account and build, destroy, and configure your various AWS services.

Lambda: A function that Amazon hosts and runs for you. Hot serverless action.

Environment: Arbitrary name given to a “different server”. Typically “dev”, “qa”, “staging” and “prod”. Ensures code can be moved between servers and not break.

Region: AWS hosts data centers in physical locations throughout the world. Each has a name such as “us-east-1” for my neck of the woods and “ap-southeast-2” for Sydney, Australia. You can have the same code and files run on those regions to be faster for your users.

API Gateway: Create URL’s that are typically used for REST API’s, but you can create them for mocks for simpler integration testing, or simple proxies to existing REST API’s.

ARN: Amazon Resource Names. Everything in AWS has a unique ARN. They’re like URL’s for websites. They look intimidating at first, but creating them yourself becomes pretty trivial once you learn the format.

Step 0: Authenticate

Good news about Step 0 is you only have to do this crap once.

Although we’ll be using Node for this tutorial, you’ll need your credentials to authenticate with your AWS account (if you don’t have one, go sign up, button on the top right). For security reasons, you should create an IAM user that doesn’t have access to all the things. However, to keep this tutorial simple, we’ll use a core credentials.

In the AWS console, click your name, choose security credentials. Choose Access Keys, then click the “Create New Access Key” button.

In the popup, click the tincy blue arrow link to “Show Access Key”.

Create a new JSON file called “credentials.json” on your computer and paste in the following code. If you’re using source control, ensure you git ignore the credentials.json file so it’s not checked into source control, yet can still remain in your code project.

{
    "accessKeyId": "your key here", 
    "secretAccessKey": "your secret access key here", 
    "region": "us-east-1"
}

Step 1: Code Setup

Make a folder, open your terminal, cd to the directory, run npm init and hit enter until it stops asking you questions, and finally run npm install aws-sdk --save. I’m using Node v7.3.0, but you can use whatever. Use nvm to switch to different versions at will.

Create a new JavaScript file called “index.js” and put in the following code. This is our basic lambda function that just returns a hello world message.

exports.handler = (event, context, callback) =>
{
    const response = {
        statusCode: '200',
        body: JSON.stringify({result: true, data: 'Hello from Lambda'}),
        headers: {
            'Content-Type': 'application/json',
        },
    }
    callback(null, response);
};

Step 2: Mad Creds

Create a new JavaScript file called “build.js”, and put in the following code.

const log    = console.log;
const AWS    = require('aws-sdk');
AWS.config.loadFromPath('./credentials.json');

That 3rd statement is synchronous and must appear before you instantiate any other AWS objects/classes, else you’ll get weird authentication errors.

Step 3: TDD

Note the below are unit tests, not integration tests. Integration tests are tricky not to cause leaky state, so we’ll start with unit tests first. We’ll be using Mocha and Chai, but if you are a Tape or Jasmine fan, feel free to use those instead. Run npm install mocha chai lodash --save. Open your package.json, and change the scripts section to look like this:

"scripts": {
  "test": "mocha build.test.js"
},

Create a new file called “build.test.js”, and put in the following code:

const expect = require("chai").expect;
const should = require('chai').should();
const _ = require('lodash');
const {
    listFunctions
} = require('./build');

Chai has various assertions, “should” reads better than “expect”, and we’ll be using Lodash for creating our own predicates. To avoid using Sinon, we’ll be making our functions as pure as possible. To do that, we’ll need a simple mock for the AWS Lambda client:

const mockLamba = {
    listFunctions: (params, cb) => cb(undefined, {"Functions": []})
};

Our first test is to read a list of functions:

describe('#listFunctions', ()=>
{
    it('should give a list of functions', (done)=>
    {
        listFunctions(mockLamba, (err, data)=>
        {
            _.isArray(data.Functions).should.be.true;
            done();
        });
    });
});

To run them, in your terminal type npm test and hit enter. You should get an error like “ReferenceError: listFunctions is not defined”.

Step 4: List Lambda Functions

Let’s define it:

const listFunctions = (lambda, callback)=>
{
    var params = {
        MaxItems: 50
    };
    lambda.listFunctions(params, (err, data)=>
    {
        if(err)
        {
            log("lambda::listFunctions error:", err);
            return callback(err);
        }
        callback(undefined, data);
    });
};

The listFunctions will list all the Lambda functions you have in your account. Here’s my personal play one:

Now re-run your test and she should be Corbin Dallas green.

Now let’s manually integration test to verify we can use the API in an authenticated way and our code works. Modify your listFunctions so we can see the goodies that come back via a logging of the data:

lambda.listFunctions(params, (err, data)=>
    {
        if(err)
        {
            log("lambda::listFunctions error:", err);
            return callback(err);
        }
        log("lambda::listFunctions, data:", data);
        callback(undefined, data);
    });

Run node index.js and you should see some JSON in your Terminal.

Let’s add 1 more test for a negative scenario in case we don’t have permissions, bad credentials, etc.

const mockBadLambda = {
    listFunctions: (params, cb) => cb(new Error('boom'))
};
// ...
it('should blow up if something goes wrong', (done)=>
{
    listFunctions(mockBadLambda, (err, data)=>
    {
        err.should.exist;
        done();
    });
});

Re-run and you should have 2 passing tests.

Step 5: Create Lambda Function

Lambda functions require (Jesse Warden’s way) 4 things:

1. A name prefix. You’ll often have at least 2 different functions to test development vs. production servers. So “myapp-dev” vs. “myapp-prod”. The “myapp” is your prefix.
2. An alias. While AWS is implying they’d like you to use dev, qa, and prod, we’ll just use one called “defaultalias”. Sometimes this is referred to as a qualifier.
3. A function name. It’s what you see in the left column of the aws console.
4. The code, heh! I put mine in a zip file called “deploy.zip”. For now we’ll zip up our index.js file into a zip first. It’s easier to do this with shell, but THE HAMMER IS JS AND EVERYTHING IS A NAIL.
5. A role.

Note On Roles (Feel free to skip)

My last article talked about creating a custom role, but we can use the default one Amazon provides called “lambda_basic_execution”. I’ll hardcode it for this tutorial. Note the number in it is your AWS account number. Given in a production environment this would something your team or a OPS/Cloud/Security team would create, we’ll create a variable for it vs. getting it dynamically. Roles are complicated and are the #1 thing to break your code next to security things (VPC, subnets, security groups).

Let’s now create a lambda function. In your build.test.js, first import it:

const {
    listFunctions,
    createFunction
} = require('./build');

Then add a new method to our Lambda mock:

createFunction: (params, cb) => cb(undefined, {"FunctionArn": 'default'})

When you create a new Lambda function, it’ll return a big ole JSON object mirroring mostly your parameters. The FunctionArn is the unique ID/URL of our function, so if we get this, we know she’s legit and it worked. We’ll add a new predicate to test for this String as well as a new test:

const legitString = (o) => _.isString(o) && o.length > 0;
const mockFS = {
 readFileSync: (filename)=> new Buffer('')
};

describe('#createFunction', ()=>
{
    it('should give us a positive response', (done)=>
    {
        createFunction(mockLamba, mockFS, (err, data)=>
        {
            legitString(data.FunctionArn).should.be.true;
            done();
        });
    });
});

Running it should fail:

Let’s make it pass. This’ll require a bit more variables setup first. Add your createFunction to the build.js:

const createFunction = (lambda, fs, callback)=>
{
    var params = {
        Code: {
            ZipFile: fs.readFileSync('deploy.zip')
        },
        FunctionName: FUNCTION_NAME,
        Handler: 'index.handler',
        Role: ROLE,
        Runtime: 'nodejs4.3'
    };
    lambda.createFunction(params, (err, data)=>
    {
        if(err)
        {
            // console.log("err:", err);
            return callback(err);
        }
        // log("data:", data);
        callback(undefined, data);
    });
};

And ensure you’ve exported the function below:

module.exports = {
    listFunctions,
    createFunction
};

No re-run npm test and it should pass:

Let’s add a negative test scenario in case the createFunction fails. Add a createFunction to the mockBadLambda:

createFunction: (params, cb) => cb(new Error('boom'))

Then add the negative test for it:

it('should blow up if something goes wrong', (done)=>
{
    createFunction(mockBadLambda, mockFS, (err, data)=>
    {
        err.should.exist;
        done();
    });
});

Your tests run now should show:

Make and Destroy Deploy ZIP

Note Shell vs. JavaScript: While you can use a pure JavaScript solution using archiver (ensure you use forceZip64 in the constructor), now that PC’s have shell baked in, it’s easier to read, less code to write, and more predictable.

When you create a Lambda function, you need to upload code in the same call. Typically our code is more than just 1 index.js file, it’s many libraries in node_modules, configure files, etc. For file size reasons and ease of portability, we zip it all up into 1 smaller file. If you look at our createFunction you’ll see she reads it into a Node Buffer.

To create it and delete it ourself, we just need to add 2 new script commands to our package.json:

"makezip": "zip deploy.zip index.js",
"deletezip": "rm -f deploy.zip"

Test yourself by running npm run deletezip and npm run makezip.

Give createFunction a Spin

Let’s give her a go. Open up index.js, and at the very bottom, just go ahead and hardcode:

createFunction(lambda, fs);

You should now be able to log into the AWS Console and see your new Lambda function in the list:

Wunderbar! Now, delete your hardcoded createFunction from index.js.

Step 6: Delete Your Lambda Function

Your function is created. If you want to update code in it, you could simple make a new zip file and call updateFunctionCode. However, to make things truly immutable and atomic, meaning each individual aspect of our Lambda we can test and update individually, we’ll just delete the whole thing.

Remember, we’re not treating our server like a nice camera. Instead, we purchase a disposable one at the local drugstore/apothecary, and if it breaks, we get a new one instead of waisting time debugging a $9.25 single use electronic. This is important for deploying specific versions of code. If you deploy a git tag called “2.1.6”, but you later update code, you’ve negated the whole point of using a specific git tag since it’s not really 2.1.6, but your own version. If something goes wrong, you know for sure (mostly) that’s that version of the code and not your modification.

In build.test.js, import the non-existent deleteFunction:

const {
	listFunctions,
	createFunction,
	deleteFunction
} = require('./build');

Add a mock method to our mockLambda:

deleteFunction: (params, cb) => cb(undefined, {})

And a mock method to our mockBadLambda:

deleteFunction: (params, cb) => cb(new Error('boom'))

And finally our 2 tests:

describe('#deleteFunction', ()=>
{
	it('should delete our lambda if there', (done)=>
	{
		deleteFunction(mockLamba, (err)=>
		{
			_.isUndefined(err).should.be.true;
			done();
		});
	});
	it('should not delete our lambda if it', (done)=>
	{
		deleteFunction(mockBadLambda, (err)=>
		{
			err.should.exist;
			done();
		});
	});
});

If our Lambda works, we get no error. The call gives you an empty Object back which is worthless, so we just bank on “no error is working code”. Let’s write the implementation. In build.js, put in the following code above your module.exports:

const deleteFunction = (lambda, callback)=>
{
    var params = {
        FunctionName: FUNCTION_NAME
    };
    lambda.deleteFunction(params, (err, data)=>
    {
        if(err)
        {
            // log("lambda::deleteFunction, error:", err);
            return callback(err);
        }
        // log("lambda::deleteFunction, data:", data);
        callback(undefined, data);
    });
};

And then add to your module.exports:

module.exports = {
    listFunctions,
    createFunction,
    deleteFunction
};

Cool, now re-run npm test:

Let’s give her a spin. Hardcode deleteFunction(lambda, ()=>{}); at the very bottom, run node build.js, then log into the AWS Console for Lambda, and you should no longer see ‘datMicro’ (or whatever you called it) in the left list.

Step 7: Making Testable Code by Testing It

There are a few more steps to go in making our Lambda fully functional with the API Gateway. However, we can at this point test her out in the AWS Console. That means we can test her out in JavaScript, too. Let’s take a look at the original Lambda function code:

exports.handler = (event, context, callback) =>
{
    const response = {
        statusCode: '200',
        body: JSON.stringify({result: true, data: 'Hello from Lambda'}),
        headers: {
            'Content-Type': 'application/json',
        }
    }
    callback(null, response);
};

A few problems with this handler. First, it’s not testable because it doesn’t return anything. Second, it doesn’t really take any inputs of note. Let’s do a few things. We’ll add some unit tests, a function that always returns true, and a random number function.

Always True

Create an index.test.js file, and add this code as a starting template:

const expect = require("chai").expect;
const should = require('chai').should();
const _ = require('lodash');
const {
	alwaysTrue
} = require('./index');

describe('#index', ()=>
{
    describe('#alwaysTrue', ()=>
    {
        it('is always true', ()=>
        {
            alwaysTrue().should.be.true;
        });
    });
});

Modify your package.json to point to this test for now:

"scripts": {
    "test": "mocha index.test.js",
    ...
},

Now run npm test. Hopefully you get something along the lines of:

To make it pass, create the predicate:

const alwaysTrue = ()=> true;

Then export at the very bottom:

module.exports = {
    alwaysTrue
};

Re-run your npm test, and it should be green:

module.exports = {
    alwaysTrue
};

Testing random numbers is hard. For now, we’ll just verify the number is within the range we specified. In index.test.js import the new, non-existent, function:

const {
	alwaysTrue,
    getRandomNumberFromRange
} = require('./index');

And a basic test, as we’re not handling bounds or typing checks for now:

describe('#getRandomNumberFromRange', ()=>
{
    it('should give a number within an expected range', ()=>
    {
        const START = 1;
        const END = 10;
        const result = getRandomNumberFromRange(START, END);
        _.inRange(result, START, END).should.be.true;
    });
});

Re-run your tests and it should fail (or perhaps not even compile).

Now implement the function in index.js:

const getRandomNumberFromRange = (start, end)=>
{
	const range = end - start;
	let result = Math.random() * range;
	result += start;
	return Math.round(result);
};

And export her at the bottom:

module.exports = {
    alwaysTrue,
    getRandomNumberFromRange
};

Re-run your tests and she should be green:

Lastly, let’s rework our main Lambda function to always return a value, respond to a test, and become its own function as we’ll manually add it to the module.exports in a bit. In index.test.js, import the handler:

const {
	alwaysTrue,
    getRandomNumberFromRange,
    handler
} = require('./index');

And write the first test that expects it to return a response. Since we aren’t a typed language, we’ll create a loose one via a couple predicates to determine if it’s “response like”.

const responseLike = (o)=> _.isObjectLike(o) && _.has(o, 'statusCode') && _.has(o, 'body');

And the test:

describe('#handler', ()=>
{
    it('returns a response with basic inputs', ()=>
    {
        const result = handler({}, {}, ()=>{});
        responseLike(result).should.be.true;
    });
});

For now the response is always an HTTP 200. We can add different ones later. Re-run your tests and she should fail:

Now let’s modify the function signature of our handler from:

exports.handler = (event, context, callback) =>

to:

const handler = (event, context, callback) =>

Move her above the module.exports and then add her to the exports. Final Lambda should look like this:

const alwaysTrue = ()=> true;
const getRandomNumberFromRange = (start, end)=>
{
	const range = end - start;
	let result = Math.random() * range;
	result += start;
	return Math.round(result);
};

const handler = (event, context, callback) =>
{
    const response = {
        statusCode: '200',
        body: JSON.stringify({result: true, data: 'Hello from Lambda'}),
        headers: {
            'Content-Type': 'application/json',
        }
    }
    callback(null, response);
};

module.exports = {
    alwaysTrue,
    getRandomNumberFromRange,
    handler
};

Our Lambda will return random numbers in the response based on the range you give it. We’ll have to create some predicates to ensure we actually get numbers, they are within range, and then return error messages appropriately. Our response in our handler will start to be different based on if someone passes in good numbers, bad numbers, bad data, or if it’s just a test. So, we’ll need to make him dynamic. Finally, we’ll add a flag in the event to make integration testing easier.

First, the litany of predicates for input checking. You’ll need 2 helper functions to make this easier. Create a new JavaScript file called predicates.js, and put this code into it:

const _ = require('lodash');

const validator = (errorCode, method)=>
{
	const valid = function(args)
	{
		return method.apply(method, arguments);
	};
	valid.errorCode = errorCode;
	return valid;
}

const checker = ()=>
{
	const validators = _.toArray(arguments);
	return (something)=>
	{
		return _.reduce(validators, (errors, checkerFunction)=>
		{
			if(checkerFunction(something))
			{
				return errors;
			}
			else
			{
				return _.chain(errors).push(checkerFunction.errorCode).value();
			}
		}, [])
	};
};

module.exports = {
	validator,
	checker
};

Now, let’s test the new, (soon to be) parameter checked handler in a few situations. At the top of index.test.js, import the handler function:

const {
	alwaysTrue,
    getRandomNumberFromRange,
    handler
} = require('./index');

Let’s add a new, more brutal negative test where we pass nothing:

it('passing nothing is ok', ()=>
{
    const result = handler();
    responseLike(result).should.be.true;
});

Let’s look at the responses and ensure we’re failing because of bad parameters, specifically, a malformed event. One test for a good event, one for a missing end, and one for our echo statement. Since the response is encoded JSON, we create a predicate to parse it out and check the result:

const responseSucceeded = (o)=>
{
    try
    {
        const body = JSON.parse(o.body);
        return body.result === true;
    }
    catch(err)
    {
        return false;
    }
};
// ...
it('succeeds if event has a start and end', ()=>
{
    const response = handler({start: 1, end: 10}, {}, ()=>{});
    responseSucceeded(response).should.be.true;
});
it('fails if event only has start', ()=>
{
    const response = handler({start: 1}, {}, ()=>{});
    responseSucceeded(response).should.be.false;
});
it('succeeds if event only has echo to true', ()=>
{
    const response = handler({echo: true}, {}, ()=>{});
    responseSucceeded(response).should.be.true;
});

None of those will pass. Let’s make ’em pass. Open index.js, and put in the predicates first. Import her up at the top:

// Note: the below only works in newer Node, 
// not the 4.x version AWS uses
// const { validator, checker } = require('./predicates');
const predicates = require('./predicates');
const validator = predicates.validator;
const checker = predicates.checker;

Then below put your predicate helpers:

// predicate helpers
const eventHasStartAndEnd = (o) => _.has(o, 'start') && _.has(o, 'end');
const eventHasTestEcho    = (o) => _.get(o, 'echo', false);
const isLegitNumber       = (o) => _.isNumber(o) && _.isNaN(o) === false

These check the event for both a start and end property, or an echo. Lodash _.isNumber counts NaN as a number, even though NaN stands for “Not a Number” and is a Number per the ECMAStandard because “design by committee”. I wrangle the insanity by writing my own predicate that… you know… makes sense: isLegitNumber.

We’ll use them to build our argument predicates:

// argument predicates
const legitEvent = (o)=> 
    _.some([
            eventHasStartAndEnd, 
            eventHasTestEcho
        ],
        (predicate) => predicate(o)
    );
const legitStart = (o) => isLegitNumber(_.get(o, 'start'));
const legitEnd   = (o) => isLegitNumber(_.get(o, 'end'));

Now we have a lot of functions to verify if our event is acceptable. However, if it’s not acceptable, we don’t know why. Worse, users of your Lambda, both you in 2 weeks when you forgot your code, and other API consumers, won’t have any clue what they did either without cracking open the CloudWatch logs + your code and attempting to debug it.

We’ll take this a step further by using those validator and checker functions we imported above. Second, the validators:

// validators
const validObject = validator('Not an Object.', _.isObjectLike);
const validEvent  = validator('Invalid event, missing key properties.', legitEvent);
const validStart  = validator('start is not a valid number.', legitStart);
const validEnd    = validator('end is not a valid number.', legitEnd);

These functions are normal, they just take advantage of JavaScript and just about everything being a dynamic Object. That first parameter, the string error message, you can store on the function, so if it returns false, you know WHY it returned false. The checkers will accumulate those errors using a reduce function. Third, the checkers:

// checkers
const checkEvent       = checker(validObject, validEvent);
const checkStartAndEnd = checker(validStart, validEnd);

2 more predicates and we’re done. All Lambdas are required to have at least 1 response to not blow up. However, you and I know code either works or it doesn’t. There is middle ground, sure, but for simple stuff, it’s black and white. We’ll break those out into 2 predicates for creating our HTTP responses of errors:

const getErrorResponse = (errors)=>
{
    return {
        statusCode: '500',
        body: JSON.stringify({result: false, error: errors.join('\n')}),
        headers: {
            'Content-Type': 'application/json',
        }
    };
};

And success:

const getResponse = (data)=>
{
    return {
        statusCode: '200',
        body: JSON.stringify({result: true, data}),
        headers: {
            'Content-Type': 'application/json',
        }
    }
};

Armed with our predicates, we can have a flexible handler, and if something blows up, we will know why. Let’s break her down into 5 steps:

const handler = (event, context, callback) =>
{
    if(_.isNil(callback) === true)
    {
        return getErrorResponse(['No callback was passed to the handler.']);
    }
    ...

We’re ok with no event and context, but no callback!? That’s crazy talk. Here’s your t3h boom.

const errors = checkEvent(event);
if(errors.length > 0)
{
    callback(new Error(errors.join('\n')));
    return getErrorResponse(errors);
}

If our event isn’t legit (either and echo, or having start and end numbers), we send the array of errors we get back to whoever triggered us in an error callback. Instead of “I didn’t work”, they’ll have a fighting chance of knowing why since we sent them the validation messaging.

Quick Security Note

I should point out AWS walks the line of being secure and not giving you verbose errors while sometimes giving you what they can without compromising security to help you debug as a developer. You’ll note that my checkers tend to be verbose in the hope they’ll help whoever made a mistake. However, as things scale, you must be careful not to expose public information, or reveal too much about what you DON’T validate. I’m not a security guy, I don’t have the answers beyond lots of peer review of code, automated quality checks, and automated security scanning. You’ll note the obvious of not throwing stack traces back to the client. You can see those in CloudWatch if you wish.

if(event.echo === true)
{
    const echoResponse = getResponse('pong');
    callback(undefined, echoResponse);
    return echoResponse;
}

That’s for our future integration tests. It’s easier if our remote code is aware she’ll be pinged to see if she’s alive and well. We test for it to ensure it doesn’t negatively affect others. You can see the work that went into validating it as well as ensuring it played nice with others, yet still supported the ability to be tested without actually doing real work that could lead to leaky state.

const startEndErrors = checkStartAndEnd(event);
if(startEndErrors.length > 0)
{
    callback(new Error(startEndErrors.join('\n')));
    return getErrorResponse(startEndErrors);
}

Finally, we check to ensure if we’re going to do the random number generation, we have what we need from the event to do so, else, blow up and explain why. The real work is the end of the function:

const start        = _.get(event, 'start');
    const end          = _.get(event, 'end');
    const randomNumber = getRandomNumberFromRange(start, end);
    const response     = getResponse(randomNumber);
    callback(undefined, randomNumber);
    return response;
};

Now re-running your tests should result in them all passing:

Manual Test

One last manual test you can do as well is simply run her in the node REPL. In the Terminal, type “node” and hit enter.

Then import your index module by typing lambda = require('./index') and hitting enter:

You’ll see our 3 functions we exposed. AWS only cares about your handler, so let’s manually test ours with some bogus data. Type lambda.handler() and hit enter, and you should see a 500 response:

Now let’s mirror our unit test by using, and give it some good inputs via handler({echo: true}, {}, ()=>{}); to get a basic 200 response:

You can Control + C twice to get out of Node.

Skills. Unit tests work, and a manual test works. Now you can be assured if you upload to AWS and she breaks, it’s them not you. Yes, your problem, but thankfully your self-esteem shall remain intact. Remember, part of programming is deflecting blame to others, backing it up with fancy terms like “I have lots of TDD code coverage”, then fixing “their” problem and looking like a hero.

Deploy Testable Code to AWS To Test There

Speaking of AWS, let’s redeploy and test our more testable code up on AWS. This’ll be a common task you do again and again by testing code locally, then re-deploying to test it on AWS. We’ll suffer through it for a bit so we appreciate the automating of it later.

For now, let’s adjust your makezip script in package.json to add our new files. We have to add predicates.js and our libraries which are in node_modules:

"makezip": "zip -r -X deploy.zip index.js predicates.js node_modules",

We’ll hardcode our build script for now to destroy our stack first, then recreate it with whatever deploy.zip it finds locally. Open up build.js, and at the bottom, let’s chain together our deleteFunction & createFunction:

deleteFunction(lambda, (err, data)=>{
    log("deleteFunction");
    log("err:", err);
    log("data:", data);
    createFunction(lambda, fs, (err, data)=>
    {
        log("createFunction");
        log("err:", err);
        log("data:", data);
    });
});

You may get an error the first time since no function may be up there to delete and that’s ok. We’re creating & fixing one thing at a time. For now, it’s good enough if she creates your zip file and uploads it to your newly created Lambda function. We’re not differentiating between dependencies and development dependencies in node_modules, so your deploy.zip will be quite large, and may take more time to upload now that she’s not just under 1kb of text.

Run npm run deletezip, then npm run makezip, then node build… or just:

npm run deletezip && npm run makezip && node build

Log into your AWS Console and under Services choose Lambda. You should see your function in the list (they’re often sorted by newest up top). Notice she’s 4+ megs, t3h lulz. #myFLAFilesWereBigRaR #backInTheDay

Click it, and let’s test it. You’ll see a big blue button at the top called “Test”. Click it. It should blow up with our custom blow up message:

Hot, let’s see if she correctly responds to our manual integration test. Click “Actions” and “Configure Test Event”. Here, you can basically make up your own event JSON to test your Lambda and it’ll run on AWS infrastructure. Ours is pretty simple, echo true. When done click Save and Test.

Now be careful; sometimes this window has a glitch where it’ll save “echo”: “true” instead of “echo”: true. The “true” String is not the same as the true Boolean that we want. All goes well, you’ll see:

DAT PONG! Last manual test, let’s generate a random number. Again click Actions and Configure Test Event, and replace the fixture with:

{
  "start": 1,
  "end": 10
}

Save and Test…

I got a 5, what’d you get?

Step 8: Build a Command-line Tool

To make our build.js more flexible, we’ll have to create our first command-line program in Node. There are a few command-line parsing libraries out there, but we’ll just create our own predicates so you learn how to do it yourself, this’ll be commonplace in creating command-line tools for yourself, and we won’t have weird test errors (hence the image).

Open build.js, and delete our buildFunction & creatFunction calls at the very bottom. We’re removing that hardcoded; instead of constantly modifying build.js, we’ll make it do what we want when we run it. Next, open up package.json, and let’s include both our tests now vs. switching between index or the build.

"scripts": {
    "test": "mocha *.test.js",

If you’ve never used globs before, that’s what the star is. It says “anything that has a .test.js suffix in the filename”. We have 2 tests, so if you run npm test now, you’ll get:

Green is gorrrrrrrgeous.

Let’s create a stub since this’ll be a constant create, add, test fail, test succeed workflow. Open build.js, and add:

const getProgram = (argsv)=>
{

};

Then at the bottom export him:

module.exports = {
    ...
    getProgram
};

In build.test.js, import our getProgram function:

const {
	...
	getProgram
} = require('./build');

We’re going to give this function our command line parameters and expect him to interpret what we meant.

describe.only('#getProgram', ()=>
{
	it('should not build with no parameters', ()=>
	{
		const program = getProgram();
		expect(program.action).to.equal(ACTION_NOTHING);
	});
});

Now re-run npm test, and she’ll fail properly:

To make it pass, we just need to return a no-op (meaning, no operation, don’t do anything).

const getProgram = (argsv)=>
{
    if(_.isArray(argsv) === false)
    {
        return {action: 'nothing'};
    }
    
};

Re-run tests and it should pass:

There are the 3 things our program should do: nothing, build, and destroy.

const ACTION_NOTHING = 'nothing';
const ACTION_BUILD   = 'build';
const ACTION_DESTROY = 'destroy';

And export ’em out at the bottom of build.js:

module.exports = {
    ...
    ACTION_BUILD,
    ACTION_DESTROY,
    ACTION_NOTHING
};

To know which one, we’ll have to parse process.argsv, an array that holds all the arguments passed when starting a Node program.

The first, the location to the Node binary running it, and second, the build that’s being run, we don’t care about. Everything AFTER is fair game… or there could be dragons there; command-line programs are easy to fat finger (mis-type) so our Array could have some cray text in it.

We’ll strip out those first 2 items, and create some predicates that search for their items. The following predicates will get an Array, and see if -b or –build is in there.

const hasBigBuild   = (array) => _.includes(array, '--build');
const hasSmallBuild = (array) => _.includes(array, '-b');
const hasBuild      = (array) => hasBigBuild(array) || hasSmallBuild(array);

And these for destroy:

const hasBigDestroy   = (array) => _.includes(array, '--destroy');
const hasSmallDestroy = (array) => _.includes(array, '-d');
const hasDestroy      = (array) => hasBigDestroy(array) || hasSmallDestroy(array);

If you have both, we’ll make the assumption now that you want to destroy your stack first, then build a fresh new one. This is a pretty common practice we’ll be doing, so we’ll bake in this assumption now vs. coding error logic for telling the developer they provided competing instructions. So let’s add that instruction to avoid ambiguity, and the predicate to test for it.

const ACTION_DESTROY_AND_BUILD = 'destroy and build';
...
const hasDestroyAndBuild = (array) => hasBuild(array) && hasDestroy(array);
...
module.exports = {
    ...
    ACTION_DESTROY_AND_BUILD
};

Let’s create a new test detecting our build instructions. In build.test.js, import the action up top:

const {
	...
	ACTION_BUILD
} = require('./build');

Then the test:

it('should build if we tell it to do so', ()=>
{
	const parameters = [
		'node is rad',
		'testing bro',
		'--build'
	];
	const program = getProgram(parameters);
	expect(program.action).to.equal(ACTION_BUILD);
});

Run your test and she should boom:

Add this to the bottom of the getProgram function:

if(hasBuild(argsv) === true)
{
    return {action: ACTION_BUILD};
}

Now re-run your tests:

… and the destroy test in build.test.js, import destroy action:

const {
	...
	ACTION_DESTROY
} = require('./build');

And the test:

it('should destroy if we tell it to do so', ()=>
{
	const parameters = [
		'node is rad',
		'testing bro',
		'--destroy'
	];
	const program = getProgram(parameters);
	expect(program.action).to.equal(ACTION_DESTROY);
});

Running it should show the failure:

To make it pass, we need to check for a destroy below (or above) the build:

if(hasDestroy(argsv) === true)
{
    return {action: ACTION_DESTROY};
}

And re-running the tests should show passing:

Finally, let’s test for both. Import the action:

const {
	...
	ACTION_DESTROY_AND_BUILD
} = require('./build');

and copy paste the test:

it('should destroy and build', ()=>
{
	const parameters = [
		'node is rad',
		'testing bro',
		'--destroy',
		'--build'
	];
	const program = getProgram(parameters);
	expect(program.action).to.equal(ACTION_DESTROY_AND_BUILD);
});

To make it pass, let’s add the final piece to our build.js:

const getProgram = (argsv)=>
{
    if(_.isArray(argsv) === false)
    {
        return {action: ACTION_NOTHING};
    }

    if(hasDestroyAndBuild(argsv) === true)
    {
        return {action: ACTION_DESTROY_AND_BUILD};
    }

    if(hasBuild(argsv) === true)
    {
        return {action: ACTION_BUILD};
    }

    if(hasDestroy(argsv) === true)
    {
        return {action: ACTION_DESTROY};
    }

    return {action: ACTION_NOTHING};
};

Re-run your tests:

Now that we can parse command-line parameters, let’s write the code that will perform those actions. No tests for this function, just copy pasta this into build.js:

const performActionIfPassed = (callback)=>
{
    const program = getProgram(process.argv);
    if(program.action === ACTION_BUILD)
    {
        createFunction(lambda, fs, callback);
    }
    else if(program.action === ACTION_DESTROY)
    {
        deleteFunction(lambda, callback);
    }
    else if(program.action === ACTION_DESTROY_AND_BUILD)
    {
        deleteFunction(lambda, (err, data)=>
        {
            createFunction(lambda, fs, callback);
        });
    }
};

Then at the bottom, we’ll use a snippet to identify if we’re running through node build.js then run our build code; if not, our build.js is being required, so do nothing.

if(require.main === module)
{
    performActionIfPassed((err, data)=>
    {
        log("Done.");  
    });
}

Let’s take her for a spin. Open up your Terminal, cd to the code directory, and run node build --build --destroy. If you left your logging uncommented, she may just sit there for a bit as it uploads the zip file (remember, she’s big currently). Also, deleteFunction may throw an error if the function isn’t created, and that’s ok.

My deleteFunction through an error, but that’s ok, the create worked. Now that we have a function, let’s try just a destroy via node build --destroy:

And looking at our Lambda function list again:

Yay! Lastly, let’s just do a single build via node build --build:

Last step in this adventure is to make this all 1 command. Open your package.json, and let’s add a new script:

"scripts": {
    ...
    "deploy": "npm run deletezip && npm run makezip && node build --destroy --build"
  },

Now, anytime you change code and want to re-test it, simply run npm run deploy.

Step 9: Multiple Environments & Green Blue Deployment

Background on Environments

We use multiple environments for a lot of reasons. Here are the 2 most important:

1. Your code can continue serving customers while you work on it.
2. Your code often breaks when it moves to a new environment. You create at least 3 so there is no penalty of breaking production when you move your code from a dev to qa. Once you’ve practiced this a couple times, you can make the hopefully more confident move from qa to prod.

Environments are a higher level term, but usually mean a different server with a different URL or IP address. Even servers that are the exact same hardware, OS, versions, and software still can have code break. Some issues you know about, others are part of the discovery process of moving environments.

There is no standard. The most typical is dev for “developers to play with code”. QA for “quality assurance people to test the most solid code”. Staging as the last environment change check before prod. Prod is production.

Some teams use more. Some use less.

While there is a strong push towards immutable infrastructure, prod still remains special. In immature software organizations, prod is often the only thing users see. There is a trick to ensure prod remains working when you push new code to it called green / blue deployment. You create 2 virtual servers on prod. They are identical with one difference: one has a name of green, and one of blue. Users are using a URL to see the green server. You move code to blue. Once you validate blue is working, you switch the URL to point to blue. This is really fast. Sometimes. This is also really easy to undo if something goes wrong. Sometimes.

Environments for Lambdas?

Given that Lambdas are “always up”, yet you’re only charged for when they actually run, do you really care about environments? What about green / blue deployment?

Sadly, yes.

Security

you’ll still potentially have different security in place for each environment. In AWS’ case, this is VPC (virtual private cloud), subnets, and security groups (things that determine what you can do, what ports are open, etc).

Dev’s Use Dev

You’ll have users and other developers in the case of microservices, actively depending upon certain environments to be accessible and behave a certain way.

Dev’s Use QA for their QA

Microservices additionally are often a chain in a larger application, many they intentionally don’t know about. As such, other teams may use your QA service in their QA environment to mirror how you’re developing software.

No Downtime

Updating a Lambda, while super quick (seconds if just updating the function code without publishing a new version), you’ll still experience downtime if someone’s hitting that server.

Rollback Window

If you update a Lambda and break the code, you increase that break window until your rollback to old code is finished. Lambda versions can potentially help here. (I don’t use them)

Green / Blue with Lambdas

Finally, the easiest way to update production Lambdas is still green/blue. Users can actively be using them, or they can be being used in an activate AWS Step Function process, and you can ensure you don’t cause them downtime. The muddy ground here is “What is the URL?”

For example, not all Lambdas are triggered by an API Gateway URL. While you can change the Lambda function an API Gateway is aimed at by just changing the ARN, things get a tincy bit more complicated with S3 bucket, SNS, and CloudWatch notifications. For example, at the time of this writing S3 buckets only allow 1 bucket notification, per bucket. You can remove that notification and add it back, but if you’re dealing with extremely low latency systems, you can miss Object:put events to the bucket while your infrastructure code is running. There are remedial steps here, no doubt, just recognize that green / blue still is a useful deployment pattern to minimize downtime, regardless of environment.

For use, we’ll use the name, such as “myLambda-green” and “myLambda-blue”. The S3 bucket policy, or API Gateway URL, or whatever, points to that named Lambda.

Where Does Environment Go?

Two places. There are 2 people who care about the environment: You, and the Lambda. You put it as part of the Lambda name so can easily scan the function list in the Lambda console, debug in CloudWatch easier, and the Lambda itself can help log. You’ll end up with many lambdas in a list that have the name “myLambda-dev-green”, “myLambda-dev-blue”, “myLambda-qa-green”, etc.

The Lambda consumes what environment it’s in from environment variables which can be encrypted if you wish. This ensures your code has no state, and just adopts whatever is there, regardless of it’s name. Given many enterprises have separate AWS accounts for both different parts of the organization, and production environments, you can help minimize the pain by making your code stateless and configurable by the environment variables.

Regions

Regions get a bit more complicated, but the short version is you just follow the same pattern of explicitly naming them and use environment variables. Many AWS function require a region parameter, so avoid typing ‘us-east-1’ directly into your code.

So… what do I do again?

Same Lambda, just different name for it to differentiate between dev, qa, and prod as well as green and blue. Your createFunction just has a different string. For example, “myLambda-green” and “myLambda-blue”. If you use environments, you can put those in the name too: “myLambda-qa-green” and “myLambda-qa-blue”.

In our code above, we’ve defined dev without a color, but you could simple add more strings:

const DEFAULT_ENVIRONMENT = 'dev';
const COLOR               = 'green';
const FUNCTION_NAME       = NAME_PREFIX + '-' + DEFAULT_ENVIRONMENT + '-' + COLOR;

Then run node build --build fro each color and environment.

Wait, Why Not Aliases and Versions?

Lambda Alises (dev, qa, and prod, green, blue, staging, live), and the Versions they point to (1, $LATEST, Git tag v2.3.7) are great. They allow for potentially easier management of environments and code versions to be managed in a centralized place if you use the AWS Console heavily. However, configuration and environment variables are global and can only be modified in $LATEST version.

Given I work in a high security environment where I test multiple configurations, and unlike S3 buckets, we currently do not have a cost in creating multiple Lambdas, it’s more flexible for the workflow my team has.

Step 10: Integration Test

We unit tested her locally. We manually tested her locally. We manually tested her remotely. In Step 7 we added an echo to more easily test if our Lambda is working without having it do actual work. Let’s automate all that.

Our first integration test invokes the Lambda function. It’s almost exactly the same thing as clicking the Test button, except Node is doing it instead of your finger. Create a new file called index.integrationtest.js and copy pasta this code into it:

const AWS      = require('aws-sdk');
AWS.config.loadFromPath('./credentials.json');
const lambda   = new AWS.Lambda();
const log = console.log;

const expect = require("chai").expect;
const should = require('chai').should();
const {
    handler
} = require('./index');


describe('#index integration', function()
{
    this.timeout(10 * 1000);

    describe('#echo', ()=>
    {
        it('responds to an echo', (done)=>
        {
            var params = {
                FunctionName: "datDJMicrodev", 
                Payload: JSON.stringify({echo: true})
            };
            lambda.invoke(params, (err, data)=>
            {
                log("err:", err);
                log("data:", data);
                done(err);
            });
        });
    });
});

This will send the echo to our Lambda, and she’ll send the pong back. Run it by mocha index.integrationtest.js and she should report a true:

Conclusions

Creating a microservice for AWS Lambda is easy and straightforward. We’ve built one here in this article. Typically, though, you build many of them. Even if you’re building a monolith application, and your one microservice is pretty large, all the deployment problems still exist. As you’ve hopefully seen, AWS API’s give you powerful tools to put both the deployment and testing of those deployments in your hands.

As a developer, you can now write the microservice, tooling, testing, and deployment all in the same language in the same code repository. This code that handles your infrastructure follows the same peer review and unit/integrating testing best practices. Your infrastructure is now testable, high quality code controlled by you, the developer. While we’ve created multiple Lambdas that represent their environment and version, AWS provides powerful tools to handle Lambda Aliases and Versions, both in code and in the AWS console.

Frameworks like Serverless help you manage multiple services, have a nice way to test locally, and help you if you’re not familiar with AWS. YAML isn’t code, however. Writing your service, and infrastructure, in the same language helps you adopts the same testing and debugging practices for it.

Code (again in case you missed it)

The source code for this article is on Github.

At the beginning of October, I started a new project to help my team move a 12 year old application to the cloud. I knew next to nothing about Amazon, or their infrastructure offerings called AWS. In the past 3 months I’ve learned a ton. Today I wanted to share what I’ve learned about Lambdas: Functions you run in AWS.

Also known as serverless architecture.

What is AWS?

AWS is short for Amazon Web Services. Amazon provides you a bunch of different services that help you build software, almost all of which they host.

What Are Lambdas?

Lambdas are one of their offerings that have been out over a year. The selling points of Lambdas are Amazon sets up and maintains the server for you, and you’re only charged for how long your code actually runs. You upload code and it just works.

Why Not Use an EC2 Instead?

An EC2 is Amazon’s virtual machine servers. You have to build and maintain them yourself. You pay to keep them running even when they aren’t handling any traffic. You have to rebuild them when new AMI’s (virtual machine templates) come out from Amazon. Things get really challenging when you scale dynamically for spikes in traffic.

Why Not Docker?

While Docker makes some of the above less painful, especially when testing in Jenkins or testing on other developers machines, you still have all the same issues. You build the docker image and manage them through a container service like DockerSwarm or Consul. The images themselves have to be managed by someone in DockYard.

Why Lambdas

I’m A Front-End Developer

If I want to run my JavaScript in the cloud, I don’t care about the OS, what’s installed on it, and all the details about keeping it running and refreshed with security updates. Amazon does and handles all that for you.

Lamba’s are just code, you don’t have to maintain the server or infrastructure, you only pay for what you use, and they automatically scale based on traffic.

If you’re a front-end developer like me who just wants to write code, and you’re keen on building a back-end API for the front end, they’re perfect.

Lambdas Are Reactive and Event Driven

One additional feature of Lambdas is their built in events. They are “reactive” by default. It’s assumed Lambdas are triggered by other AWS things. Things in this case being API Gateway for REST API’s such as GET and POST, S3 when you drop files on it, SNS for a basic message bus, or even JSON written into logs through CloudWatch. There also are the building blocks for distributed architecture in Step Functions.

Simpler Continuous Deployment

Trying to get monoliths, big applications where all the code is bundled together, deployed to production is hard. Getting this done often is even harder. One of the benefits microservices give you is easier deployment. You can deploy “one thing” without having to worry about the “other things”.

Anyone who’s had back-end developers deploy new API’s, or you deploying new front end code knows that’s a loaded statement. Unless you’ve done some integration and functional testing or you’ve deployed the code to a QA or staging environment which mimics production, you don’t “know” the code works.

If you define clear API’s upfront, keep the microservices truly simple in their functionality, this process becomes much simpler, and eases CI down the line.

Are Lambdas Microservices?

They can be microservices, but no, not by default. They are just a function. Microservices typically are accessed by a REST API. If you trigger them by an API Gateway (a restful URL provided by Amazon), then yes, they become microservices.

Build A Microservice

If you want to build a RESTful microservice with Lambda’s, you first have to have an AWS account. Assuming you have that, login and access the console (the AWS web interface).

Step 1: Build A HelloWorld Lambda

Click the “Services” button/menu, and select/search for “Lambda”, then click “Create a Lambda Function”.

In Runtime select “Node”; version doesn’t matter, just ensure not Edge for now. For template, click on “Blank Function”.

Step 2: Choose A Trigger

Click the empty box and choose “API Gateway”. Make up an API Name, choose “api” for deployment stage, and choose “Open” for security. Click “Next”.

Step 3: Configure

Pick a camel case (ex: likeThisMan) name. Scroll down to Role, and choose “Create a new role from template(s)”. Create a basic camel case role name such as “myCustomTemplateRole”. Scroll to the bottom and click “Next”, then on the next screen scroll to the bottom and click “Create function”.

You’re done!

Note it usually dumps you on the Trigger tab (what causes your Lambda function to run). In our case, it should show the URL for our API Gateway. Copy this to your clipboard or a notepad/textedit for later use.

It generates a basic function for you. You can see it by clicking the Code tab:

exports.handler = (event, context, callback) => {
    callback(null, 'Hello from Lambda');
};

Test & Iterate On Your Microservice

Step 4: Test

Click the blue “Test” button. In the new popup, the sample event template should be “Hello World”. Click “Save and Test”, and you should see “Hello from Lambda” below the execution result. Great, your Lambda function runs.

Step 5: Test Your API Gateway

If you copy and paste your API Gateway URL in the browser, it should say “internal server error”. This is because that Lambda creation wizard you ran created a new REST API for you, but returns a JavaScript string. By default, you need to use JSON. Let’s modify your Lambda code to do that.

exports.handler = (event, context, callback) => {
    
    const response = {
        statusCode: '200',
        body: JSON.stringify('Hello from Lambda'),
        headers: {
            'Content-Type': 'application/json',
        },
    }
    callback(null, response);
};

Now re-open the API Gateway URL in your browser, and you should see your “Hello from Lambda” message.

Debug Your Microservice

Step 6: Custom Logging

The event and context parameters allow you to pass dynamic information to your Lambda, and let you know how your Lambda was invoked. Add 2 log statements to show the event and context variables:

exports.handler = (event, context, callback) => {
    console.log("event:", event);
    console.log("context:", context);

Now click “Save and Test”. The log output at the bottom right should show your 2 log messages. The event is probably the custom Hello World default test data of key1, value1, etc.

Think of “event” as parameters your Lambda can choose to parse and use to dynamically act on. The context gives your Lambda information about itself while running (log group name, how long your lambda function has to run, etc). Let’s see how the event can differ.

Step 7: CloudWatch Logs

Click the “Monitoring” tab, and to the far right you should see “View logs in CloudWatch”. Click it.

Hopefully you should see at least 1 log, but probably more.

They are sorted by most recent, so click on the top one. These should match what’s in your log output when you tested it.

Click the back button, select all the log streams by clicking the box to their left, and click “Delete Log Stream” then “Yes Delete”.

Reopen the API Gateway URL, then go back to your CloudWatch logs and click the little refresh button (the spinning circle arrows) on the top right. It should have just 1 log which you just caused by re-running your API Gateway URL. Click it, and you’ll notice the data is much different.

CloudWatch is pretty good at parsing arbitrary strings and JSON. Click the event area in the list, and it should expand it to show you the full JSON. Notice API Gateway sends us a lot of parameters.

What Languages Can I Use?

You can write your Lambdas in JavaScript, Python, Java, and C#. You can also use your normal libraries. The tutorial above edits the code inline in the AWS console, but a future tutorial shows you how you can automate your code.

What Else Can Lambdas Do?

Amazon’s documentation gives you a list of events that can trigger your lambda. However, some real world use cases I’ve used are:

REST API’s

API Gateway for basic REST Services instead of using Node + Restify + Docker + DockerSwarm (You can still use Node + Restify, heh)

Backend For Front End

Database guy had 1 Lambda for his DynamoDB access. My Lambda was an API for my front-end, and took his JSON and formatted it the way I needed it (my microservice invoked his microservice).

Unexpected Spikes In Traffic == No Worries

Trying to sell a client on how Lambda’s scaled better, and were cheaper than a dozen WebSphere instances in a row. We wired up Gatling to blast it with hundreds of thousands of users over 2 minutes. Amazon was like, “That’ll be $10 please.” lol! #awesome (This as opposed to the Terminator 2 nuke scene that typically resulted in a spike of traffic with the servers melting).

Content Mangement Updating Servers

Dropping updated content from a Content Management System on S3 (a hard drive in the cloud). Then, updating content on some EC2 instances behind an ELB. Using the SDK to describe instances (what servers do I have, and of those, are they running or not), you can then snag their IP addresses. Our EC2’s were in a different security group, so we put a lambda in the same group, gave it a URL from API Gateway, then had Lambda A call Lambda B with the IP addresses to call & updated content to give them.

Conclusions

As you can see, Lambdas provide a wonderful way to build microservices for both your application, and your infrastructure. You can start with a monolith first and make your Lambda’s 1 function turn into a ton, or build a bunch of little ones that may or may not know about each other, but all have a well defined API contract.

Not worrying about servers has freed up me and others to no longer worry about infrastructure, and instead write code.