Friends have asked that I report a from the trenches update on my thoughts on Lambdas. I fell in love about a year ago, so how do I feel now? I’ll cover the pro’s and con’s I faced, what I learned, and what I still don’t know.

Why

If you read my previous article on what Lambdas are, and how you build an automation workflow around them, you’ll see why I like them. In short, a front end developer with a bit of back-end Node experience can use continuous deployment for working REST services (or non) quickly, without worry of maintenance, or sudden spikes in traffic. I’m a coder, and with the AWS Node and Python SDK , I can code my way onto the cloud vs. learning Unix, various weirdness with Terraform/Chef/Ansible/Docker, etc.

I have Amazon manage my infrastructure so I can focus on coding. I love it.

Different Environments

If you utilize multiple AWS accounts and/or environments, the gotchas are basically your permissions. Specifically, IAM roles and what permissions those IAM roles have. If you follow the whole dev/qa/staging/prod etc. life cycle of testing your code in each environment and then “moving” to a higher environment once the code passes all testing, you’ll find you do the same thing here. In my experience, you’re just testing two things: Can my Lambda use it’s permissions the same (log to CloudWatch, access a particular S3 bucket, etc) as well as ensuring connectivity that you could of fat fingered somewhere (i.e. the subnet/security group in the Terraform that’s creating your Lambda isn’t correct for the prod environment)? Beyond that, everything seems to operate the same between environments which I loved.

Lesson: As long as you avoid configuration drift, you’re fine.

No API Gateway?

Lambdas are built for burst traffic or the occasional code run for simple services. However, it does NOT require an API Gateway. Meaning, you do not need a URL to have your code run. It truly is a “function”, or a bunch of code. How that code is run doesn’t have to be a URL being put in the web browser.

AWS is flexible. Lambdas are actually functions, not actual “API’s”. Meaning, anything can trigger them; they ARE functions. Files put on an S3 bucket (hard drive), some log messages in CloudWatch that match a string, and even periodic cron jobs managed by AWS. I can personally attest that during the S3 outage last year, when it came back up, not 1 of my messages over a 2 hour period was lost in my CloudWatch cro

More important, though, is that Lambdas can invoke other Lambdas.

Lesson: Even with AWS, you can code yourself out of any negative situation. :: flexes bicep :: Take a look the triggers that AWS provides for Lambdas. You truly can react to a variety of events.

Lambda’s Calling Lambda’s

Lambda’s can invoke other Lambda functions through lambda.invoke. Given the fact that each Lambda (Python or JavaScript) with a reasonable amount of code runs less than 100 to 200 ms is awesome… and doesn’t cost me much since I’m not running a high TPS in my personal projects, this is great.

Another subtle benefit is that you can call them in either a request / response for a more REST type of feel, or a fire and forget for longer running processes. While I haven’t been able to have the need yet, I loved playing with the step functions for when you have a bunch of microservices you’re trying to coordinate in a concurrent fashion.

Errors and Logging

Be aware that some connection errors aren’t verbose. This may be for security reasons. 99% of my “Lambda can’t connect to some thing” all had to do with a misconfigured security group, subnet, or IAM permission. That’s good to know since those things can be easily automated using Terraform or things like the SDK.

For logging, while Lambdas typically log to CloudWatch by default, make sure you log to a single log stream so you can utilize the console or SDK to more easily find logs in a time window. Searching various log streams with dates in the titles over time to find a particular log is harder if it’s not in a the same stream name.

Time Limit

Years ago, Lambdas were limited to 1 minute. Then they got 5 minutes. 5 minutes is still not enough once you have so much power at your finger tips, heh.

If you run into scenarios where 5 minutes isn’t enough, you have 2 options:

1. Use step functions. This can be challenging for some developers because it inverses the responsibility of those doing the work to poll for jobs vs. you being responsible to tell them. This helps solve concurrency issues, too.
2. Use an audit log + CloudWatch cron job. Either use a NoSQL database like Dynamo or an S3 bucket folder. Your Lambda’s will do their work, then audit that they “started and stopped” with a UUID. This could be a timestamped data entry in Dynamo, or a file in S3 bucket. You then have that long running process make an entry with the same UUID/correlation ID in the same audit log (Dynamo or S3). Finally, you have a CloudWatch cron job launch a Lambda every minute or so to check on the status. When the Lambda finds a stopped job audit entry that matches a start, it can mark the job as successful, error’d, or timed out based on your criteria. This CloudWatch cron jobs are beast, yes, they can be trusted.

Deployment

Deployment is huge contention amongst many developers. I’m a huge proponent of automating everything with the AWS SDK. Others like to use Terrform/Chef/Ansible, etc. Others like to upload ZIP files through the console.

Whatever method you do, I still encourage the use of existing integration testing solutions. AWS’s update process is super easy; you’re literally uploading code to the Lambda directly or an S3 bucket, and “poof”, you’re Lambda is updated. You can use the environment & versioning if you wish. I personally didn’t like that, though, and instead liked using a green/blue deployment model where you have 2 Lambdas with a blue/green suffix in the name. When you upload new code, you simply switch the trigger as the last step once all your integration tests have been run. If you screw up, no big deal, it’s near instant to switch it back without uploading new code.

Things like ping and health checks are still extremely important, as are dry runs. Ping’s are simply a way of ensuring your Lambda function is there. If it’s API Gateway, that’s a URL you can hit to ensure it works. If it’s an S3 bucket trigger, you can drop a particular filename in a particular place, and your Lambda can respond either there or in CloudWatch. For health checks, this is the most important; it ensures your Lambda can talk to all the required services it needs and report back. This goes hand in hand with a dry run to ensure it can talk to those things, but doesn’t affect them; like making accidental database entries. Instead, it’s just ensuring your connectivity (i.e. IAM role permissions, subnet/security groups, VPC’s) are all correct.

Where Do We Go From Here?

A guy reached out to me on Slack asking for help in Python. He’s never coded in his life, about my age (38), and manages databases that are recently moved from an on-prem situation to his AWS. He wanted to basically have an alarm (an AWS log event) trigger a Lambda, and if it’s cause was the database was running out of hard drive, to allocate 50 megs more. I gave him a crash course in Python and what I had learned of AWS and Lambdas, and he was up and running in about a month in prod with a 30 line Python Lambda.

That’s so awesome.

The people using AWS Batch and other short lived EC2’s for minutes to seconds worth of work are moving some of that to Lambdas for price reasons. Various Lambdas are in the background, facilitating various logging, monitoring, or reactionary/reactive roles to help functionality in Ops or for developers. It’s become “just another tool in the toolbox”, not some new, shiny thing to not be trusted by the cynics.

We’re still at the beginning of serverless. It is growing, and more and more people from a variety of skillets want to use them. That’s telling. If you’re not doing a lot of server-side development and just frontend websites, I encourage you to take a look at hosting your files on S3 and simply putting CloudWatch in front of it. AWS has a good wizard for this.

At the beginning of October, I started a new project to help my team move a 12 year old application to the cloud. I knew next to nothing about Amazon, or their infrastructure offerings called AWS. In the past 3 months I’ve learned a ton. Today I wanted to share what I’ve learned about Lambdas: Functions you run in AWS.

Also known as serverless architecture.

What is AWS?

AWS is short for Amazon Web Services. Amazon provides you a bunch of different services that help you build software, almost all of which they host.

What Are Lambdas?

Lambdas are one of their offerings that have been out over a year. The selling points of Lambdas are Amazon sets up and maintains the server for you, and you’re only charged for how long your code actually runs. You upload code and it just works.

Why Not Use an EC2 Instead?

An EC2 is Amazon’s virtual machine servers. You have to build and maintain them yourself. You pay to keep them running even when they aren’t handling any traffic. You have to rebuild them when new AMI’s (virtual machine templates) come out from Amazon. Things get really challenging when you scale dynamically for spikes in traffic.

Why Not Docker?

While Docker makes some of the above less painful, especially when testing in Jenkins or testing on other developers machines, you still have all the same issues. You build the docker image and manage them through a container service like DockerSwarm or Consul. The images themselves have to be managed by someone in DockYard.

Why Lambdas

I’m A Front-End Developer

If I want to run my JavaScript in the cloud, I don’t care about the OS, what’s installed on it, and all the details about keeping it running and refreshed with security updates. Amazon does and handles all that for you.

Lamba’s are just code, you don’t have to maintain the server or infrastructure, you only pay for what you use, and they automatically scale based on traffic.

If you’re a front-end developer like me who just wants to write code, and you’re keen on building a back-end API for the front end, they’re perfect.

Lambdas Are Reactive and Event Driven

One additional feature of Lambdas is their built in events. They are “reactive” by default. It’s assumed Lambdas are triggered by other AWS things. Things in this case being API Gateway for REST API’s such as GET and POST, S3 when you drop files on it, SNS for a basic message bus, or even JSON written into logs through CloudWatch. There also are the building blocks for distributed architecture in Step Functions.

Simpler Continuous Deployment

Trying to get monoliths, big applications where all the code is bundled together, deployed to production is hard. Getting this done often is even harder. One of the benefits microservices give you is easier deployment. You can deploy “one thing” without having to worry about the “other things”.

Anyone who’s had back-end developers deploy new API’s, or you deploying new front end code knows that’s a loaded statement. Unless you’ve done some integration and functional testing or you’ve deployed the code to a QA or staging environment which mimics production, you don’t “know” the code works.

If you define clear API’s upfront, keep the microservices truly simple in their functionality, this process becomes much simpler, and eases CI down the line.

Are Lambdas Microservices?

They can be microservices, but no, not by default. They are just a function. Microservices typically are accessed by a REST API. If you trigger them by an API Gateway (a restful URL provided by Amazon), then yes, they become microservices.

Build A Microservice

If you want to build a RESTful microservice with Lambda’s, you first have to have an AWS account. Assuming you have that, login and access the console (the AWS web interface).

Step 1: Build A HelloWorld Lambda

Click the “Services” button/menu, and select/search for “Lambda”, then click “Create a Lambda Function”.

In Runtime select “Node”; version doesn’t matter, just ensure not Edge for now. For template, click on “Blank Function”.

Step 2: Choose A Trigger

Click the empty box and choose “API Gateway”. Make up an API Name, choose “api” for deployment stage, and choose “Open” for security. Click “Next”.

Step 3: Configure

Pick a camel case (ex: likeThisMan) name. Scroll down to Role, and choose “Create a new role from template(s)”. Create a basic camel case role name such as “myCustomTemplateRole”. Scroll to the bottom and click “Next”, then on the next screen scroll to the bottom and click “Create function”.

You’re done!

Note it usually dumps you on the Trigger tab (what causes your Lambda function to run). In our case, it should show the URL for our API Gateway. Copy this to your clipboard or a notepad/textedit for later use.

It generates a basic function for you. You can see it by clicking the Code tab:

exports.handler = (event, context, callback) => {
    callback(null, 'Hello from Lambda');
};

Test & Iterate On Your Microservice

Step 4: Test

Click the blue “Test” button. In the new popup, the sample event template should be “Hello World”. Click “Save and Test”, and you should see “Hello from Lambda” below the execution result. Great, your Lambda function runs.

Step 5: Test Your API Gateway

If you copy and paste your API Gateway URL in the browser, it should say “internal server error”. This is because that Lambda creation wizard you ran created a new REST API for you, but returns a JavaScript string. By default, you need to use JSON. Let’s modify your Lambda code to do that.

exports.handler = (event, context, callback) => {
    
    const response = {
        statusCode: '200',
        body: JSON.stringify('Hello from Lambda'),
        headers: {
            'Content-Type': 'application/json',
        },
    }
    callback(null, response);
};

Now re-open the API Gateway URL in your browser, and you should see your “Hello from Lambda” message.

Debug Your Microservice

Step 6: Custom Logging

The event and context parameters allow you to pass dynamic information to your Lambda, and let you know how your Lambda was invoked. Add 2 log statements to show the event and context variables:

exports.handler = (event, context, callback) => {
    console.log("event:", event);
    console.log("context:", context);

Now click “Save and Test”. The log output at the bottom right should show your 2 log messages. The event is probably the custom Hello World default test data of key1, value1, etc.

Think of “event” as parameters your Lambda can choose to parse and use to dynamically act on. The context gives your Lambda information about itself while running (log group name, how long your lambda function has to run, etc). Let’s see how the event can differ.

Step 7: CloudWatch Logs

Click the “Monitoring” tab, and to the far right you should see “View logs in CloudWatch”. Click it.

Hopefully you should see at least 1 log, but probably more.

They are sorted by most recent, so click on the top one. These should match what’s in your log output when you tested it.

Click the back button, select all the log streams by clicking the box to their left, and click “Delete Log Stream” then “Yes Delete”.

Reopen the API Gateway URL, then go back to your CloudWatch logs and click the little refresh button (the spinning circle arrows) on the top right. It should have just 1 log which you just caused by re-running your API Gateway URL. Click it, and you’ll notice the data is much different.

CloudWatch is pretty good at parsing arbitrary strings and JSON. Click the event area in the list, and it should expand it to show you the full JSON. Notice API Gateway sends us a lot of parameters.

What Languages Can I Use?

You can write your Lambdas in JavaScript, Python, Java, and C#. You can also use your normal libraries. The tutorial above edits the code inline in the AWS console, but a future tutorial shows you how you can automate your code.

What Else Can Lambdas Do?

Amazon’s documentation gives you a list of events that can trigger your lambda. However, some real world use cases I’ve used are:

REST API’s

API Gateway for basic REST Services instead of using Node + Restify + Docker + DockerSwarm (You can still use Node + Restify, heh)

Backend For Front End

Database guy had 1 Lambda for his DynamoDB access. My Lambda was an API for my front-end, and took his JSON and formatted it the way I needed it (my microservice invoked his microservice).

Unexpected Spikes In Traffic == No Worries

Trying to sell a client on how Lambda’s scaled better, and were cheaper than a dozen WebSphere instances in a row. We wired up Gatling to blast it with hundreds of thousands of users over 2 minutes. Amazon was like, “That’ll be $10 please.” lol! #awesome (This as opposed to the Terminator 2 nuke scene that typically resulted in a spike of traffic with the servers melting).

Content Mangement Updating Servers

Dropping updated content from a Content Management System on S3 (a hard drive in the cloud). Then, updating content on some EC2 instances behind an ELB. Using the SDK to describe instances (what servers do I have, and of those, are they running or not), you can then snag their IP addresses. Our EC2’s were in a different security group, so we put a lambda in the same group, gave it a URL from API Gateway, then had Lambda A call Lambda B with the IP addresses to call & updated content to give them.

Conclusions

As you can see, Lambdas provide a wonderful way to build microservices for both your application, and your infrastructure. You can start with a monolith first and make your Lambda’s 1 function turn into a ton, or build a bunch of little ones that may or may not know about each other, but all have a well defined API contract.

Not worrying about servers has freed up me and others to no longer worry about infrastructure, and instead write code.

Introduction

I have been learning Functional Programming over the past year with a friend of mine. We’ve both cut our teeth on finding who to learn from, what articles are useful, and what actually translates into your day to day programming job. I’ve also learned a lot of natural problems that arise as you start a new project from scratch with an OOP background, or if you’re refactoring some OOP code to be more functional.

In this article, I wanted to share some of what I learned. Specifically what functional programming is, why it matters, how to use pure functions and list comprehensions in the real world. While I use JavaScript in this article, I’ve learned that the concepts and libraries supporting it are available in many programming languages.

Contents

• Why Even Bother?
  • Why Not?
  • Job Market
  • Time & Cognitive Load
  • The Good News
  • The Bad News
  • Beware Professors & The Elite
  • Is This For Apps or For Games or Both?
  • Quick Glossary of Terms
• What is Functional Programming?
• Why is Functional Programming This Way?
• Pure Functions
  • Academic Pure Functions
  • Real World Pure Functions
  • Predicates
  • Node Predicates
  • Building The Predicate Reverse Pyramid
• Array Comprehensions
  • Humble For Each
  • Every
  • Some
  • Filter
  • Reduce
  • Combining Comprehensions
• Checkers
• Conclusions
• Additional Resources
• Special Thanks

Why Even Bother?

Here’s my list derived AFTER actually doing it vs. rehashing marketing I’ve read elsewhere.

1. Code works more than it doesn’t.
2. Unit tests are smaller, and easier to write. Smaller mocks, little to no setup/teardown for test suites.
3. Unit tests break faster allowing me to fix broken code they found faster.
4. Easier to figure out what the hell is going on in a larger code base. If something breaks, it’s easier to pull things apart and improve the testing of those parts to prevent it from happening again. Not like usual spaghetti that can occur with mutable state. Caveat is hard to reason what’s happening in deeply composed predicates; still learning how to debug this.

I’ll elucidate more on those topics below.

Why Not?

Job Market

The echo chamber, albeit a large one, that is JavaScript community (both Node and web) would have you believe the world is moving Functional, and that if you’re using OOP, you clearly missed the memo. Facebook started it with Immutable.js, Flux, and Redux, and Google, Mozilla, & crew have followed suit by nuking Object.observe, and Google slowly morphing Angular 2‘s development to adopt these newer ideas of unidirectional data flow, Observables, etc. Netflix, too, has made huge strides by rewriting much of their site in RxJS, and helping promote the Reactive Manifesto as “the new way” to build applications, both client and server (yes, it’s 2 pages, and no, you won’t walk away knowing how to build reactive applications after reading it). Microsoft’s F# push has added more fuel.

I can tell you from consulting experience and my new job that OOP, mutable state, and languages that support them are alive and well.

Time & Cognitive Load

The other 2 reasons are it takes longer to make things mathematically correct, and as you learn, you often refactor functions a lot to make them more pure. While that’s great, you bubble up the variables storage and creation to a higher level. Over time this results in you eventually having to store/get that data from somewhere. Frameworks like Redux help here, but it’s takes more time to make things nice and pure. While the cognitive load (keeping how the program works in your head as you read the code) actually lowers with pure functions, assembling & testing them can take a lot out of you once you start dealing with higher order functions and work out how to test the larger functions. This is assuming you’re doing tests in tandem with your code.

It’s New

If it’s new, you don’t have an inborn talent for functional programming, nor a penchant for Lambda Calculus,  then it’ll be hard at first. Like unit testing, deadlifting, or doing a hand stand, you have to practice. This takes time. It can reduce the accuracy of time estimates. You’ll get frustrated as you try, fail, repeat.

The Good News

Typically I start these articles with the bad news first, but given the scope of the endeavor we’re about to embark upon, I feel it necessary to start positive.

You most likely already know a lot of this already.

Additionally, there is less overhead in applying these techniques into existing code bases. You can pull in functional techniques (and you probably already have) while still staying true to an existing code base’s OOP bent, nor with adversely affecting anything. Some is language agnostic.

This stuff is easier to grok than a lot of these academics and 300 IQ programmers make it out to be. I’m an art student learning it with good old hard work, which means so can you. Like anything, it just takes practice.

You don’t need to know Haskell, Clojure, or Scala.

The Bad News

You’ll have to wade through a lot of Google search results to find blog posts, even StackOverflow answers, written in easy to understand english. A good many, well meaning academics write these posts using proper terminology, but you leave not knowing how any of it relates to your day to day job and how these techniques can help your daily problems. Smart programmers have taken it upon themselves to give code examples that help you understand how to write functional code, but you’d never actually need a function that adds 1 to 10 in the real world.

If you grew up in OOP like me, it’s hard to break a decade or more of habits you’ve ingrained in your brain. Worse, you’ll have patterns and ways of thinking that shape how you approach solving programming problems.

The industry as a whole, at least for web front end (and some Node) is built around mutability and state. Reading text files, databases, encapsulation with classes, and other state full things have copious tutorials and StackOverflow answers with state. Because it’s easy. Because it’s quick. Because it works.

Simplification of many functional programming terms also poses a challenge because some people think you either cheapen it by doing so, incorrectly explain it by not using all the appropriate terms, or are flat out wrong by pointing out the core value. Ignore it, there are diamonds inside, you just have to ignore all the yuck that goes with mining. Think Minecraft: no grime, just your time.

Beware Professors & The Elite

It’s worth re-iterating: beware academics and elitists. Functional Programming can be for the masses, not some secret cult. Additionally, you can use pieces and get the benefit; you do not need to go full bore to realize positive results in your code and day job.

Try it in pieces. You won’t hurt yourself, nor destroy your codebase. If you offend some PHD, oh well, the world goes on.

Is This for Apps or for Games or Both?

I’ve done no active research for functional programming in games, but did see that Elm Repeatable Mario demo. Using the Component Entity System model, I’ve been having fun trying make a mash up of Final Fantasy 6 & Final Fantasy All The Bravest.

Quick Glossary of Terms

Mutable: You can set a variable again and again. Opposite of a constant. Mutable state means I can set the variable to something else later.

var age = 37;
// cool
age = 38;

const age = 37;
// "TypeError: Assignment to constant variable.
age = 38;

Side Effect: When you call a function and it modifies a variable outside it’s scope.

var age = 37;
var birthday = () => age++;

Specifically, you modify variables outside the function. If it wasn’t a parameter passed in, or it’s not within the function closure, it’s outside, so you should not touch it.

var birthday = (age) => age + 1;

Imperative: Programming languages where you describe how to do something, use a lot of mutable state, and are typically associated with non-functional programming.

Higher Order Function: When you pass a function to another function. JQuery’s click, event handlers, etc. are examples:

// notice first and only parameter passed to click
$( "#target" ).click(function() {
  alert( "Handler for .click() called." );
});

What Is Functional Programming?

The simple definition of Functional Programming is writing functions like you’d write Algebra and instead of changing data, you get new data.

You write pure functions and try to ensure your data is immutable as possible.

Why is Functional Programming this way?

Predicability.

In math, if I add 1 + 1, I know that the answer is always 2. That’s predictable. If I grabbed a random function from anywhere in your code base, and called it twice with the same parameters, would it return the same result each time? Exactly.

In Algebra, you “solve for x”. In math, they’ll say x = 1 + 2. If you’ve done Algebra, you know x is equal to 3. In code, it’d be:

function getX()
{
   return 1 + 2;
}

The important thing here, which may seem painfully obvious, is if you call getX(), you’ll always get 3 back.

If you’ve done any non-functional programming, you know that’s NOT always the case. Observe.

function getX()
{
   return 1 + 2 + data;
}

What is X now? No clue. If someone sets data to 0, then it’s 3. If someone sets data to ‘0’, then it’s ’30’. That, my friends, is not Algebra. You can no longer “mathematically prove” getX returns 3. You have to know the history of what happened before, or even while, you call getX.

Lastly? No errors. If your functions are truly pure, you don’t get thrown exceptions. You only need try/catch for particular nasty/scary sections of your code, 3rd party code, or known mutable sections like database calls or text file read/writes, or just gnarly data parsing/mangling.

Once you can start depending on functions that don’t break, you can start to build more resilient and testable applications.

Pure Functions

Academic Pure Functions

You do this through pure functions: Functions that always return the same result with certain parameters, no many how many times they are called. You saw above the first getX function had no parameters. Let’s give it some:

function add2(a, b)
{
   return a + b;
}

Assuming the function is always passed non-decimal numbers, you’ll always get the same result.

add2(1, 1); // 2
add2(3, 5); // 8
add2(0, 0); // 0

In fact, you can pass in strings, arrays, objects, and even Errors, changing the order of the parameters, and Chrome has improved the pureness of the addition operator, unlike in the past.

Real World Pure Functions

Let’s take a look at the real world. In a Node project I’ve worked on, there are 2 core problems pure functions have helped me with. Specifically using Predicates and Higher Order Functions. DAT POCKETWATCH, THO.

Predicates

Predicates are pure functions that always return true or false and are used for 2 things: To assert obvious things, and used as building blocks to make more ambitious, complicated assertions.

While available in any language, and highly sought after in languages with dynamic typing, strong typing can only take you so far. Eventually you’ll create a domain model, aka “your version of the world described in code” that you want to prove. Like proofs and postulates in Geometry, you start small, and work your way up. The stronger your simple predicates, the stronger your composed predicates become. Composed just means more complex functions made from many smaller functions.

Reminder: These are not unit tests. They’re just functions.

Node Predicates

In my Node API’s, I make assertions about the data you send me for creating a new user. I use predicates to verify the POST variables you send over the wire as JSON are legit. Easy, right? Not in JavaScript. It has no types, nor runtime support for type mis-matching.

Strongly-typed languages still have to deal with parsing unstructured data, and in-memory, already parsed data that they have to make some assertions, NOT assumptions, against. Thus, predicates are still applicable both at runtime as well as in tests.

Let’s start small with some good news: There are already some wonderful, low-level predicates written for you in a library called Lodash. Others like Underscore, Ramda, and 1-liners exist as well. This is important, so we don’t have to deal with the differences between null and undefined, == and ===, nor ever care again.

Here is the REST call’s signature:

URL: https://some.server.com/api/user
Type: POST
Body: Object which has firstName, lastName, EID, adjustmentNumber, activated, creatingEID, and roleID.
Description: When you want to create a user, send their name, company ID, who’s creating this user, what role they are assigned to, whether they are start out activated in the system, and the adjustment number for tracking purposes.

Building the Predicate Reverse Pyramid

Let’s start small and work our way up. Lodash will give us the firm foundation. Strong-typing people, bear with me as we work our way up to higher level predicates.

In Node, your client requests come in this object called Request, whether you’re using Express or Restify. For PUT and POSTS, it should have a body object that has all the JSON the client sent us.

… wait, “should”? No no, strong people, strong, MUST! MUST OR AND DEN OR U SHALL NOT PASS!

Notice, in JavaScript, a null or undefined request is still a worthless request. We gotta have strong basics, remember. Lodash has the opposite of what we need, a _.isNil predicate (is something null or undefined). We need the opposite. Let’s create our own.

var notNil = _.negate(_.isNil);

Cool, let’s test it. I’m just using JSBin.com to play around, but you can use Node & Mocha, Jasmine, Tape, or Assert locally, whatever works for you.

var notNil = _.negate(_.isNil);
console.log(notNil()); // false
console.log(notNil(null)); // false
console.log(notNil(undefined)); // false
console.log(notNil('')); // true
console.log(notNil({})); // true

The first 3 are important. Nothing, which in JavaScript is undefined, we interpret as nil. The 2nd two, being explicit, also are nil. An empty string and an object, the basics of what we get in JSON from clients, are legit. Next up, the body:

var isThereAndHasBody = (req) => notNil(req) && notNil(req.body);
// or
function isThereAndHasBody(req)
{
   return notNil(req) && notNil(req.body);
}

While both functions basically do the same thing, the declared (or named function) is hoisted, defined when the outside script is run, and has a this. The last part is the impure part; a function’s this can change based on who calls it, where, and how using things like bind and Function.call/apply. As an ex-Flash Developer, I saw this take 6 years to solve. In JavaScript, half avoid it entirely whereas others clamor for binds all over the place in ES6 classes.

While your first temptation may be to omit the notNil(req), there are 2 things I want you to note here. First, if we didn’t place it there, our function would return possibly 3 values: true, false, and an uncaught exception if you passed in a non-object or null/undefined. A predicate returns only true or false, not true, false, and boomz.

Second, notice what the function asserts. That a “request is actually there, AND has a body”. That predicate is asserting that it’s a valid object, and that object exists.

“But why can’t we just control the order you call the functions, ensure we’ve already done that check before hand?”

Then the function is no longer pure. Like the getX example way up above, you have to know the history.

There ARE other options, though, like caching the value via memoize, a safer way to get properties via a safe get, or your own controlled predicate via chain or flow.

They’re hardcore mode, though, so let’s keep it simple.

Next up, verifying the firstName is legit:

var validName = (o) => notNil(o) && _.isString(o);
var requestHasValidFirstName = (req) => isThereAndHasBody(req) && validName(req.body.firstName)

We create a validName predicate and then use it plus our previously created isThereAndHasBody predicate together.

console.log(requestHasValidFirstName()); // false
console.log(requestHasValidFirstName({body:{}})); // false
console.log(requestHasValidFirstName({body:{firstName: new Date()}})); // false
console.log(requestHasValidFirstName({body:{firstName: ''}})); // true
console.log(requestHasValidFirstName({body:{firstName: 'Jesse'}})); // true

This is the part where you start to realize, “I thought predicates were going to be a piece of cake, asserting uber obvious stuff, and yet… subtle bugs creep in.” Can you spot it? A firstName of ” is not a valid first name. Let’s modify our validName to be more stringent.

var validName = (name) =>
{
	return notNil(name) &&
	_.isString(name) &&
	name.length > 0;
};
// ...
console.log(requestHasValidFirstName({body:{firstName: ''}})); // false

Now we’re talking. Let’s re-use for the lastName:

var requestHasValidLastName = (req) => isThereAndHasBody(req) && validName(req.body.lastName);
console.log(requestHasValidLastName()); // false
console.log(requestHasValidLastName({body:{lastName: ''}})); // false
console.log(requestHasValidLastName({body:{lastName: 'Warden'}})); // true

Much easier, all I had to do was copy pasta the firstName predicate, and change req.body.lastName. Notice at this point you’ve been composing functions within functions. Your notNil uses Lodash. Your validName uses Lodash + your notNil. Your isThereAndHasBody uses your wrappers over Lodash, etc. You repeat this process to build out the rest of the predicates you need as well as create predicates that assert higher level statements about your data model.

The key point not to miss is that they are ALL pure functions. No matter what order you call, nor what parameters, you’ll always get the same result. You now have a ton of code that is dependable. (Yes, you should still unit test them).

Array Comprehensions

Now that you have a bunch of predicates, you’ll want to apply them all at once. Most languages deal with lots of things in a list. In JavaScript, that’s an Array. List Comprehensions are basically like for loops, except they act like a predicate on an array, items in the array, items in an Object, or both. I’ll show a few first, then we’ll use them with our predicates above.

Humble For Each

If you’ve done a loop on an array, you often will use the for each:

var list = ["uno", "dos", "tres"];
for(var i=0; i<list.length; i++)
{
   console.log(list[i]);
}

She’s been one of the core workhorses in my career. Here’s the functional equivalent:

_.forEach(list, i => console.log(i));

You’ll notice most all array comprehension methods follow a similar function signature of the array to operate on, a function to run on each item of the array, and will return something. “Yo, call this function on each item”.

Every

Going back to our validation predicates, how do we know if they all worked? Lodash’ every will loop through all the things. If even one of the callbacks returns false, then every will return false. If all the things return true, then every will return true.

A simple example:

var list = [true, true, true];
var result = _.every(list, function(item)
{
  return item;
});
console.log(result); // true

Or the shorthand version:

var result = _.every(list, i => i);

All fat arrow functions do an implied return for you. So not this:

_.every(list, i => return i)

but this:

_.every(list, i => i)

Cool, now let’s apply to our validation predicates the same way:

var validRequest = (request) => _.every([
  isThereAndHasBody,
  hasLegitFirstName,
  hasLegitLastName,
  hasValidEID,
  validAdjrNumNumber,
  validActvIndNumber,
  validCreatingEIDNumber,
  validSSORoleID
], predicate => predicate(request));

We pass in a request object (in Express/Restify, it’s an Object which has a params for GET’s and body for POST’s, etc). We give it a list of functions which, hopefully, return a true. The last parameter is our callback that takes the predicate, passes the request to it, and returns the result. “Yo dog, I got this request from node, can you pass it to each of these functions, and if they all say true, lemme know, else, just heck with it, false”.

Whether we pass empty, an object or even an empty body, she’ll always report false:

console.log(validRequest()); // false
console.log(validRequest({})); // false
console.log(validRequest({body:{}})); // false

but if we pass the right stuff:

console.log(validRequest({
  body: {
    firstName: 'Jesse',
    lastName: 'Warden',
    eid: 'aaa111',
    adjrNum: 1,
    actvInd: 1,
    cretNum: 'aaa111',
    ssoRoleID: 1
  }
})); // true

She’s good to go. Now we can validate all requests with assurance when they come in. Another thing, these things are cake to unit test. Here’s some Mocha examples:

it('test is a valid name', function()
{
	predicates.validName('test').should.be.true;
});
it('blank is not a valid name', function()
{
	predicates.validName('').should.be.false;
});
it('numbers are not a valid name', function()
{
	predicates.validName(1).should.be.false;
});

Some

Let’s cover 3 more so you can refer to them later in the post where we’ll use them.

In the application I’m building on at work, we do the same search against multiple databases looking for a match amongst a lot of old data. If even 1 out of the 3 searches finds something, that’s great! When I get a match from a search, I return it like this:

{
  success: true,
  result: [... the data I found]
}

If I didn’t find anything, I get:

{
  success: false,
  error: ... reason why it didn't find anything
}

Now, using old sk00l, imperative techniques, I’d go:

var searchResults = [
  {success: false, error: 'no id matches'},
  {success: false, error: 'no id matches'},
  {success: true, result: [{id: 1, name: 'Cow'}, {id: 1, name: 'Moo'}]}
];
function foundAMatch(searchResults)
{
  var len = searchResults.length;
  for(var i=0; i<len; i++)
  {
    var o = searchResults[i];
    if(o.success)
    {
      return true; 
    }
  }
  return false;
}
console.log(foundAMatch(searchResults)); // true

The new way is to use some (sometimes called any):

function foundAMatch(searchResults)
{
  return _.some(searchResults, o => o.success);
}

If any of the callbacks return true, then some returns true. If all of them return false, then some returns false. Now when I make a bunch of Promise calls, if even 1 of my searches succeeds, I can see if it found a match in a pure way. How do I get it?

Filter

Filter you give an array, and gives you an array back with only the items you want in it. It filters things in and out. Here we can filter some cows:

var animals = [{type: 'cow'}, {type: 'dog'}, {type: 'lizard'}, {type: 'cow'}];
function onlyCowsUpInThisMug(list)
{
  return _.filter(list, function(item)
  {
      if(item.type === 'cow')
      {
          return true;
      }
      else
      {
        return false;
      }
  });
}
console.log(onlyCowsUpInThisMug(animals).length); // 2

Or the more succinct, ES6‘ish way:

var onlyCowsUpInThisMug = (o) => _.filter(o, o => o.type === 'cow');

“Yo, take this list, and take out the things that aren’t cows, kthxbai.”

Using our search above, I can get only the successful searches:

var successfulSearches = (searchResults) => 
_.filter(searchResults, result => result.success);
console.log(successfulSearches(searchResults).length); // 2

Reduce

Cool, there’s just one thing left, getting combined list of those searches. However, they’re all objects of success and result… I just need the result Array, and I need all of them combined. We’ll use reduce to do that.

Reduce is used to take an array, and give you a new one with reduced data. “Reduced” is subjective, however. You can reduce an array to whatever you want. You could reduce data, add new data, or return the same thing, up to you. Unlike other list comprehension methods, reduce takes an optional 3 parameter, something called the accumulator, to accumulate, or collect all the results. This is, like most things in Functional programming, an array.

In imperative, you’d use nested for loops (or some for prop if you take this stupid whiteboard/fizz buzz tests people give you in dumb coding interviews):

var searchResultsThatWorked = successfulSearches(searchResults);
var resultsLen = searchResultsThatWorked.length;
var accumulated = [];
for(var r=0; r<resultsLen; r++)
{
  var searchResultItem = searchResultsThatWorked[r];
  var itemsLen = searchResultItem.result.length;
  for(var i=0; i<itemsLen; i++)
  {
    accumulated.push(searchResultItem.result[i]);
  }
}
console.log(accumulated.length); // 2

Vars everywhere. Let’s use reduce instead:

var accumulated = _.reduce(searchResultsThatWorked, 
(accumulated, searchResultItem)=>
{
  return accumulated.concat(searchResultItem.result); 
}, []);
console.log(accumulated.length); // 2

You’ll notice that I stored the results of reduce in the accumulated variable. However, I used an empty array as the 3rd parameter to reduce. While pure, you could set that 3rd parameter to another variable if you wanted to. We just wrap that reduce in a function called reduceAllSearches.

var reduceAllSearches = (searchResults) => {
  return _.reduce(searchResultsThatWorked, (accumulated, searchResultItem)=>
  {
    return accumulated.concat(searchResultItem.result); 
  }, []);
};

Yes, for the first 2 years it’s hard to read. With practice, your brain adapts to pick out the callback from the call from the array. I’m not being sarcastic for a change, I’m serious. When I first saw this stuff, even before ES6 fat arrow functions, I was like “What a mess”. You’re welcome to use named functions with more clear variable names if that helps your readability. The point here is to use pure functions with arrays via list comprehension methods.

Combining Comprehensions

Putting it all together:

if(foundAMatch(searchResults))
{
  var foundResults = reduceAllSearches(searchResults);
  // ... send foundResults to client
}

Checkers

There are 3 problems, sadly, with our above carefully crafted predicates.

1. If it returns false, we don’t know which one failed, nor why.
2. If we don’t know, then we can’t tell the user how to fix their request.
3. The lack of exceptions in pure functions, which at first you thought was cool, makes it hard to know who failed when they are so deeply composed. (Smartasses will go “well, perhaps if you unit tested them, you’d know”)

I’ll show you how to write checker functions, a trick taught to me by Chris Petrilli. Checkers are like predicates, except instead of always returning true or false with no exceptions, they always return an array. If the array is empty, great, no errors. If it has even 1 item in it, it’s the reason(s) why the predicate failed.

For small checkers, this is obvious stuff, and pointless in strongly typed languages: “Yo dog, Date’s aren’t a String”. Well duh Cap’n Obvious, tanks fo nuthin’. For higher level stuff, and across strongly typed boundaries (HTTP requests, database, parsing data, etc), they are immensely helpful in debugging, both your own code and others. They also take a lot of work, whether you’re a n00b at functional programmer, or an ace.

Let’s add some to our above predicates to see how they work. To do that, we need to create these new type of functions easily. In strongly typed languages, you can just create an interface. For dynamic ones, we’ll take advantage of functions being basically Objects and create special ones by adding data to them we need.

Let’s create something called a validator. It’s basically a predicate, but if it returns false, we can inspect it and snag a descriptive error message of it as to why it failed.

var validator = (errorMessage, funk) =>
{
	var predicateWrapper = function()
	{
		return funk.apply(funk, arguments);
	};
	predicateWrapper['errorMessage'] = errorMessage;
	return predicateWrapper;
}

We wrap our predicate, and add the error message to it. Let’s use with our validName predicate:

var nameValidator = validator('Not a valid name, Holmes.', validName);
console.log("no name:", nameValidator()); // false
console.log("Jesse:", nameValidator('Jesse')); // true

Ok, seems to work exactly like a predicate. But check this out:

console.log(nameValidator.errorMessage); // "Not a valid name, Holmes."

Cool, so if it fails, people can look at that errorMessage to see why. Those of you with OOP deep in yo bones are probably like:

Yo, why can’t we just go validName.errorMessage = “Not a valid name, Holmes” and call it a day?

That’s not a Pure Function, brah! But sure, you could. As you use more list comprehensions, you’ll see your “default mode in life” is to start creating functions for everything so you can use them with things that expect functions. That’s why I created notNil up above instead of using !_.isNil. You can’t pass !_.isNil to list comprehensions.

You don’t use validators directly like you use predicates, they are just a function that houses an error message so when they boom, people know why. Those people in this case are the checkers. Let’s make the function for that:

var checker = function()
{
  var validators = _.toArray(arguments);
  return (thingYouWantToValidate)=>
  {
    return _.reduce(validators, (errors, validatorFunk)=>
                    {
      if (validatorFunk(thingYouWantToValidate))
      {
        return errors;
      }
      else
      {
        return _.chain(errors).push(validatorFunk.errorMessage).value();
      }
    }, []);
  };
};

Whaaa… ok, in a nutshell please?

“Yo, I’m going to send you some predicate functions I made. If you could give me a function back that runs them and gives me an empty array when they work, or errors in that array for the ones that don’t.”

We’ll start with 1, our name validator.

var checkName = checker(nameValidator);
console.log(checkName()); // ["Not a valid name, Holmes."]
console.log(checkName('Jesse')); // []

Make sense, now? Our checker is smart enough to go, “If this thing returns false, look for that errorMessage”.

Let’s do something more sophisticated, and combine our request verifier. Let’s create 3 new validators, and throw ’em in a checker:

var isThereAndHasBodyValidator = validator('No body or invalid request.', isThereAndHasBody);
var firstNameValidator = validator('Not a valid firstName.', hasLegitFirstName);
var lastNameValidator = validator('Not a valid lastName.', hasLegitLastName);
var checkNameRequest = checker(isThereAndHasBodyValidator, firstNameValidator, lastNameValidator);

The checkNameRequest will first verify it’s an object with a body, then verify both the first and last name are valid. For the ones that fail, it’ll put the errorMessages in the array, and when all done, send that array back. Full of errors for invalid, empty for valid:

console.log(checkNameRequest()); // ["No body or invalid request.", "Not a valid firstName.", "Not a valid lastName."]
console.log(checkNameRequest({body: {
  firstName: 'Jesse'
}})); // ["Not a valid lastName."]
console.log(checkNameRequest({body: {
  firstName: 'Jesse',
  lastName: 'Warden'
}})); // []

Once you put all your predicates in a checker for the request, you can then use it to verify if the request is good and if not, both debug it yourself, as well as send it back so whoever is calling it knows what they did wrong.

var errors = checkRequest(req);
if(errors.length > 0)
{
  // ... return errors
}
else
{
  // ... use the request
}

Conclusions

You’ve seen how pure functions allow your code to be predictable, easily unit testable, and prevent you from having to use try/catch to handle errors. Composing many of them together allows you to build a more solid code base.

You’ve seen how you already use higher order functions all over the place in the existing programming you’ve probably already done. Passing functions to other functions is commonplace.

You’ve seen how using list comprehensions allows you not just easily consume data from arrays, but also using them to call and interact with many other pure functions.  In many cases, it results in less code compared to the imperative equivalents.

You’ve seen how predicates allow you to practice writing pure functions, validate your inputs, and give you great runtime feedback when combined with checkers.

Most importantly, you’ve probably seen how you can use some functional programming in your existing code base.

In a future post, I’ll refactor some existing OOP code that illustrates the bubbling problem you run into using pure functions (forcing functions higher up to store the actual mutable data). I’ll also showcase how this larger amount of pure functions can simplify your unit tests, and how it changes your usage of mocks. Finally, I’ll show some small tips on how to more easily test some of the async parts of your code.

Additional Resources

1. Master the JavaScript Interview: What is a Pure Function?
2. Functional Programming in JavaScript: Array methods tutorial
3. Goodbye, Object Oriented Programming

Special Thanks

Thanks a lot to Andrew Purdon, Drew Bourne, Joel Hooks, John Lindquist, Dan Abramov, Eric Elliot, Alan Shaw, Paul Taylor, John-David Dalton, Chris Petrilli, and Matt Lancaster for helping me learn all this new functional stuff.