A co-worker asked about code organization in Functional Programming. He’s working with a bunch of Java developers in Node for a single AWS Lambda, and they’re using the same style of classes, various design patterns, and other Object Oriented Programming ways of organizing code. He wondered if they used Functional Programming via just pure functions, how would they organize it?

The OOP Way

If there is one thing I’ve learned about code organization, it’s that everyone does it differently. The only accepted practice that seems to have any corroboration across languages is having a public interface for testing reasons. A public interface is anything that abstracts a lot of code that deals with internal details. It could be a public method for classes, a Facade or Factory design pattern, or functions from a module. All 3 will utilize internal many functions, but will only expose one function to use them. This can sometimes ensure as you add things and fix bugs, the consumers don’t have to change their code when they update to your latest code. Side effects can still negatively affect this.

Single Class Module

Suffice to say, the OOP way, at least in Node, typically consists of 2 basic ways. The first way is to create a class, and then expose it as the default export:

// CommonJS
class SomeThing { ... }

module.exports = SomeThing

// ES6
class SomeThing { ... }
export default SomeThing

Export Multiple Things

The second is to expose many things, including classes, functions, event variables, from the same module:

// CommonJS
class SomeThing { ... }

const utilFunction = () => ...

const CONFIGURATION_VAR = ...

module.exports = {
    SomeThing,
    utilFunction,
    CONFIGURATION_VAR
}

// ES6
export class SomeThing { ... }

export const utilFunction = () => ...

export const CONFIGURATION_VAR = ...

Once you get past these 2 basic ways of exporting code, things stop looking the same from project to project, and team to team. Some use different frameworks like Express which is different than how you use Nest. Within those frameworks, 2 teams will do Express differently. One of those teams will sometimes organize an Express project differently in a new project than a past one.

The FP Way

The Functional Programming way of organizing code, at least in Node, follows 2 ways.

Export Single Function

The first exports a single function from a module:

// CommonJS
const utilFunction = () => ...

module.exports = utilFunction

// ES6
const utilFunction = () => ...
export default utilFunction

Export Multiple Functions

The second way exports multiple functions from a module:

// CommonJS
const utilFunction = () => ...
const anotherHelper = () => ...

module.exports = {
    utilFunction,
    anotherHelper
}

// ES6
export const utilFunction = () => ...
export const anotherHelper = () => ...

Variables?

Sometimes you’ll see where they’ll export variables alongside functions where others who are more purist and want to promote lazy evaluation will just export functions instead:

// pragmatic
export CONFIGURATION_THING = 'some value'

// purist
export configurationThing = () => 'some value'

Examples

We’ll create some examples of the above to show you how that works using both single and multiple exports. We’ll construct a public interface for both the OOP and FP example and ignore side effects in both for now (i.e. HTTP calls) making the assumption the unit tests will use the public interface to call the internal private methods. Both will load the same text file and parse it.

Both examples will be parsing the following JSON string:

[
	{
		"firstName": "jesse",
		"lastName": "warden",
		"type": "Human"
	},
	{
		"firstName": "albus",
		"lastName": "dumbledog",
		"type": "Dog"
	},
	{
		"firstName": "brandy",
		"lastName": "fortune",
		"type": "Human"
	}
]

Example: OOP

We’ll need 3 things: a class to read the file with default encoding, a class to parse it, and a Singleton to bring them all together into a public interface.

readfile.js

First, the reader will just abstract away the reading with optional encoding into a Promise:

// readfile.js
import fs from 'fs'
import { EventEmitter } from 'events'

class ReadFile {

    readFile(filename, encoding=DEFAULT_ENCODING) {
        return new Promise(function (success, failure) {
            fs.readFile(filename, encoding, function(error, data) {
                if(error) {
                    failure(error)
                    return
                }
                success(data)
            })
        })
    }
}

export DEFAULT_ENCODING = 'utf8'
export ReadFile

parser.js

Next, we need a parser class to take the raw String data from the read file and parse it into formatted names in an Array:

// parser.js
import { startCase } from 'lodash'

class ParseFile {
    
    #fileData
    #names

    get names() { 
        return this.#names
    }
    
    constructor(data) {
        this.#fileData = data
    }

    parseFileContents() {
        let people = JSON.parse(this.#fileData)
        this.#names = []
        let p
        for(p = 0; p < people.length; p++) {
            const person = people[p]
            if(person.type === 'Human') {
                const name = this._personToName(person)
                names.push(name)
            }
        }
    }

    _personToName(person) {
        const name = `${person.firstName} ${person.lastName}` 
        return startCase(name)
    }
}

export default ParseFile

index.js

Finally, we need a Singleton to bring them all together into a single, static method:

// index.js
import ParseFile from './parsefile'
import { ReadFile, DEFAULT_ENCODING } from './readfile'

class PeopleParser {

    static async getPeople() {
        try {
            const reader = new ReadFile()
            const fileData = await reader.readFile('people.txt', DEFAULT_ENCODING)
            const parser = new ParseFile(data)
            parser.parseFileContents()
            return parser.names
        } catch(error) {
            console.error(error)
        }
    }

}

export default PeopleParser

Using PeopleParser’s Static Method

To use it:

import PeopleParser from './peopleparser'
PeopleParser.getPeople()
.then(console.log)
.catch(console.error)

Your folder structure will look like so:

Then you unit test PeopleParser with a mock for the file system (fs).

Example: FP

For our Functional Programming example, we’ll need everything in this article, heh! Seriously, a list of pure functions:

Function for Default Encoding

export const getDefaultEncoding = () =>
    'utf8'

Function to Read the File

const readFile = fsModule => encoding => filename =>
    new Promise((success, failure) =>
        fsModule.readFile(filename, encoding, (error, data) =>
            error
            ? failure(error)
            : success(data)
        )

Function to Parse the File

const parseFile = data =>
    new Promise((success, failure) => {
        try {
            const result = JSON.parse(data)
            return result
        } catch(error) {
            return error
        }
    })

Function to Filter Humans from Array of People Objects

const filterHumans = peeps =>
	peeps.filter(
        person =>
            person.type === 'Human'
    )

Function to Format String Names from Humans from a List

const formatNames = humans =>
    humans.map(
        human =>
            `${human.firstName} ${human.lastName}`
    )

Function to Fix Name Casing and Map from a List

const startCaseNames = names =>
    names.map(startCase)

Function to Provide a Public Interface

export const getPeople = fsModule => encoding => filename =>
    readFile(fsModule)(encoding)(filename)
        .then(parseFile)
        .then(filterHumans)
        .then(formatNames)
        .then(startCaseNames)

Using getPeople

To use the function:

import fs from 'fs'
import { getPeople, getDefaultEncoding } from './peopleparser'

getPeople(fs)(getDefaultEncoding())('people.txt')
.then(console.log)
.catch(console.error)

Your folder structure should look like this:

Then you unit test getPeople using a stub for fs.

Conclusions

As you can see, you can use the basic default module export, or multiple export option in CommonJS and ES6 for both OOP and FP code bases. As long as what you are exporting is a public interface to hide implementation details, then you can ensure you’ll not break people using your code when you update it, as well as ensuring you don’t have to refactor a bunch of unit tests when you change implementation details in your private class methods/functions.

Although the FP example above is smaller than the OOP one, make no mistake, you can get a LOT of functions as well, and you treat it the same way; just export a single function from a another module/file, or a series of functions. Typically you treat index.js in a folder as the person who decides what to actually export as the public interface.

const add = (a, b) => a + b

vs. a total function:

const addNumbers = (a, b) =>
  ( isNumber(a) && isNumber(b) )
  ? {ok: true, data: a + b}
  : {ok: false, error: new Error(Either a or b aren't Numbers.)}

While same input, same output, no side effects sounds like the end all, be all… it’s not.

A pure function like this will always return 3:

console.log(add(1, 2))

… but this will always return “1Error: moo”:

console.log(add(1, new Error('moo')))

Not helpful.

This is where types help, no doubt, but at runtime, validation of your your data becomes important when you’re veer away from what types can help you find (range of a number, shape of data from remote JSON, etc).

Using the total function:

console.log(addNumbers(1, 2))

will always give return:

{ok: true, data: 3}

however, if the types aren’t correct:

console.log(addNumbers(1, new Error('moo')))

will always return:

{ok: false, error: {…}}

A much more useful result to the developer, and less burden on their part.

Not all functions need be total. Just make your privates pure, and your publics total, for example. That way the privates don’t get bad data.

Welcome to Part 6, the final installment in this series. Below we cover unit testing the noop next, how to create pure functions that wrap noop so you can compose them, and finally using code coverage to strategically hit the last code that’s not covered.

Contents

This is a 6 part series on refactoring imperative code in Node to a functional programming style with unit tests. You are currently on Part 6.

• Part 1 – Ground Rules, Export, and Server Control
• Part 2 – Predicates, Async, and Unsafe
• Part 3 – OOP, Compose, Curry
• Part 4 – Concurrency, Compose, and Coverage
• Part 5 – Noops, Stub Soup, and Mountebank
• Part 6 – Next, Logging, and Conclusions

Next is Next

The last part is deal with next. THIS is where I’d say it’s ok to use mocks since it’s a noop, but you do want to ensure it was called at least once, without an Error in the happy path, and with an Error in the bad path.

… but you’ve made it this far which means you are FAR from ok, you’re amazing. NO MOCKS!

Here’s the basic’s of testing noops. You sometimes KNOW if they did something, or how they can affect things based on their arguments. In next‘s case, he’ll signal everything is ok if passed no arguments, and something went wrong and to stop the current connect middleware chain if an Error is passed, much like a Promise chain. We also know that next always returns undefined.

Ors (Yes, Tales of Legendia was bad)

One simple way is to just inject it into the existing Promise chain we have:

.then( info => next() || Promise.resolve(info))
.catch( error => next(error) || Promise.reject(error)))

This’ll make the connect middleware happy as well as satisfying our unit tests expecting either an email info out, or a rejected promise with error.

Pass Through

Much like you log Promise or other Monad chains, simply make a pass through function.

const nextAndResolve = curry((next, info) => next() || Promise.resolve(info))
const nextAndError = curry((next, error) => next(error) || Promise.reject(error))

And since we already know what the next is, we can just attach ’em to the end of the Promise chain, similar to above:

.then(nextAndResolve(next))
.catch(nextAndError(next))

“Wat!?” you may be saying, “all you did was wrap that previous example in functions”. Yup, they’re pure, and you can test those in isolation vs. test that gigantic Promise chain with 20 billion stubs.

describe('nextAndResolve when called', ()=> {
    it('should fulfill', () => {
        return expect(nextAndResolve(noop, 'info')).to.be.fulfilled
    })
    it('should resolve', () => {
        return expect(nextAndResolve(noop, 'info')).to.become('info')
    })
})

Let’s do the same for error:

describe('nextAndError when called', ()=> {
    it('should fulfill', () => {
        return expect(nextAndError(noop, new Error('they know what is what'))).to.be.rejected
    })
    it('should resolve', () => {
        const datBoom = new Error('they just strut')
        return expect(nextAndError(noop, datBoom)).to.rejectedWith(datBoom)
    })
})

If you’re Spidey Sense left over from your Imperative/OOP days is tingling, and you really want to create a mock or spy, go for it.

I would rather you focus your time on creating pure Promise chains vs. investing in better testing side effects using the connect middleware.

Mopping Up

Only 2 functions to go, and we’re up in dem hunneds .

sendEmail … or not

The sendEmailOrNext is a textbook case of copy-pasta coding. We’ll duplicate our previous sendEmail unit tests since they have some yum yum stubs:

describe('sendEmailOrNext when called', ()=> {
    ...
    const reqStubNoFiles = { cookie: { sessionID: '1' } }
    ...
    it('should work with good stubs', ()=> {
        return expect(
            sendEmailOrNext(
                readFileStub,
                configStub,
                createTransportStub,
                getUserEmailStub,
                renderStub,
                reqStub,
                resStub,
                nextStub
            )
        ).to.be.fulfilled
    })
    it('should fail with bad stubs', ()=> {
        return expect(
            sendEmailOrNext(
                readFileStubBad,
                configStub,
                createTransportStub,
                getUserEmailStub,
                renderStub,
                reqStub,
                resStub,
                nextStub
            )
        ).to.be.rejected
    })
    it('should resolve with false if no files', ()=> {
        return expect(
            sendEmailOrNext(
                readFileStub,
                configStub,
                createTransportStub,
                getUserEmailStub,
                renderStub,
                reqStubNoFiles,
                resStub,
                nextStub
            )
        ).to.become(false)
    })
})

I’ve left out all the stubs, save the new one that shows, if we have no files, it just bypasses the entire thing, calls next with no value signaling to connect we’re good to go and finished, but still returning a useful value of false to say we didn’t do what we’re supposed to since we couldn’t find any files.

There And Back Again

Oh look, coverage found yet another noop, imagine that.

const logPort = port => console.log(`Example app listening on port ${port}!`) || true

And the 1 test:

describe('logPort when called', ()=> {
    it('should be true with a port of 222', ()=> {
        expect(logPort(222)).to.equal(true)
    })
})

Should You Unit Test Your Logger!?

If you really want to create a mock/spy for console.log, I admire your tenacity, Haskell awaits you with open arms in your future. Remember me when you’re at the top and you’ve bested Mr. Popo on the Lookout.

… however, if you’re not using console.log, and instead using a known Node logger like Pino, well, heh, you SHOULD!

I should point out some well known Clojure guys think I’m going overboard:

If you read the Twitter thread, an Elm guy schools me, and the Haskell cats show me how even there you can trick Haskell, but still acknowledge (kind of?) the purity pain.

Conclusions

Running coverage, we get:

Welcome to hunneds country. Congratulations on making this far, and not one import of sinon to be found. Don’t worry, deadlines and noops are all over the place in the JavaScript world, and sinon’s mock creation ability is not something to be shunned, but embraced as a helpful addition to your toolset. Like all technical debt, create it intentionally with notes about why, what to remove, context around creating it, etc in code comments around it.

Hopefully you’ve seen how creating small, pure functions, makes unit testing with small stubs a lot easier, faster, and leads to more predictable code. You’ve seen how you can use Functional Programming best practices, yet still co-exist with noops in imperative code, and classes and scope in OOP.

I should also point out that writing imperative or OOP code like this is a great way to learn how to write FP. Sometimes our brains are wired to flesh out ideas in those styles, and that’s great. Whatever works to get the ideas down. On the front-end, using vanilla React’s classes vs. recompose… or even just components functions only. Functional Programming will still hurt just as much as Imperative or OOP if you’ve got a not-so-figured out idea on how some problems should be solved and structured. Write it so it feels comfortable, the refactor & test to purity.

You’ve seen how to compose these functions together synchronously via operands like && and ||, using flow/compose. You’ve also seen how to compose them in Promise chains and Promise.all, curried or not.

You’ve also seen how 100% unit test coverage in a FP code base still has bugs that are only surfaced with concrete implementations using integration testing, such as with Mountebank.

Node is often used for API’s that have little to no state and are just orchestration layers for a variety of XML SOAP, JSON, and text. The no mutable state of Functional Programming works nicely with that concept in ensuring those no state to debug.

Conversely, if you’re dealing with a lot of state such as database connections and streaming text files, not only can you see how you can test such challenging areas using pure functions, but also how many of those things can be run at the same time to speed things up with stepping on each other through global variables.

Props to Cliff Meyers for teaching me what Mountebank is.

Code & Help

The Code is on Github. Got any questions, hit me up on YouTube, Twitter, Facebook (I’m Jesse Warden), or drop me an email.

Welcome to Part 5 where we cover more about noops with a utility to test them, the “stub soup” that can happen if you don’t create small pure functions, and how we can utilize stubs instead of mocks to unit test larger functions. The most important part, though, is setting up Mountebank to show how integration tests can show problems in your unit tests despite 100% coverage. We use wrapping class instances as an example to show you the pitfalls Object Oriented Programming code can make for you.

Contents

This is a 6 part series on refactoring imperative code in Node to a functional programming style with unit tests. You are currently on Part 5.

• Part 1 – Ground Rules, Export, and Server Control
• Part 2 – Predicates, Async, and Unsafe
• Part 3 – OOP, Compose, Curry
• Part 4 – Concurrency, Compose, and Coverage
• Part 5 – Noops, Stub Soup, and Mountebank
• Part 6 – Next, Logging, and Conclusions

The Final Battle?

At this point, we’ve shrunk our Express route as much as we’re going to without some serious refactoring. Let’s unit test it using (read copy pasta) all the stubs we’ve already made.

Noops

Before we proceed, let’s explain what a noop is. Pronounced “no awp”, it’s slang for “no operation”. It means a function that doesn’t return a value so it apparently has no effect because we have no proof it did any operation. That isn’t true; we all live and die by console.log which always returns undefined. If you run in ECS, all those console.log calls are putting text in standard out and you’re probably collecting all those logs for into ELK or CloudWatch or Splunk. That’s certainly an “operation with noticeable effect”. Functional Programmers call that a “side effect” of the function.

Often you’ll stub them in unit tests like () => undefined or the less clear, but shorter () => {}. Save yourself some typeing and use Lodash’ noop, Ramda always if you’re a Ramda purist or noop in Ramda Adjunct.

My God, It’s Full Of Stubs

Here’s the unit test with stubs needed above it for sendEmail t3h lelz:

describe('sendEmail when called', ()=> {
    const readFileStub = (path, encoding, callback) => callback(undefined, 'email')
    const configStub = { has: stubTrue, get: () => 'email service' }
    const createTransportStub = () => ({
        sendEmail: (options, callback) => callback(undefined, 'info')
    })
    const getUserEmailStub = () => 'email'
    const renderStub = stubTrue
    const reqStub = {
        cookie: { sessionID: '1' },
        files: [{scan: 'clean', originalname: 'so fresh', path: '/o/m/g'}]
    }
    const resStub = {}
    const nextStub = noop
    it('should work with good stubs', ()=> {
        return expect(
            sendEmail(
                readFileStub,
                configStub,
                createTransportStub,
                getUserEmailStub,
                renderStub,
                reqStub,
                resStub,
                nextStub
            )
        ).to.be.fulfilled
    })
})

A successful test, but the stubs, while succinct, almost outnumber the lines of code for the test. As you can see, Functional Programming, even when attempted with best effort, doesn’t necessarily “solve” your unit tests having to create a lot of “test code”. Mocks often get a bad rap for being verbose and hard to maintain. Stubs I believe are included in this, but at least with stubs they’re smaller, easier, and “mostly pure”. Still, as soon as you refactor your implementation, you’ll have to fix your tests, and sometimes your stubs will have to change too.

Trust me, this is much more preferable than to refactoring mocks.

The best thing to do is remember your training of the basics, like DRY: don’t repeat yourself, and keep your tests organized with commonly used good and bad stubs within reach (meaning you don’t have to scroll too far to read them). Like OOP, FP functions should be short and focused to be re-usable, and most importantly, composeable. It’s hard and gets easier with practice. A fully fleshed out idea of the problem you’re trying to solve helps a lot, don’t be afraid to prototype in imperative code.

sendEmail Unit Tests

Let’s add the failing which is easy because basically any stub could fail and the whole function fails:

it('should fail when reading files fails', ()=> {
    return expect(
        sendEmail(
            readFileStubBad,
            configStub,
            createTransportStub,
            getUserEmailStub,
            renderStub,
            reqStub,
            resStub,
            nextStub
        )
    ).to.be.rejected
})

And the most important, what sendEmail eventually resolves to. Yes, returning a Promise every time is important, but let’s ensure it resolves to something:

it('should resolve to an email sent', ()=> {
    return sendEmail(
        readFileStub,
        configStub,
        createTransportStub,
        getUserEmailStub,
        renderStub,
        reqStub,
        resStub,
        nextStub
    )
    .then(result => {
        expect(result).to.equal('info')
    })
})

And coverage is now:

The feels.

Class Composition is Hard, Get You Some Integration Tests

Sadly, this is where Functional Programming and Object Oriented Programming stop working together. Our code has a bug that you won’t find in unit tests, only integration tests. In languages like JavaScript, Python, and Lua, when you call functions on Classes without the class attached, they’ll often lose scope (this or self will be undefined/nil). Eric Elliot breaks down the details of a lot of these cases in his article Why Composition is Harder With Classes.

Integration tests using Supertest or Mountebank help with different levels of integration tests. We’ll use Mountebank in this article. Suffice to say your unit tests are only as good as the stubs you provide. The stubs you provide basically fake or emulate functionality of the dependencies and are as small and simple as possible. Stubs are different code than the concrete (real) implementations, and unless they capture it exactly, don’t always test the same. You’ll see an example of this below testing OOP code.

Pitfalls When Stubbing Class Methods

Notice none of our stubs use any classes or Object.prototype, and yet, this is exactly how nodemailer works. It all comes down to class instances losing their this scope. Let’s write a basic pure wrapper around nodemailer, unit test it, show it pass, then setup Mountebank so we can show it breaks at runtime when you use the real nodemailer vs. stubs. Given Mountebank is a large topic, I won’t cover too much how it works, but the code is included. Just know it’ll listen on the email port, act like an email server, and send real email responses so nodemailer believes it truly is sending an email.

Let’s create a sandbox.js file to play and a sandbox.test.js in the tests folder.

const nodemailer = require('nodemailer')

const sendEmailSafe = (createTransport, mailOptions, options) =>
    new Promise((success, failure) => {
        const { sendMail } = createTransport({mailOptions})
        sendMail(options, (err, info) =>
            err
            ? failure(err)
            : success(info))
    })

module.exports = sendEmailSafe

And the unit test:

...
describe.only('sendEmailSafe when called', ()=> {
    it('should work with good stubs', () => {
        const createTransportStub = () => ({
            sendMail: (options, callback) => callback(undefined, 'email sent')
        })
        return sendEmailSafe(createTransportStub, {}, {})
        .then(result => {
            expect(result).to.equal('email sent')
        })
    })
})
...

Integration Test

So it passes. Let’s try with the integration test that uses a real nodemailer instance vs. our unit test stubs. I’ve setup mountebank to listen on port 2626 at localhost for emails. If someone sends an email with the from address equaling “jesterxl@jessewarden.com”, it’ll respond with an “ok the email was sent, my man”. Before we automate this, let’s do it manually first.

Setting Up Mountebank

Run npm i mountebank --save-dev first, then once it is complete, open your package.json and add this script:

"scripts": {
    ...
    "mb": "npx mb"
  },
...

Now when you run npm run mb, it’ll run Mountebank. It’ll basically start a Node server on port 2525. As long as it’s running, you can send it JSON to add imposters (mocks for running services).

Setting Up Your Imposters

An imposter is basically a mock or stub for services. Typically mocks and stubs are used for unit tests and are created in code. In Mountebank, however, you create one by sending Mountebank REST calls. For example a POST call, typically on localhost:2525. You send it JSON describing what port it’s supposed to listen on, what other things to look for, and what JSON & headers to respond with. Mountebank will then spawn a service on that port listening for incoming connections. If the request has attributes it recognizes (you do a GET on port 9001 with a path of /api/health, you send an email on port 2626 from ‘cow@moo.com’, etc), it’ll respond with whatever stub you tell it too. For example, some hardcoded JSON with an HTTP 200 status code. While stubs typically respond immediately or with Promises, these respond as a REST response.

Check out the import top part of our setupMountebank.js that I’ve placed in the “test-integration” folder:

...
const addNodemailerImposter = () =>
    new Promise((success, failure)=>
        request({
            uri: 'http://localhost:2525/imposters',
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'Accept': 'application/json'
            },
            body: JSON.stringify({
                protocol: 'smtp',
                port: 2626,
                stubs: [{
                    predicates: [
                        {
                            contains: {
                                from: 'jesterxl@jessewarden.com'
                            }
                        }
                    ]
                }]
            })
        },
    ...

As you can see, a simple POST request to tell Mountebank, “Yo, if any dude sends an email on port 2626 and it is from me, respond it worked”. Typically Mountebank wants to define that response in the stubs Array, but for emails, you can leave it blank and it’ll default to a success response. At the time of this writing, there is no way to make an email fail, only REST calls. As long as Mountebank is running, it’ll remember to do this unless you delete it, or close Mountebank.

Now, to play around, I’ve run it manually to register. Check the bottom code:

if (require.main === module) {
    Promise.all([
        addNodemailerImposter()
    ])
    .then(() => console.log('Mountebank Imposters initialized successfully.'))
    .catch(error => console.error('Mountebank Imposters failed:', error))
}

Open a new terminal, and run node test-integration/setupMountebank.js, and you should see the Mountebank terminal light up:

> npx mb

info: [mb:2525] mountebank v1.14.1 now taking orders - point your browser to http://localhost:2525 for help
info: [mb:2525] POST /imposters
info: [smtp:2626] Open for business...

That “Open for business…” line is key; that means Mountebank understood what you want and worked and you can now send emails to port 2626. Let’s do that.

Sending the Email

Open up sandbox.js and check the code that’ll run if we use Node to run it:

if (require.main === module) {
    sendEmailSafe(
        nodemailer.createTransport,
        {
            host: 'localhost',
            port: 2626,
            secure: false
        },
        {
            from: 'jesterxl@jessewarden.com',
            to: 'jesterxl@jessewarden.com',
            subject: 'what you hear, what you hear is not a test',
            body: 'Dat Body Rock'
        }
    )
    .then(result => {
        console.log("result:", result)
    })
    .catch(error => {
        console.log("error:", error)
    })
}

Swing and a Miss

Rad, now try to send an email via node src/sandbox.js:

error: TypeError: Cannot read property 'getSocket' of undefined
    at sendMail (/Users/jessewarden/Documents/_Projects/fp-node/node_modules/nodemailer/lib/mailer/index.js:143:25)
...

Whoa, wat? Let’s go into Nodemailer’s sourcecode and see what the heck is going on:

/* node_modules/nodemailer/lib/mailer/index.js line 134 */
sendMail(data, callback) {
    ...
        /* broken below this comment */
    if (typeof this.getSocket === 'function') {
        this.transporter.getSocket = this.getSocket;
        this.getSocket = false;
    }
...

It’s doing a type check to see if this.getSocket is a function vs. a Boolean. That’s fine, but they should of check for this being undefined first.

Or should they? Once you’re in class world, and you’ve been doing OOP for awhile, you shouldn’t have to check for this; it’s just a normal part of how classes work. If it doesn’t, something more fundamental is messed up, the most common being forgetting to setup bind in the constructor for callbacks for example.

We’re not going to fix Nodemailer to make it more friendly to FP developers. In fact, this is a common trend in that many libraries you use both in Node and in the Browser (i.e. from node_modules) will be written in all sorts of ways. You need to be accommodating.

Instead, we’ll assume that we’re not allowed to call sendEmail alone, and ensure it’s always something.sendEmail; whatever to the left is the instance, and will retain scope if you call it like that. This is why a lot of my examples you’ll see I’ll make fs the dependency and then call fs.readFile vs. just readFile.

FP-Fu

Let’s first fix our implementation to be OOP friendly, and ensure the tests still work, else fix ’em. The old code:

const { sendEmail } = createTransport({mailOptions})
        sendMail(options, (err, info) =>

The new code:

const transport = createTransport({mailOptions})
        transport.sendMail(options, (err, info) =>

Re-run the tests:

Cool, the original public interface still works and hides the OOPy stuff. Let’s test the integration test now with sending a real email with real, concrete implementation vs. stubs:

result: { accepted: [ 'jesterxl@jessewarden.com' ],
  rejected: [],
  envelopeTime: 3,
  messageTime: 3,
  messageSize: 590,
  response: '250 2.0.0 Ok: queued as f19f95e62da4afeade02',
  envelope:
   { from: 'jesterxl@jessewarden.com',
     to: [ 'jesterxl@jessewarden.com' ] },
  messageId: '<ab0bab09-7628-007b-f497-3d53806704ae@jessewarden.com>' }

I was like Whee!!! Let’s fix our original implementation now that we’ve proven we know how to wrangle OOP with FP now. We’ll change:

...
sendEmailSafe(
  createTransport(
    createTransportObject(emailService.host, emailBody.port)
  ).sendEmail,
...

To:

...
sendEmailSafe(
  createTransport(
    createTransportObject(emailService.host, emailBody.port)
  ),
...

And switch sendEmailSafe:

const sendEmailSafe = curry((sendEmailFunction, mailOptions) =>
  new Promise((success, failure) =>
    sendEmailFunction(mailOptions, (err, info) =>
      err
      ? failure(err)
      : success(info)
    )
  )
)

To the new OOP-friendly version:

const sendEmailSafe = curry((transport, mailOptions) =>
  new Promise((success, failure) =>
    transport.sendEmail(mailOptions, (err, info) =>
      err
      ? failure(err)
      : success(info)
    )
  )
)

This breaks 1 of the tests because the stub used to be a function; now it needs to be an Object with a function. However… we should be a little more honest and use a true class in the unit tests to be 100% sure. We’ll take the old 2 tests:

describe('sendEmailSafe when called', ()=> {
    const sendEmailStub = (options, callback) => callback(undefined, 'info')
    const sendEmailBadStub = (options, callback) => callback(new Error('dat boom'))
    it('should work with good stubs', ()=> {
        return expect(sendEmailSafe(sendEmailStub, {})).to.be.fulfilled
    })
    it('should fail with bad stubs', ()=> {
        return expect(sendEmailSafe(sendEmailBadStub, {})).to.be.rejected
    })
})

And change ’em to:

describe('sendEmailSafe when called', ()=> {
    class TransportStub {
        constructor(mailOptions) {
            this.mailOptions = mailOptions
        }
        sendEmail(options, callback) {
            callback(undefined, 'info')
        }
    }
    class TransportStubBad {
        constructor(mailOptions) {
            this.mailOptions = mailOptions
        }
        sendEmail(options, callback) {
            callback(new Error('dat boom'))
        }
    }
    it('should work with good stubs', ()=> {
        return expect(sendEmailSafe(new TransportStub({}), {})).to.be.fulfilled
    })
    it('should fail with bad stubs', ()=> {
        return expect(sendEmailSafe(new TransportStubBad({}), {})).to.be.rejected
    })
})

Notice once you go OOP, things get more verbose. Bleh.

Welcome to Part 4 where we show how to do concurrency which is a lot easier to get “for free” using pure functions, we compose both async and synchronous functions, we and utilize a test coverage report to where next to focus our refactoring and testing efforts.

Contents

This is a 6 part series on refactoring imperative code in Node to a functional programming style with unit tests. You are currently on Part 4.

• Part 1 – Ground Rules, Export, and Server Control
• Part 2 – Predicates, Async, and Unsafe
• Part 3 – OOP, Compose, Curry
• Part 4 – Concurrency, Compose, and Coverage
• Part 5 – Noops, Stub Soup, and Mountebank
• Part 6 – Next, Logging, and Conclusions

Compose Again

Assuming getUserEmail succeeded, we’ll have the user’s information so we can send an email. We now need to read the text email template to inject that information into. We’ll compose that readFile function we wrote. The existing code is imperatively inside the getUserEmail‘s then:

...
userModule.getUserEmail(get('cookie.sessionID', req))
    .then(value => {
        fs.readFile('./templates/email.html', 'utf-8', (err, template) => {
            if (err) {
                console.log(err)
                    err.message = 'Cannot read email template'
                    err.httpStatusCode = 500
                    return next(err)
            }
...

Let’s fix that and compose them together #connectDemTrainTrax:

...
userModule.getUserEmail(get('cookie.sessionID', req))
.then(userInfo => readEmailTemplate(fs)))
.then(template => ...)
...

Great! We even removed all the error handling as that’s built into our pure readEmailTemplate function.

Parallelism, Not Concurrency (Who Cares)

However, that’s one new problem; we need userInfo later on once we’ve gotten all the email info setup and ready. Since it’s only in scope for this function it’s now gone. One of JavaScript’s most powerful and taken for granted features, closures, we just threw out the window to remain “pure” for purity’s sake.

We can fix it, though, with one of JavaScript’s other features: non-blocking I/O. We can return 3 Promises and wait for all 3 to complete, and use all 3 values in the same function. It doesn’t matter if one takes longer than the others; Promise.all will wait for all 3 to be done, then give us an Array with all 3 values in order. If even 1 has an error, it’ll just pop out the .catch. This has 2 bad problems, but we’ll tackle that in another article. This also has the benefit of being faster in that we don’t have to wait for each in line, they all happen “at the same time Node style” which is not the same as “happening at the same time Elixir/Erlang or Go style” but that’s ok, we can get into the same dance club.

For now, we’ll refactor to:

...
Promise.all([
      getUserEmail(get('cookie.sessionID', req)),
      readEmailTemplate(readFile),
      mapFilesToAttachments(filterCleanFiles(get('files', req)))
    ])
    .then( ([userEmailAddress, emailTemplate, fileAttachments]) => ...)
...

Now we’re talking. Loading from an external web service, reading from a local file, and a synchronous function call all can happen “at the same time”, and we don’t have to worry about how long each one takes. We use Array Destructuring to get our arguments out. Note they come in the same order we put the functions into the Promise.all.

We now have our user’s email address, the text template to inject information into, along with the file attachments used for both in the same function scope.

Synchronous Compose

One thing to nitpick. Sometimes you refactor FP code for readability purposes, not just for the mathematical purity reasons. In this case, check out the 3 levels of nesting:

mapFilesToAttachments(filterCleanFiles(get('files', req)))

In imperative code, if you see if/then statements nested more than 2 levels deep, that tends raise concern. Developers are sometimes fine with creating that code to ensure they truly understand the different cases in playing with ideas, but once complete, they don’t like LEAVING it that way. Nested if statements are hard to read and follow. If you DO follow them, you can sometimes get a rush or high in “figuring it out”. That’s not the goal, though; nested if’s are considered bad practice.

For FP, deeply nested functions like this have the same problem. It’s compounded by the fact we attempted to use verbose names for the functions to make what they do more clear vs. short names. This ends up making the problem worse.

For Promises, it’s not so bad; you just shove them in the .then. But what about synchronous code?

You have 2 options:

1. Simply wrap in them in a Promise; most promises except for a couple of edge cases are fine getting a return value of a Promise or a value as long as the value isn’t an Error.
2. Use Lodash’ flow function, or Ramda’s compose.
3. Use the pipeline operator.

Sadly, at the time of this writing, the pipeline operator is only at Stage 1 for JavaScript, meaning it’s not even considered a possibility for inclusion in the ECMA Standard yet. None of this code is asynchronous so we’ll use the Lodash flow (I like Ramda’s compose name better).

Let’s put the functions in order, just like we would with a Promise chain:

const filterCleanFilesAndMapToAttachments = flow([
  get('files'),
  filterCleanFiles,
  mapFilesToAttachments
])

Note the use of get('files'). The get function takes 2 arguments, but we only supply 1. We know it’s curried by default, meaning it’ll be a partial application if we just say get('files'); it’s waiting for the 2nd argument. Once it gets that, it’ll search for the ‘files’ property on it, else give undefined. If it DOES find undefined, filterCleanFiles will just spit out an empty Array, and mapFilesToAttachments will spit out an empty Array when you give it an empty Array. Otherwise, they’ll get the good Array full of files, and both of those functions will do their thang.

See how we use curried functions that create partial applications to help compose other functions? I know… for you guys, not a good pickup line, but you never know, she me might be a Data Scientist who digs Scala. Or she’s lit and you look good and anything you say doesn’t really matter at that point. Either way, it’s alllll good.

Now to use that composed function, we take what we had:

Promise.all([
  getUserEmail(get('cookie.sessionID', req)),
  readEmailTemplate(readFile),
  mapFilesToAttachments(filterCleanFiles(get('files', req)))
])

And replace it with our composed function:

Promise.all([
  getUserEmail(get('cookie.sessionID', req)),
  readEmailTemplate(readFile),
  filterCleanFilesAndMapToAttachments(req)
])

Much better eh? Speaking of lit, I’m feeling that hard root beer right now, but I STILL remember we need to unit test our composed function. Let’s do that… and with confidence because we already have 3 unit tested pure functions, and we composed them together with a Lodash pure function. DAT CONFIDENCE BUILDING! Also, you MAY have to install config and nodemailer: npm i config nodemailer and then require them up top. Also, depending on the order of functions, you may have to move some functions around giving while we’re creating pure functions, they’re defined IN an imperative way, and so order matters. i.e. you have to create the const app = express() first before you can app.post.

describe('filterCleanFilesAndMapToAttachments when called', ()=> {
    it('should give an attachment from a request with clean file', ()=>{
        const reqStub = {files: [{scan: 'clean', originalname: 'so fresh', path: '/o/m/g'}]}
        const result = filterCleanFilesAndMapToAttachments(reqStub)
        expect(result[0].filename).to.equal('so fresh')
    })
    it('should give an empty Array with no files', ()=>{
        const reqStub = {files: [{scan: 'dirty south', originalname: 'so fresh', path: '/o/m/g'}]}
        const result = filterCleanFilesAndMapToAttachments(reqStub)
        expect(result).to.be.empty
    })
    it('should give an empty Array with undefined', ()=>{
        const result = filterCleanFilesAndMapToAttachments(undefined)
        expect(result).to.be.empty
    })
})

:: chink chunk :: NIIIIICCCE!

Composing the Promise Way

You can also just compose the Promise way, and they’ll work for Promise based functions as well as synchronous ones allowing you to use interchangeably. Let’s first delete all the no-longer-needed imperative code:

let attachments = []
files.map(file => {
  if (file.scan === 'clean') {
    attachments.push({ filename: file.originalname, path: file.path })
  }
})

value.attachments = attachments
req.attachments = attachments

And we’ll take the remaining mix of synchronous and imperative code, and one by one wire together:

let emailBody = Mustache.render(template, value)
let emailService = config.get('emailService')
const transporter = nodemailer.createTransport({
  host: emailService.host,
  port: emailService.port,
  secure: false,
})

const mailOptions = {
  from: emailService.from,
  to: emailService.to,
  subject: emailService.subject,
  html: emailBody,
  attachments: attachments,
}

transporter.sendMail(mailOptions, (err, info) => {
  if (err) {
    err.message = 'Email service unavailable'
    err.httpStatusCode = 500
    return next(err)
  } else {
    return next()
  }
})

You hopefully are getting trained at this point to start noticing “globals in my function”. Note our current line of code is:

let emailBody = Mustache.render(template, value)

But nowhere in the function arguments do we pass the render function to use. Let’s quickly modify the ever-growing Express route function signature from:

const sendEmail = curry((readFile, config, createTransport, getUserEmail, req, res, next) =>

to:

const sendEmail = curry((readFile, config, createTransport, getUserEmail, render, req, res, next) =>

We’re already in a Promise at this point, so we can return a value here, or a Promise and we’ll be sure we can add another .then if we need to. One trick VSCode, a free text and code editor by Microsoft, has is highlighting variables. Before we shove this rendered email template variable in the Monad train, let’s see if anyone down the tracks needs it. We’ll select the whole variable, and watch how VSCode will highlight usage of it as well:

Crud… it’s a ways down, AND it’s mixed in with this emailService thing. Let’s highlight him and see where he’s grouped:

This’ll be tricky. Good news, rendering the email and loading the email service configuration can be done at the same time. Let’s keep that INSIDE the Promise now until we feel comfortable we no longer need the userEmailAddress, emailTemplate, fileAttachments in scope. A lot more pragmatic people would be fine with keeping the code this way, and using JavaScript’s built in feature of closures, and move on with life. However, imperative code is harder to test, and results in LONGER code vs. smaller, pure functions that are easier to test. You don’t always START there, though. It’s fine to write imperative, then write “kind of pure” and keep refactoring your way there. That’s part of learning, figuring our the idea of how your code should work, or both.

...
.then( ([userEmailAddress, emailTemplate, fileAttachments]) => {
    return Promise.all([
      render(render, template, userEmailAddress),
      getEmailService(config)
    ])
    .then( ([emailBody, emailService]) => ...
...

And we’ll clean up the code below to use our pure functions first imperatively:

...
.then( ([emailBody, emailService]) => {
  const transportObject = createTransportObject(emailService.host, emailBody.port)
  const transport = createTransport(transportObject)
  const sendEmailFunction = transport.sendEmail
  const mailOptions = createMailOptions(
    emailService.from,
    emailService.to,
    emailService.subject,
    emailBody,
    fileAttachments
  )
  return sendEmailSafe(sendEmailFunction, mailOptions)
})

… and then refactor to more functional:

...
.then( ([emailBody, emailService]) =>
  sendEmailSafe(
    createTransport(
      createTransportObject(emailService.host, emailBody.port)
    ).sendEmail,
    createMailOptions(
      emailService.from,
      emailService.to,
      emailService.subject,
      emailBody,
      fileAttachments
    )
  )
})
...

Note the fileAttachments comes from the scope higher up. The sendEmailSafe function requires a nodemailer transport. We create that from our function that creates the Object from the emailService. Once created we need that sendEmail function to pass it to the sendEmailSafe so we just immediately go .sendEmail in the first parameter. The createMailOptions is another function that simply creates our Object from the emailService object, the rendered via Mustache emailBody, and the virus scanned fileAttachements. One last touch is to remove the squiggly braces {} as we’re no longer writing imperative code, and the return statement as Arrow functions have an implicit return when you remove the squiggly braces.

This last part is left over from the callback:

), reason => {
      return next(reason)
    })

Typically you defer Promise error handling higher up the call stack; meaning, “let whoever is calling me deal with error handling since Promises that call Promises have their errors propagate up”. That’s fine, so… we’ll delete it.

After all that refactoring, here’s what we’re left with:

const sendEmail = curry((readFile, config, createTransport, getUserEmail, render, req, res, next) =>
    Promise.all([
      getUserEmail(get('cookie.sessionID', req)),
      readEmailTemplate(readFile),
      filterCleanFilesAndMapToAttachments(req)
    ])
    .then( ([userEmailAddress, emailTemplate, fileAttachments]) => 
      Promise.all([
          render(render, template, userEmailAddress),
          getEmailService(config)
        ])
        .then( ([emailBody, emailService]) =>
          sendEmailSafe(
            createTransport(
              createTransportObject(emailService.host, emailBody.port)
            ).sendEmail,
            createMailOptions(
              emailService.from,
              emailService.to,
              emailService.subject,
              emailBody,
              fileAttachments
            )
          )
        )
    ))

Coverage Report

Let’s unit test it; it’ll be hard because we have a lot of stubs, but we can borrow from the ones we’ve already created in the other tests. I’m not going to DRY the code at all in the tests as that would require too much brainpower at this point, but when you get an Agile Sprint or time to pay down technical debt, this is one of the stories/tasks you add to that list.

… before we do, let’s run a coverage report to see how much work we have cut out for us (we’re ignoring my fake npm module and the user stuff for now). Run npm run coverage && open coverage/lcov-report/index.html:

And the details around our particular function:

Status Quo at This Point

Wonderful; the only thing we need to test is the composition of those functions in Promise.all. Rather than create 20 billion stubs, and ensure they’re setup “just so” so the sendEmail unit test passes or fails, we’ll continue to our strategy of pulling out teency pieces, wrapping them in pure functions, testing those, repeat. Let’s start with the first Promise.all:

const getSessionIDFromRequest = get('cookie.sessionID')
const getEmailTemplateAndAttachments = curry((getUserEmail, readFile, req) =>
  Promise.all([
    getUserEmail(getSessionIDFromRequest(req)),
    readEmailTemplate(readFile),
    filterCleanFilesAndMapToAttachments(req)
  ]))

Then we’ll unit test the getEmailTemplateAndAttachments (he’ll end up ensuring we’ve tested the new getSessionIDFromRequest):

describe('getEmailTemplateAndAttachments when called', ()=> {
    const reqStub = {
        cookie: { sessionID: '1' },
        files: [{scan: 'clean', originalname: 'so fresh', path: '/o/m/g'}]
    }
    const getUserEmailStub = () => 'email'
    const readFileStub = (path, encoding, callback) => callback(undefined, 'email')
    const readFileStubBad = (path, encoding, callback) => callback(new Error('b00mz'))
    it('should succeed with good stubs', ()=> {
        return expect(
            getEmailTemplateAndAttachments(getUserEmailStub, readFileStub, reqStub)
        ).to.be.fulfilled
    })
    it('should succeed resolve to having an email', ()=> {
        return getEmailTemplateAndAttachments(getUserEmailStub, readFileStub, reqStub)
        .then( ([userEmail, emailBody, attachments]) => {
            expect(userEmail).to.equal('email')
        })
    })
    it('should fail if reading file fails', ()=> {
        return expect(
            getEmailTemplateAndAttachments(getUserEmailStub, readFileStubBad, reqStub)
        ).to.be.rejected
    })
})

And we’ll then swap it out for the raw Promise.all:

const sendEmail = curry((readFile, config, createTransport, getUserEmail, render, req, res, next) =>
    getEmailTemplateAndAttachments(getUserEmail, readFile, req)
    .then( ([userEmailAddress, emailTemplate, fileAttachments]) => 
      Promise.all([
          render(render, template, userEmailAddress),
          getEmailService(config)
        ])
        .then( ([emailBody, emailService]) =>
          sendEmailSafe(
            createTransport(
              createTransportObject(emailService.host, emailBody.port)
            ).sendEmail,
            createMailOptions(
              emailService.from,
              emailService.to,
              emailService.subject,
              emailBody,
              fileAttachments
            )
          )
        )
    ))

… and then re-run coverage. Just run npm run coverage and you can refresh the coverage in the browser:

As you can see, coverage isn’t going to let us off that easy. That’s ok, we can re-use these stubs for the final battle. Let’s do the last Promise.all.

describe('renderEmailAndGetEmailService when called', ()=> {
    const configStub = { has: stubTrue, get: () => 'email service' }
    const renderStub = stubTrue
    const renderStubBad = () => { throw new Error('intentionally failed render')}
    it('should work with good stubs', ()=> {
        return expect(
            renderEmailAndGetEmailService(configStub, renderStub, 'template', 'user email')
        ).to.be.fulfilled
    })
    it('should resolve to an email', ()=> {
        return renderEmailAndGetEmailService(configStub, renderStub, 'template', 'user email')
        .then( ([emailRendered, emailService]) => {
            expect(emailRendered).to.equal(true)
        })
    })
    it('should fail if rendering fails', ()=> {
        return expect(
            renderEmailAndGetEmailService(configStub, renderStubBad, 'template', 'user email')
        ).to.be.rejected
    })
})

And swap it out:

const sendEmail = curry((readFile, config, createTransport, getUserEmail, render, req, res, next) =>
    getEmailTemplateAndAttachments(getUserEmail, readFile, req)
    .then( ([userEmailAddress, emailTemplate, fileAttachments]) => 
      renderEmailAndGetEmailService(config, render, emailTemplate, userEmailAddress)
      .then( ([emailBody, emailService]) =>
        sendEmailSafe(
          createTransport(
            createTransportObject(emailService.host, emailBody.port)
          ).sendEmail,
          createMailOptions(
            emailService.from,
            emailService.to,
            emailService.subject,
            emailBody,
            fileAttachments
          )
        )
      )
    ))