Sadly, the 1.0 docs for Polymer are quite awful. Even more frustrating is most of the Google links still point to v0.5 content, including Github repos that don’t all point to the newer repos. For example, go figure out how to know how to programmatically listen when a <paper-radio-group> has changed a selected item by the user clicking it… ready, go. Oh look, you found misery! I reckon this is because they are still trying to keep ahead all of the Web vendor API changes, general code optimizations, and just resource issues. Not sure the size of the Polymer team. On a whim this morning I checked their blog and finally found what I was looking for since I gave up on the documentation (which was quite great in 0.5).

While I primarily work in raw JavaScript all day and sometimes Angular, after messing with Polymer for about 2 weeks now, I really really don’t get the allure of React. When I was messing with Angular 2 months ago in Dart using the new component syntax, it was immediately apparent how quickly you could build UI’s. It felt like Flex. Just focusing strictly on the UI, not caring about models and controllers and blah blah blah MVC whatever. Just git-r-done.

Polymer feels the same way; I never messed with the Dart version so just doing the JavaScript version, but even so, it’s a ton like Angular 1.0, just less… weird. That, and you can include styles WITH your component vs. “Uh… Bootstrap will… like… handle that somehow. I don’t know man, go ask the designer, I’m just the coder.”

Took me a few weeks to get re-acclimated to Material Design. Bootstrap provides you with SO many components including general typography out of the box that work quite quickly, whereas Material Design, or even if you start with Material Design Lite, there’s still a ton of onus on you to have SOME designer chops, or have a designer as a friend that can help you.

Anyway, glad to know now how to make my CSS styles follow DRY principles.

JavaScript is now the assembly of the web. Rather than wait on language innovation in the design by committee web standards world, many choose to use more advanced languages and tooling to develop in, that then compile to JavaScript.

Transpilers, also known as source to source compilers or transcompilers, are used to translates source code to another form of source code. They are popular and numerous, not just for JavaScript, but also for C, and many other languages. In this article I cover why they are valuable for JavaScript developers, give details on the popular ones in the community, and go over the use case workflow for each.

I’ve chosen 6 of the most popular JavaScript transpiled languages used for client web application development:

1. CoffeeScript
2. TypeScript
3. AtScript
4. ClojureScript
5. ES6 to ES5
6. Dart

What

Transpilers allow you to convert your source code in one language, either 1 file or many, into another language. In JavaScript’s case, you typically write in 1 language such as CoffeeScript, and using a build system such as Gulp or Grunt, it will compile your .coffee files to .js JavaScript files. Most also generate source maps for more streamlined debugging.

Another use case is the use of annotations where you write comments in the code that have special meaning and reflection API’s are utilized at runtime to inspect the code’s meaning and act accordingly. Examples include signifying what type the class is without the need for inheritance which is what AtScript does on top of the TypeScript compiler. Another is describing the static typing contract the function has in JavaScript using the Google’s Traceur compiler. They both will interpret those code comments and generate JavaScript with those special meanings applied.

Why

Transpilers in the non-JavaScript past were utilized to convert large code bases to a newer language (such as Python 2 to 3), an old API to a new one, or runtime platform where doing it by hand was impractical and error prone.  In the JavaScript world, there are a variety of reasons.

The primary reason is that JavaScript innovation is slower than open source and private offerings. It takes a long time to get an ECMAScript feature from committee standard to implemented in all green field browsers. You can use some of those features, and more, in the other languages, today, and still have it run on the standards based, ubiquitous web platform.

Here are few more often cited reasons:

1. JavaScript evolution slow compared to other languages
2. open source and private languages evolve faster, have more features
3. sometimes less code
4. better code
5. organized code
6. more language features
7. leverage existing talent
8. leverage existing libraries
9. more efficient runtime
10. developers are more productive
11. future proof
12. clearer code

History

Source to source compiling is not a new concept.

In 1981, Digital Research had a program that converted ASM source code for the Intel 8080 processor into A86 source code for the Intel 8086.

In 1983 Cfront was the original compiler for C++ which compiled C++ to C.

In 2000, the LLVM project was created with a collection of modular and reusable compiler and toolchain technologies. It is used for compiling a variety of of gcc 4.2.1 languages to C. A few of these include Lisp, ActionScript, Ada, D, Fortran, OpenGL, Go, Java, Objective-C, Python, Ruby, C#, and Lua are just a few.

In 2006, Google released Google Web Toolkit, called GWT. It enables developers to build cross browser web applications in Java.

Many other source to source compilers have been written for everything to Fortran to Python to either convert an older language to a newer platform, or from an old API to a new one.

Source to source compilers are an old, accepted practice.

CoffeeScript

Why

CoffeeScript attempts to expose the good parts of JavaScript and hide the bad parts. Created by the guy responsible for Underscore and Backbone. Benefit is class syntax, fixes JavaScript’s operators, and has a terse & safe syntax.

Features

• classes with inheritance and super keyword which abstracts away Object.prototype
• fixes == equality to work as expected
• arrow functions and other shortened initializers for terse code
• shortened loop syntax with optimized output and additional keywords
• literate coffeescript option which allows code to be written in Markdown
• lexical soping of variables (i.e. no need for var keyword)
• terse if else then statements, no parentheses or brackets required
• rest based parameters
• array slice syntax similiar to Python
• implied return statement, no need to type it
• statements converted to expressions inside closures
• uses english for equality operators and compiles down to safe usage
• provides existential operator for checking existence of variables
• Array/Object destructuring which allows for multiple return values from functions
• Function binding for ensuring this scope remains intact
• safter switch statements
• chained comparrisons
• String interpolation
• multiline and block strings
• Block Regular Expressions (borrows from Perl’s /x)

Pros

• Compiles to nice JavaScript providing good escape hatch if team ends up not liking it
• Basically JavaScript, not much to learn
• terse code
• fixes many of JavaScript’s basic language problems with safety included
• makes classes useable in JavaScript

Cons

The only con is some developers wish to code only in CoffeeScript, and others do not wish to code in CoffeeScript. These are often quite strong feelings.

TypeScript

Why

TypeScript provides opt-in static typing, class and module syntax, and is based on the ES6 spec. It was created by Anders Hejlsberg of Turbo Pascal, Delphi, and C# fame. Benefit is strong typing with ability to use against 3rd party libraries, package & classes, and backed by Microsoft.

Features

• define strongly typed globals via declare
• integrate strong typing to 3rd party libraries via declaration files
• strong typing for both built-in primitives, classes, and interfaces
• type inference with basic primitives, classes, and interfaces
• interfaces can be applied to basic Objects and classes
• function literals
• class constructor initializers
• default parameters
• multiple constructor signatures
• class inheritance with super keyword
• package support via private and public modules
• compiler supports generating AMD or CommonJS modules
• arrow syntax for function binding
• generics
• … and a lot more.

Pros

• Compiles to nice JavaScript providing good escape hatch if team ends up not liking it
• strong typing allows larger projects and teams to find bugs early w/o writing unit tests
• classes are made usable in JavaScript with inheritance
• Offers package management for classes, unlike CoffeeScript
• Like CoffeeScript, a lot of syntactic sugar

Cons

Typing is not enforced at runtime, only compile time. Also, same problem with CoffeeScript: developers either love or hate it.

AtScript

Why

Angular’s growth in the Enterprise has shown the limits of JavaScript, thus the natural evolution is to a more mature language: TypeScript. However, TypeScript is missing application development features, so the Google Angular team is working with the Microsoft TypeScript team to modify the language & compiler. Specifically class annotations and runtime assertions resulting in an enhanced TypeScript called AtScript.

Features

Same as TypeScript. The two additional ones are class metadata annotations and runtime assertions for type safety at runtime. The class metadata annotations are used to denote the Angular component types without having to inherit from a specific class or work within a specific lifecycle API. While it may seem Angular specific, any framework can benefit from class and variable metadata annotations consumed at runtime. Aurelia is another MVC framework that supports AtScript.

Pros

All the Pros of TypeScript including:

• your types being enforced at runtime
• can swap out the assertion library with your own
• easier & more readable way to declare Angular component types compared Angular 1.0’s methods
• backed by both Google and Microsoft

Cons

Same as TypeScript, including at the time of his writing, these features are in active development.

ClojureScript

Why

Clojure has gained in popularity because it is a functional language that runs on the JVM with concurrency. This same language can be utilized to build client side applications through compilation to JavaScript via the Google Closure compiler (note the “s” vs. a “j”). Touting the same benefits of JavaScript with Node, being the same language used on the client and server sharing some of the same libraries.

Features

• functional language, dialect of Lisp
• runs on the JVM
• supports concurrency written in a synchronous way like Go
• optional type hints and type inference to avoid Java reflection
• immutable data structures
• same language used on the server

Pros

• one language for front end and back-end development
• supports streams and promises native to the language
• immutable data structures are the norm with mutable support enabling easy concurrency with rewind testing support
• this makes working with React’s shouldComponentUpdate method insanely efficient and succinct to write
• Clojure is a Functional Language

Cons

• No popular MVC frameworks ported yet (ie Angular, Backbone, Ember). Although one can utilize Purnam, most of these MVC frameworks are OOP based and encourage Clojure Models to be some sort of atom as opposed to being Functional based and using Collections and Streams as the Models.
• … like the Pro, Clojure is a Functional Language

ECMAScript 6 to ECMAScript 5

Why

Utilize ECMAScript 6 features of the future, today, while supporting the older browsers of yesterday. Still writing JavaScript, no need to learn a new language. Get a leg up on writing the JavaScript of tomorrow to ensure code bases are easier to modify/reuse in the future. The most common way to do that today is to utilize the 6to5 or Google Traceur transpilers.

Features

• classes – no need to agree on an OOP JavaScript pattern or library
• modules – standardized way of organizing code like Java packages
• module loading supporting dynamic loading, execution sandbox, and module cache manipulation
• private variables
• tighter code through arrow functions, object literals
• template strings – safer string creation without a library
• destructuring with soft fail pattern matching
• constants
• iterators like Java and “for of” support
• generators for creating green threads for long running processes
• Map, Set, and Weak versions – updated collection data structures commonly used
• Proxy support
• Symbols, more powerful Strings for lookups in Maps and Objects
• easier to sub-class native classes
• various new API’s to String, Number, Object, and Math
• native Promises
• reflection api for easier Proxy creation and other higher level tooling
• tail calls – safer recursion

Pros

• most ES6 features
• team doesn’t have to learn a new language, it’s simply JavaScript compiling to JavaScript
• you’re coding the future, today
• classes and modules help code quality because of consistency, a core problem in JavaScript applications
• module loader helps solve common performance and security issues with JavaScript applications
• various features that aren’t yet implemented are emulated via shims/polyfills that perform well enough
• using technology based on web standards ensuring longer term support

Cons

• existing JavaScript libraries and frameworks need to be ported, brought into build system, or maintained separately
• generators cannot be effectively polyfilled, though polyfills do exist
• modules and their loaders do not solve module problem thoroughly… yet. They are client specific, avoids Common JS, and makes sharing code with Node/io.js challenging. Solution is to use Browserify and put ES6 classes on module.exports.
• most JavaScript coding standards do not yet exist for ES6.
• This includes syntax, code quality, and coverage tooling as well.
• For many teams not yet used to developing in more enterprise languages (i.e. heavy use of class and module keywords in large codebase), this can be a huge change for them. This is not necessarely the “safe option” because “it’s just newer, standards based JavaScript”

Dart

Why

For Java developers, a mature OOP language with functional elements on both client and server. For full stack developers using JavaScript frameworks on the client and Node on the server, a better language & compiler for larger scale applications. For gaming developers, much of the same benefits they get with TypeScript, but with more functional elements, cross browser API’s abstracted for them, with a more powerful compiler to optimize their code. All get a Google run package management system and gain immense runtime speed when code runs in the Dart Virtual Machine. Dart is the full package.

Features

• Dart language – like JavaScript, object oriented, optionally strongly typed, generics, classes, packages, libraries, promises, streams, and other functional features including basics like no undefined, and == that works like you’d expect it to.
• Dart SDK – libraries for client and server, cross browser DOM and other HTML5 API abstractions.
• pub – package management system, much like npm or bower
• Dart Editor – Eclipse based IDE with code hinting, intellisense, and basic refactoring abilities. Includes a GUI on top of pub so you can add/remove dependencies as well as create you own.
• Dart VM – runtime utilizing .dart code to run in a virtual machine for great speed. Currently implemented in Chromium with future plans to bring to Chrome and hopefully implement in other browsers.
• dart2js – advanced compiler that converts dart code to JavaScript. Includes generic and data type optimization, tree shaking, and various metadata/reflection/annotation capabilities. You can optionally leave in runtime assertions for data typing.

Pros

• language better suited for larger teams & applications built from scratch specifically for building client and server web applications
• mature compiler built with web applications in mind, often generating faster executing JavaScript
• Built & run by Google
• Core part of the Angular 2.0 build lifecycle
• A suite of tools for building web applications
• Where Chrome only is an option (kiosks, enterprise networks, etc) Dart VM helps gain fast speed, yet not having to use C/C++ for Native Client development
• same language & libraries used on client and server
• easier runtime module creation with retained strong-typing for application optimization (as opposed to custom grunt/gulp builds + ocLazyLoad or require.js packaging).
• supports debugging directly from an Android device

Cons

• Unlike CoffeeScript, TypeScript, ClojureScript, and ES6 to ES5, Dart does not provide an easy escape hatch. The transpiled JavaScript is built for a specific runtime. Thus it is not easy to “try before you buy” and thus salvaging your work.
• This Dart JavaScript runtime specific code causes challenges when attempting to integrate into existing applications. There is JS interop, however, that allows Dart JavaScript to non-Dart JavaScript communication

Conclusions

There are many other transcompiled languages, platforms, and compilers not covered here. The desire for developers to improve various other parts of JavaScript and customize to their team’s workflow has resulted in a plethora of choices for teams. While it’s not always clear which choice is the best for the team, what is clear is that transcompiling is a normal practice in JavaScript development, integrates nicely with most build systems, and will continue to improve.

Special Thanks

Thanks to Alan Shaw and Daniel Glauser (hire Daniel) for the impromptu Google Hangouts to give my peers and I the lo down on Clojure. Thanks to David Nolen for answering my questions and hooking me up with some good Clojure resources to learn. His blog is crazy and awesome, btw, encourage you to read even if you’re a client dev.  Thanks to Joel Hooks, Robert Penner, and Ben Clinkinbeard for clarifying some some finer points and sharing code. Thanks to beer for helping me get through Clojure training.

Title artwork by Ry-Spirit.

Conclusions

As you can see, each messaging system has pro’s and con’s. Also, each can, and often is, used in tandem with each other. They aren’t always stand alone, and many are used in the same code base. As you better understand the basics of messaging systems, you can more easily make your Object Oriented code bases easier to encapsulate. It also makes it easier when using Design Patterns to have disparate parts of the code talk to each other in a flexible way.

Why Write This and Other Random Notes

For context, the whole reason I wrote this article is I had no idea why Promises were so lauded by JavaScript developers. I also had no clue what Streams were and if they were valuable or not. Coming from Flash and Flex, Events worked just fine as a core messaging system for years. There were a few who used Signals, and a few pumped strangeness across the global, and not well known loaderInfo.sharedEvents.

Many articles online explained what Promises and Streams were, but none explained why they mattered, nor why I should use them over some other system. WHY AM I DOING THIS!? The Stream guys are the worst. They use computer science terminology that’s extremely obtuse & unapproachable, and they assume these words plastered by themselves is enough to justify their usage. “The Ancients used Stargates, or course we’re supposed to use them as well”.

Before writing this, I didn’t realize CustomEvent’s only worked with DOM elements. Polymer’s done a good job at this via their fire method, no doubt, but… wow. Talk about having it rough. Those poor JS devs.

I also didn’t realize JQuery, to solve the above, gave them callbacks for async programming, and basically introduced the concept of Promises to the JS world. I was in my spoiled world of events that worked in visual components or in data components. What I didn’t realize was that when it came to orchestration, the JQuery crowd actually had a much easier time of it. Once you have more than 2 async events, you start having to hunt through the code to mentally map what function fires what event and what function handles that. When you start chaining those things… good luck. With Promises, it’s all centralized, making the code much easier to read. AS3 is basically like TypeScript or Dart; and thus we were all willing to have more verbose code because of strong typing’s value. Anything that can help make it less more verbose, like Promises, would of been a god send in some of the larger enterprise code bases I’ve been thrust into fixing.

Sadly, Flash Player garbage collector worked extremely differently post Flash Player 8, so things like local variables can’t just magically “exist” like they do in JavaScript. This was so deeply ingrained, it made it quite hard for me to “get” why Promises actually work, and how to clean them up.

I was also confused why Angular’s $q differed so much from the official Q. I then realized they wrote their own version of it. Some of the more important features, however, of long stack traces would of made many people’s lives easier. The biggest frustration I’ve had learning the ins and outs of Promises this year was completely useless stack traces. As as spoiled Flex developer, I’ve watched Chrome in the past 3 years insanely improve it’s runtime errors and stack traces. None of which seems to make working with nested Promises that much easier when the developers use anonymous functions everywhere. This has completely changed the way I code Promises in JavaScript if not using a Promise library (most of my work is Angular’s $q so when creating Promise chains, I avoid anon functions like t3h plague if I can, and try/catch throw like a mofo).

After doing a lot of orchestration in larger code bases from a data perspective, Promises really started to make sense once I started doing the same work in JavaScript. Remember, this was before Node, and all Factory / data orchestration had to be done on the client WITHOUT something like Cocoa’s CoreData. Once you put Node into the picture, this gets a TON easier. No matter how complicated the Java/.NET developers make your life, you can get yourself the 1 API call you need, with no parsing, in about 2 hours. That. Is. Hot. Shit.

Now, I still don’t get Streams on the client beyond gaming, or an improved event system for component frameworks (like Polymer or Angular UI). I’m attempting to recreate Final Fantasy 6’s battle system as a research exercise. In doing so, I’ve played with various Stream facets in Dart as well as Frappe (which works more like Bacon/RX).

Although research is still on going, for client side, Streams are way easier to deal with than Events, GUI or not. It took me 3 months to get comfortable, but once I got it, there’s basically no going back to Events if it I can help it. I’m not sure how to resolve that for Polymer as I’m a huge fan of the project, want to use it, but I’ve already lived through the world of Events and am over it. One of these days I need to hack into it and see if there’s an easier way to convert “fire” and it’s ilk to Streams vs. just using the common adapters on top. Implementing pause and cancel is where the real work would be.

It’d be nice of application frameworks would implement these, such as Angular UI. A few hours playing with Polymer’s ObservableList and you’ll be hooked. Array’s are too low level for creating truly dynamic application UI controls.

That said, in the data layer, especially when Node comes into play to support your front end REST API, I don’t have many disparate async events. The value for Streams beyond the GUI layer doesn’t seem helpful, and smaller promise chains work fine.

In Node, however, the complete opposite. If I had Node back in my Flex consulting days, things would have been much simpler. Waiting weeks, sometimes never, getting the API’s to be simpler, moved a ton of technical debt on the client side to compensate for this, with large Facade layers over the data. Orchestrating many different API’s, often from different servers with different request and response formats is ripe for Streams to make them easier to work with. Even better is that Node is in a language I already know: JavaScript. I can use the same libraries in the same build stack on the same CI server.

And yet, even here I struggle to convince others. My guess is they just haven’t spent 6 months like I have playing with Streams and Promises simply to write a blog post.

I will say the need for a Stream spec is pretty clear. It makes all the Promise libraries immediately understandable in how they work. You have the exact opposite problem with Streams. Sadly, I don’t think Streams will catch on as hard as Promises do. Promises solve such a core problem in ajax and only have a small native footprint to code whereas Streams require a TON of things.

First off, most browsers, even nightlies, barely scratch the surface of what RX/Bacon/Dart have for functional array methods. We’ve seen how long it takes browser vendors to acknowledge their value and add them in natively hence why lodash or underscore still reign supreme in most peoples tool belts.

Second, they rely a common expectation of how Streams work and the consensus, especially on cold vs. hot, and Observables vs. Properties, just ain’t there either yet. Until that mess settles down, the libraries will be the go-to places to embrace Streams. Well, except in Dart where they are in there out of the box. It’s still unclear to me in Polymer where Events end and Streams begin. Need more time to play.

One small note, I used a variety of text editors to write this post over the past 6 months. After about 4 pages, even if the most recent version of WordPress with the improved single page scrolling, it’s still brutal and version control is confusing. I tried Mou, my current favorite Markdown editor, but after about 10 pages performance quickly went down. I gave a few random text editors for the Mac a try and finally settled on Scrivener. It’s not safe with my HTML tags as it keeps upper casing b and pre tags, and breaks hyperlinks with it’s weird double-quote choice, but I’m sure it’s a setting I don’t know yet. Either way, at nearly 20,000 words, ZERO performance problems. Faster to boot up and use compared to Word for Mac. Props to Cliff Hall for the recommendation.

Special Thanks

Thanks to Joel Hooks, Paul Taylor, Dan Schultz, and the Dart community for answering my questions.

Art:

• Ao2 40th day: Antennas
• grad lvl

<< Part 6 – Streams

Streams

Streams are merely Array’s that emit change events through a callback when items are added to it.

That’s it.

People try to make them sound magical, complicated, or like some core savior technique of programming life. They’re not. They’ve been around long before me in networking code. They’re how Github visualizes different branches. They’re just Arrays that emit change events to a callback you can filter before you get the event. You don’t even have to use events; you could just use any value you want. Most implementations also play nice with Promises.

Streams are like Events, except you can use Array methods on them BEFORE the event is given to the callback handler, and listeners can pause further events from being emitted. They take the asynchronous flattening abilities of Promises, the filtering abilities of functional Arrays (think all those Array methods on Underscore), and the 1 to many broadcast abilities of Events. They have a flattening power of their own once you start merging streams together. When used with Promises, you can work around the race conditions much like you do for Events.

As a side note, Node’s current version of Streams at the time of this writing is actually based on EventEmitter.

For example, you’ll use a command to get the weather in JSON:

String url = 'http://api.openweathermap.org/data/2.5/weather?q=Richmond,VA&units=imperial';
EventStream links = new EventStream(HttpRequest.request(url, responseType: 'json').asStream());
links.listen((data)
{
	print(data.response);
});

In Dart using Frappe, you use listen to hear about when streams fire events. Here’s RxJS doing the same thing:

function getWeather()
{
    return $.get('http://api.openweathermap.org/data/2.5/weather?q=Richmond,VA&units=imperial')
    .promise();
}
var currentWeather = distinct.flatMapLatest(getWeather);
currentWeather.subscribe(function(data)
{
   	console.log(data);
});

And in Bacon.js, another rad, and less intimidating, JavaScript Stream library:

function getWeather()
{
    var promise = $.get('http://api.openweathermap.org/data/2.5/weather?q=Richmond,VA&units=imperial')
    .promise();
    return Bacon.fromPromise(promise);
}
getWeather().onValue(function(data)
{
	console.log(data);
});

Like addEventListener, you pass in a callback to fire when they happen. Like events, listen will pass in the parameter the Stream emits. While events have the convention to pass events as the single and only parameter, many who are used to smaller promises, or putting streaming code on top of existing promise code, more primitive values along with multiple values may be common, and that’s ok. JavaScript will work.

Strongly Typed Streams

Dart however will give you compile warnings if you don’t match the types. If you don’t use strong-typing and expect a certain type, you’ll get a runtime exception.

For example, this is good:

StreamController controller = new StreamController();
EventStream nameChanges = new EventStream(controller.stream);
nameChanges.listen((String someName)
{
	print("someName: $someName");
});
controller.add("Jesse");

This will give you a compiler warning:

StreamController controller = new StreamController();
EventStream nameChanges = new EventStream(controller.stream);
nameChanges.listen((int someName)
{
	print("someName: $someName");
});
controller.add("Jesse");

And this will compile just fine, yet blow up at runtime:

StreamController controller = new StreamController();
EventStream nameChanges = new EventStream(controller.stream);
nameChanges.listen((someName)
{
	int oldAge = 70 + someName;
	print("oldAge: $oldAge");
});
controller.add("Jesse");
//	Exception: Uncaught Error: type 'String' is not a subtype of type 'num' of 'other'.
//    Stack Trace:
//    #0      int.+ (dart:core-patch/integers.dart:15)

#inb4HaxeDoesntHaveThisProblem

The strong-typing helps both while coding, when you start merging/piping streams or using Composition. It also helps in helping the compiler create faster JavaScript at runtime. This helps when writing Dart for the client or for the server (just like JavaScript in the client and Node).

Streams Being ‘Done’ vs. Infinite

If you map your mind to Streams firing multiple times like an event, they are immediately familiar. Here’s Bacon creating an event stream (a Stream of Events, usually from something that spits out events from addEventListener type deal):

$('#someButton').asEventStream('click').subscribe(function(mouseEvent)
{
	console.log("mouseEvent:", mouseEvent);
});

“JQuery, find me this button hidden in the DOM soup, create a Stream from it’s click addEventListener, and let my anonymous function know about events as they are added in the Stream, kthxbai!”

Since Events work very similarly to Streams, most Streaming libraries have these kinds of converters to quickly make them work like Streams, yet retain the original Event interface. Lowers the barrier of entry.

Filtering

As previously mentioned unlike Events, you can filter them BEFORE you’re subscribe/listen function is fired. For example, when designing a game, I only want to know about events where a particular character is ready so I can redraw the UI:

initiativeStream.where((InitiativeEvent event)
{
	return event.type == InitiativeEvent.CHARACTER_READY;
})
.listen((event)
{
	print("character ready: ${event.character}");
});

You’ll notice I’m treating the initiativeStream here as an Array, using a where function. If you’re not familiar with uber functional Array stuff, check out Underscore’s definition of where or try your hand at the low-level implementations tutorial to understand why you even need or would want this stuff.

What if I didn’t have filtering abilities? You’d write crap like this:

initiativeStream.
.listen((event)
{
	if(event.type == InitiativeEvent.CHARACTER_READY)
	{
		print("character ready: ${event.character}");
	}
});

Bottom line, every time an initiative event is fired, only let me know about the ready events. Cool. However, anyone who has played RPG games know’s there is one issue with the above; I’ll hear about the Monster events as well. Again, without functional Array methods:

initiativeStream.
.listen((event)
{
	if(event.character is Player)
	{
		if(event.type == InitiativeEvent.CHARACTER_READY)
		{
			print("character ready: ${event.character}");
		}
	}
});

Machete don’t write nested ifs.

With functional Arrays, you can filter on filters. We’ll ensure it’s the correct event type, only from Players, and convert it so we get the player out since that’s all we care about:

initiativeStream.where((InitiativeEvent event)
{
	return event.type == InitiativeEvent.CHARACTER_READY;
})
.where((InitiativeEvent event)
{
	return event.character is Player;
})
.map((InitiativeEvent event)
{
	return event.character;
})
.listen((Player player)
{
	print("player is ready: ${character}");
});

Good stuff. Remember, though, like events, that listen callback will get called multiple times if more things come into the stream. The docs for the various libraries and runtimes are confusing in that many methods will go “when the stream is done”. Done is a strange word because if you never un-listen, is it REALLY done? The whole chain above will get executed if 5 seconds later when another IntiativeEvent is pumped through the system. In implementations like Dart’s, they actually can emit a ‘done’ type of event if you wish to know. However, for UI Streams like the Bacon one I gave above, it’s never really done because the user could click anytime they want.

Merging and Piping Streams

Many systems use Streams to link a series of inputs and outputs together with very little code. The most common are string operations on the command line in many Unix systems.

$ ps aux | grep conky | grep -v grep | awk '{print $2}' | xargs kill

To read that in Jesse Engrish, “List my running processes, throw that list to a search program called grep, and look for ‘conky’ within it, exclude all the lines that don’t match, use a program called awk to snag out the process ID (called a PID) I need from the string, and throw that PID at xargs kill which will stop the process from running. Notice in this case the pipe is literally a pipe vs. a “Stream.pipe()” function. You can imagine what happens if someone starts changing ‘conky’ and keeps re-running ps aux… continuously streaming pipes in hot action, sucka!

It should be noted that Node already has a way to pipe streams together which is more performant and shorter to write:

var errorsFromClient = getErrorPostStream();
var errorLogFile = fs.createWriteStream('error.log');
errorsFromClient.pipe(errorLogFile);

You’ll often “pipe” the data from one command to the next, whether manually doing it like I showed, or using the platform/libraries built in pipe command.

Streams can also indicate a “series of events”. As previously mentioned, Node.js uses streams much in this fashion using EventEmitters. They’re basically Events that have start, progress, and stopping points and are called Streams (api in flux at time of this writing).

Finally, Streams can be merged. One use case is when you’re building an orchestration on the Node server to have 4 API calls be 1 to make the client side code easier to write and not have to worry about authentication & parsing details. Another use case is flattening all the Events on a client that cause the same thing to happen.

For example, you know how when building a login form, either clicking the “Login” button will submit the form as well as hitting enter while within the password field? You can merge Streams to make that easier to read. Here’s a Bacon example:

var loginButton = $("#loginButton").asEventStream("click");
var loginField = $("#passwordField").asEventStream("click");
var loginStream = loginButton.merge(loginField);
loginStream.onValue(function()
{
	$.post('server.com/login');
});

The same technique can be used server-side or client side to flatten orchestrations when making many REST API calls. This example first loads the client’s location using a geoIP service (uses the server vs. navigator.geolocation) to identify the user’s latitude and longitude, ensures it’s not called more than every half a second, parses the result into a Point object, loads the weather passing in the latitude and longitude, and ensures the result isn’t the same as last time, else it ignores it:

GetLocation locationService = new GetLocation();
EventStream getWeatherFromGeoIP = new EventStream(locationService.asStream())
.debounce(new Duration(milliseconds: 500))
.map((HttpRequest request)
{
	return new Point(request.response.lat, request.response.lon);
})
.asyncMap((Point point)
{
	return getWeather(point.x, point.y);
})
.distinct();
getWeatherFromGeoIP.listen((num weather)
{
	print("It is $weather degrees.");
});

new Timer(new Duration(minutes: 1), ()
{
	locationService.getLocation();
});

If you didn’t catch it, check out line #10 (getWeather method) where the Stream supports asynchronous filters as well, specifically Promises. This is also an option where you can use Composition vs. Merging of Streams.

Here’s another example with a less linear approach where you can see different streams emitting events at different times, in this case, Monsters and Players attacking each other. One is controlled by AI, the other controlled by the user:

EventStream monsterAttacks = new EventStream(new StreamController().stream);
EventStream playerAttacks = new EventStream(new StreamController().stream);
EventStream attacks = monsterAttacks.merge(playerAttacks);
attacks.listen((AttackResult result)
{
	textDropper.addTextDrop(result.attackTarget, result.hitValue);
});

In the case of chat applications which will do a request/response in the background, but a real-time socket when you have the application active, like on your phone with HipChat. Two different application modes, your data comes out the same:

EventStream ajaxStream = new EventStream(new StreamController().stream);
EventStream socketStream = new EventStream(new StreamController().stream);
EventStream messages = ChatMessage.merge(socketStream);
messages.listen((ChatMessage message)
{
	querySelector("#chatField").text = message.text;
});

Note On Observables

While not always mentioned in some Stream API’s, an Observable is often included when describing Stream API’s. All they’re referring to is the Observer pattern applied over an Array, like a Collection. When you do an arrayInstance.push(), there are no change events. No one else knows that the Array changed unless you use a CustomEvent or pub sub to inform others that it changed. Thus, many use the Observer pattern to wrap Array’s to create Collections; similar API, except you know what changed, where, and how. Backbone’s Model and Collection classes are perfect examples of this.

Streams, however, represent the data coming into their Array IN ORDER, even when merged streams. This is why all the cray cray Functional Array stuff works on top, no matter how many Streams you use. Now, your listen function may fire only 1 time because your Array/Stream only has 1 data item. As that data “changes” however, you’ll get more events. Hence why they call it an Observable; somewhere deep in the API is the Observer pattern watching a piece of data, usually passed by ref if possible to emulate Promise state encapsulation.

You’ll also sometimes see Properties. These are sometimes like Observables depending on the library. Other times they’re like Promises that cache data.

Be aware of the context of Observable. This is one of the reasons Streams are so hard to grok because people often cite Observable in many different ways without defining what the hell it actually is. For example, the RX family calls Streams:

RxJS = Observables + Operators + Schedulers.

Except it’s not. That quote form the docs is actually talking about RX, not Streams. You could be easily fooled (I was) to think using a Stream library called RX would be defined as Observers, Operators, and Schedulers.

Streams ARE an Observable. Like the Observer pattern: “This thing changed, let the world know.” When you change a Stream, it lets the world know. You use functional Array methods to “operate” on that data. Operators == Functional Array Methods. And putting Schedulers there makes it sound like Streams are one third made up of these scheduler things. Except their not. It’s 1 plugin out of the 11 RX has… and that’s only for some advanced unit testing/functional testing mojo. Most of us ain’t going to use that.

If you think of a Stream as an Array that emits events when it changes, and thus is an Observer, you’ll get it. The 2 diverging extremes are “the Observable’s current value”, I.e. The latest value in the Stream. The other end of the spectrum is what I call them as; the Stream is just an Array of values, and you get them in order; so the Array’s an Observable. They’re both not technically correct because some Streaming libraries give you insane ballz control over Observables. Some like Frappe even go the Property route.

This is why specs exist after the innovation love has died down and developers have self-loathing when their toys start to get annoying.

Cold vs. Hot

When they start speaking really advanced about “cold vs. Hot”, don’t sweat it. All this means is something is Cold when it doesn’t dispatch events until someone listens. In a cold stream, if I click the mouse 50 times on a Hot button Stream, none of those Events are actually dispatched on the Stream until someone calls listen/subscribe. If it was a hot stream, she’d act like Events; the Stream would emit the events even if no one is listening.

Another example is reading text from a file. You won’t start receiving events for it until someone says, “Yo, read the this text file, and lemme know each line of text read as an event”. On demand type stuff like ajax calls, reading serialized data from window.localeStorage, querying a Model/Collection for the first time, etc.

Hot is the opposite, and works like Events; they’re fired whether anyone’s listening or not. This is for things that are happening whether you’re there or not. If you’re listener is late to the party, he’ll start getting new events from that point forward. If I click the mouse 50 times on a button Stream, and you only add a listen/subscribe callback after 25th has fired, you’ll only get 25 Events. Same as if you did 25 mouse clicks, then domElement.addEventListener(“click”, mouseCallback), then clicked 25 more times.

Dart streams are hot by default, whereas RX ones are cold unless you explicitly make it hot.

Stream Pros

The core goal with Streams is to reduce the amount of code you have to write, centralize the event handling, and filter the events all with Promise support. They do this by embracing functional Array filter methods, improving upon the Events API, support of Promises, with the ability to merge & pipe streams together.

Promises deal with 1 event, not many. Although you can chain and/or bubble the event forward, it’s still 1 specific event that starts each part of the chain. Streams deal with many events with many listeners.

Promises usually are a 1 time affair. With their caching, while you can restart the sequence with many Promises in a chain and it’s a lot faster, it’s still the same piece of data. Streams on the other hand represent new data each time an event occurs by default unless combined with Promises or Observables/Properties. For cold observables, this helps solve the Event race condition problem by using the fact Streams will not usually post events until someone has subscribed/listened. In the case of many, you can utilize Promises to help.

Most Event mechanisms are synchronous whereas Streams can support asynchronous operations as well, yet like Promises, be written in a synchronous way. While not 100% solving race conditions, it helps. Some API’s such as Dart’s provide options allowing the Stream to be synchronous or asynchronous. Event emitters can’t be paused by default, where in some Stream implementations, its assumed those who are listening to the streams can pause them. This means any consumer, anywhere, can pause the stream without having knowledge of what Stream their pausing, whether that Stream is merged or via Composition. Since you can get multiple instances, you can have multiple people with varying levels of control. With great power comes great responsibility. Those who use Streams are now responsible for managing this subscription. This is an additional instance that you must store and later destroy. Here’s an RX example:

var stream = getAStream();
var subscription = stream
  .subscribe(
    function (hotness) {
      console.log('Droppin it: ' + hotness);
    });
subscription.dispose();

Events make this simpler by just forcing everyone to get the emitter instance, and just manager the listeners directly from the emitter. Except in larger code bases, finding this instance isn’t so simple. Some frameworks in more dynamic languages will offload the listeners secretly through the Decorator pattern onto those who call addEventListener so that garbage collection can more easily deal with the references.

For larger applications, flattening the complexity with Streams is very similar to flattening a complex architecture through Composition and Facades. The difference with Streams is a common API to do so for both consumption of Events as well as the abilities to wire Streams together. In heavy GUI applications in Flash/Flex, we’d do this with Events, or some other messaging system like Signals.

They also scale better than Events in larger systems. As Stream / pub sub / Event systems grow, you start to run into the cognitive load challenge of keep track of all the different Event types and who dispatches them in your head including the data and types of data they contain. The difference with Streams is you have the ability to query both the data/Events you’re getting and operate on that data, all from a higher level structure. Using CSS classes and a DOM query syntax allows a more flexible UI. You can change & refactor things under the surface API, and the surface API still works. Streams are the same way. No matter what you modify, move around underneath, you can still have flexible queries up top to ensure your data is in the format you need when it arrives. For game developers this may seem like something horrible; deeply nested, disparate structures IN SUPPORT OF allowing disparate parts to flexibly talk to each other.

For applications developers on a deadline with ever changing API’s across different domains, however, different story. You can quickly see how “querying” your data and composing it into something more useful from smaller parts, all in synchronous looking code that has a small line count, sounds hot.

To be clear; this isn’t a panacea; a large code base still has a heavy cognitive load. Streams just help. Helping is good.

For unit testing purposes, it’s slightly easier in some cold Streams to write unit tests specifically because many Streams will not dispatch events until the 1st listener is subscribed. Some of this behavior is predictable in some API’s, thus you can write better tests to ensure the listener actually received and reacted on the API regardless of what transformations it and the data composing went through. This all continues to work regardless of how many Streams you merge.

Stream Cons

There is no spec right now. As such, they are quite confusing to learn. This also inadvertently makes them quite intimidating. Unless you have a game development, animation, or networking hardware background, visualizing how streams scale in larger systems can be quite hard. Bottom line, the lack of a spec means a lack of consensus on how these things are implemented. That’s really the only valid con for Streams, right now. Everything below is just nitpicking.

Streams also assume you already have good experience with Functional Array programming, and can answer why you’d use a map, where, and zip on the same Array… yet applied to Events. Wait, what? If you don’t, it can be hard to extricate what is part of a Stream, and what’s part of the library. You have to “get” why lodash/underscore are awesome libraries first before you can see how they can “improve” events.

Many also assume you already are comfortable with Promises. Many developers are comfortable with Promises, but still create nested Promise chains vs. flattening them. Most Streaming libraries embrace Promises to make dealing with asynchronous code in a multitude of events easier. This can be a real mind-bender. It takes time to get used to coding this way. If your teammates don’t, they’ll get mad when they need you debug the code if they’re not up to speed yet.

Managing, or rather, choosing NOT to manage Stream subscriptions can get you into trouble memory wise. Most Promises you see on the web don’t really show how to clean them up. Most Stream examples are the same way. Just be sure you realize their writing pseudo code to get the point across and assume you will do your due diligence by disposing of your subscriptions.

Lastly is the merge/pipe mania. Many will use Stream features simply because they a Stream features. New, fresh, shiny, hot. Once they get bored, they’re just another way of doing events or async Promises. Be wary of people who want to merge/pipe all the things. If they go all Katamari Damacy with Streams attempting to eat and consume MVC, you know they’ve jumped the shark. That said, I encourage you to do this in your spare time off of project/client work. The various libraries approach things differently so it’s neat to learn those differences.

<< Part 5 – Promise Deferred | Part 7 – Conclusions >>

Promise and Deferred (or Futures and Completers)

We’ve glossed over asynchronous coding up to this point. Many from languages which have reasonable event API’s (ActionScript) to extremely nice ones (C#), it may not at first look like a problem, or even appear to be an edge case. Coming from ActionScript, it took me years to get comfortable, and understand why, Promises were helpful. Also, many in those languages either create, or have facilities that help create, orchestration code to help mitigate common asynchronous coding issues.

But first, why asynchronous code? Synchronous code in GUI’s is challenging for 2 reasons.

The first is most UI runtimes have the code execution and drawing in the same thread. The Elastic Racetrack metaphor is often used to explain how this works. This means we have an our code doing calculations and handling user events (like clicks) done at the same time as the code that draws and redraws things to the screen. At first this seems reasonable given how fast computers are. However, as code complexity grows inline with sophisticated UI such as dynamic graphs, hardware accelerated animation, and complicated redrawing of multiple DOM elements for text and graphics in the browser… It’s a lot to do in a few milliseconds so the user doesn’t notice. That, and this all assumes the code has no orchestration to optimize this redraw part of the track, and the data processing part of the track such as invalidation strategies, also known as deferred rendering.

Second is network or I/O (input output) calls. For example, I want to show some text on this page; I know you’re German speaking based on the browser string, so I make a network request to my CMS (content management system) to get the German content. This may take a second or many seconds. In the synchronous or “blocking” world, the code would stop executing at that point and wait for that request to the server to spit back the text we need. The user’s mouse would work, but no clicks, drags, touches, double-clicks, rollovers, touch indicators, keyboard usage would function, nor would the user be able to see the UI visually update; the app looks like it’s broken. Even if it had a loading icon, it’d be paused. The UI is “locked” or “blocked”.

Cool, so things that require us to go outside of JavaScript for data like the server or the local disk/database, we’ll just make those asynchronous, which the Browser developers have done for us. The same holds true for server development platforms like Node and gaming ones like Corona and Unity. Other blocking languages like Google’s Go don’t have this issue/feature.

This, however, creates 2 new problems specifically how to write code that “waits” for things and how to chain these scenarios without making the code unreadable or hard to follow.

Callbacks are the first way. We’ve seen for simple code, they are the fastest, easiest, and most flexible way to do that. “Whenever you’re done, homey, call me”. Below is how JQuery does asynchronous loading of data from t3h interwebz:

$.ajax({
  url: "http://jessewarden.com",
  success: function(data)
  {
  	console.log("loaded data:", data);
  }
});

“Load data from here, and call this function when it worked, passing me the data as the 1st parameter, kthxbai.”

Events were another attempt to do that for multiple people who wish to know about some async event. You can utilize the same API, and your code looks the same, and the key here is that it works the exact same whether synchronous or asynchronous. The pseudo code below shows synchronous click code with asynchronous image loading code. Notice how the API looks and works the same, yet the first callback function fires immediately when you click in the same call stack, whereas the 2nd fires after some random amount of time, never in the same execution stack.

function onClick(event)
{
   console.log("clicked");
}
button.addEventListener("click", onClick);

function onImageLoaded(event)
{
   console.log("image loaded");
}
image.addEventListener("onload", onImageLoaded);

That’s all well and good until you start utilize class composition; meaning you utilize many objects that have events or callbacks in a parent class. One convention is to register all the event listeners up top, and then define the event handlers below, in the order they were added. If the event handlers utilize functions, well, you just either put them nearby and make finding the functions in order no longer work, or put them below the callbacks and force those reading/debugging to jump around your class.

For the sake of the reader, I’ll keep the below brief, but tripling the lines of code below it is something I commonly see:

define(["jquery",
		"Underscore",
		"com.company.project.views.InventoryView",
		"com.company.project.views.ScheduleView",
		"com.company.project.views.ProfilePopUp",
		"com.company.project.events.EventBus",
		"com.company.project.utils.InventoryUtils"], 
		function($,
					_,
					InventoryView, 
					ScheduleView, 
					ProfilePopUp,
					EventBus,
					InventoryUtils)
{
	var SomeClass = function()
	{
		init: function()
		{
			// TODO: use _.bindAll
			_.bind(this.onClick, this);
			_.bind(this.onInventory, this);
			_.bind(this.onScheduleTask, this);
			_.bind(this.onToggleProfile, this);
			_.bind(this.onToggleEditProfile, this);

			$("#SubmitButton").click(this.onClick);
			$("#InventoryButton").click(this.onInventory);
			$("#ScheduleTask").click(this.onScheduleTask);
			$("#ToggleProfile").click(this.onToggleProfile);
			$("#ModalProfileEditForm").click(this.onToggleEditProfile);
		},

		onClick: function()
		{

		},

		onInventory: function()
		{

		},

		onScheduleTask: function()
		{

		},

		onToggleProfile: function()
		{


		},

		onToggleEditProfile: function()
		{

		},

		someHelperMethodReferencedRandomlyAbove: function()
		{

		}
	};
	return SomeClass;

});

It looks nice at first, but then becomes a long class you have to constantly scroll up and down through. It gets worse as those event handlers and methods either grow in size, number, or have comments attached to them. You’ll notice the above is using UI events; this can be deceiving if some of those callbacks are actually listening to global pub sub messages. It gets confusion which functions are responding to global events and which are responding to local UI ones. Your unit tests also start to get pretty long as well. Languages which handle scope like CoffeeScript and TypeScript can help a little, but you still end up with the same problem: lot’s of code that’s hard to pin down to localized functionality; i.e. “all the stuff this class does about this one thing is right here” vs. spread out all over the place.

Now, remember, there are those of us from other languages/platforms where this was the norm and considered ok (even if you didn’t have a deadline). Worse, anonymous functions in some languages needed to be scoped to SOMETHING else the garbage collector would eat them leading to less terse code capabilities without a language change (i.e. Lambdaâ€s in t3h newer Java). JavaScript and Lua’s friendless to lamda/anonymous functions in terms of the Garbage Collector not eating them in Mark and Sweep phases means they are a common way to reduce code length in larger code bases at a tiny cost to function creation time (anonymous functions vs. class member / Object.prototype ones).

Where async code really starts to break down is with data orchestration and animation: 2 common asynchronous operations central in UI development. Additionally, we have no idea if any of those events already happened before the class executed. The UI events that are user gestures (the user clicked on something), we don’t care, but the model or other UI asset changes, we do. This is why you’ll often see those event handlers called at the bottom of the init to “set the class up in case she’s late to the data party”.

Jake Archibald has a great, and often used example, that wonderfully illustrates how the above provides a false sense of security, specifically around loading of an image. I’ll copy that example here verbatim:

var img1 = document.querySelector('.img-1');

function loaded() {
  // woo yey image loaded
}

if (img1.complete) {
  loaded();
}
else {
  img1.addEventListener('load', loaded);
}

img1.addEventListener('error', function() {
  // argh everything's broken
});

2 important points in this example.

First, we burden the consumer, the developer using the class that emits events, to check if it already fired, and if so, manually fire her event assuming the event handler accepts a function call without an event parameter; JavaScript does, CoffeeScript(?), TypeScript, and Dart you have to specifically mark those functions to have the events as optional parameters.

Second, my SomeClass example above doesn’t have any error handling at all. In most apps, at best, they log errors with little to no user experience thought given around those scenarios. That said, very rarely do any of them react to errors that happened before their instantiated for the user (ie a graph is shown, but the data that was supposed to be loaded before it was shown failed to load). No one wants to spend time creating user experiences around a broken server, but it’s important. Pitching clients to spend a lot of their application development budget on designing user experiences around things breaking seems counterintuitive to quality software craftsmanship. Thus, this is often left to developers who have little time for such concerns, hence why error handling is often such a horrible experience in many applications. It’s not their responsibility, its the Designer(s). Catch 22, sucka!

When you take into account both events already happening before you’ve arrived on the scene as well as errors, you can see how the code above gets much more verbose, and complicated to follow.

One common work around at least in MVC applications, is to allow View classes to be extremely friendly to null values in the Models. That way, if the data isn’t loaded let, they can show either a loading or something to the user while it loads or fails, otherwise they can immediately draw the model when they’re shown. The thinking goes “someone else will handle orchestrating everything in the correct order later” and even “somehow allow this to be easily refactored”. This can work if your Models aren’t that complicated, but still makes the code quite verbose.

The second place the SomeClass example starts to break down is when you have multiple asynchronous events that depend on each other.

var userView = {
	userModel: null,
	imageTag: null,
	showAvatar: function()
	{
		var imageURL = userModel.avatarURL;
		imageLoader.get(imageURL, function(bitmapData)
		{
			imageTag.showBitmapData(bitmapData);
		});
	}
};

You’ll notice in the above simple JavaScript View, if both the model data for the user is not yet loaded from the server AND the HTML and CSS needed to display the image which also needs to be loaded, the View won’t work. 2 options here: either check for null and do nothing assuming “you didn’t load the data for me idiot, I’m not showing anything, but will be nice to you and not throw errors” or put the burden on the developer using the View to get both of those things ready ahead of time.

Many MVC and web application tutorials out there assume you have good REST services that give you only the data you need, and you only need to make 1 call for the entire app or per section the user see’s with a good set of data caching code behind it. Backbone is notoriously seductive in this respect regarding who it’s Models are built around this with no guidance given to those who aren’t actually hitting REST services.

If you’re a JavaScript client developer, and you have a bad ass Node developer who has your back and y’all can communicate often as requirements change… you have it quite well, my friend.

This is simply not how enterprise applications work in the real world, although I’m seeing this change once .NET/Java business tier developers realize their jobs aren’t in jeopardy because of Node. You’re often building atop of .NET or Java backends that are several years old, many of which were in support of page based web sites or desktop applications long before you got involved. Getting the back-end team to refactor this for the best user experience takes an act of God, especially if you’re one of hundreds of applications that consume their services.

Unless you got Node into the mix, you’ll be making many calls, often multiple times, before a particular View/Screen can be shown to the user… in addition to your other data orchestration duties. Sometimes you’re lucky if you abstract these multiple calls away into a single class that dispatches 1 event. If you’re not, you usually have the two common problems: multiple classes that only have pieces of the data you need that continually needs refreshing or lots of data on the client you have to parse, such as the case in dashboards and graphing applications. This isn’t just ajax calls we’re talking about either; this could be JavaScript code running on the client that that needs to run long running parsing code that may or may not include an ajax call, image and other media assets, and various multiple authentication calls from a variety of servers.

For example, many financial applications often pull data from a variety of back-end data sources to show on 1 screen. Whether these are different sub-domains, or a single REST/SOAP/RPC server, they could still all work slightly differently and take varying amount of times to respond. Sometimes you have to call them in a certain order since each successive call gives you a token/id/session cookie needed for the following call for the actual data, some of which you can’t cache because the nonce tokens (prove you’re a user who’s logged in) can expire quickly for financial data.

Let’s show an example where you first need to get some user info that needs to be obtained to snag the user’s current token (which changes often) so you can then get info on a particular financial transaction (something the user bought) which is then used to charting data related to it’s id + type (which the server has conveniently stashed a bunch of huge JSON files on a Heroku friendly CDN somewhere so you can parse on the client to show charts).

However, that last call has a TON of data, so you’re using a polyfill (a library that allows you to use future HTML5 functionality, today) that supports WebWorkers (threads in JavaScript) so you can parse the data safely and attempt save to LocalStorage to avoid the slow latter part of the process again if the user views a particular transaction again later. I’ve also had cases where the type dictates which type of data we have, thus requiring another parsing call and a basic fork in the path of how you parse the code. I’ve also left out the fun parts of debugging unknowingly neutered ArrayBuffers… but lets keep it simple (lulz).

Quite a mouthful yet common in the screwed up world of Enterprise Software.

var currentClientToken = null;
var clientID = null;
var transactionID = null;
var graphWorker = null;
var done = false;

function getCompleted()
{
	return this.done;
}

function getClientToken(clientID, transactionID)
{
	this.done = false;
	this.clientID = clientID;
	this.transactionID = transactionID;
   $.get("/getClientInfo", clientID, this.onGetClientTokenSuccess, this.onGetClientTokenError);
}

function triggerGenericError(error)
{
	EventBus.trigger("GetChartingInfo:error", error);
}

function onGetClientTokenError(error)
{
	this.done = true;
	this.triggerGenericError(error);
}

function onGetClientTokenSuccess(data)
{
	try
	{
		var clientVO = ClientFactory.get(data);
		this.currentClientToken = clientVO.token;
		$.get("/getTransactionInfo", 
			{token: this.currentClientToken, transactionID: this.transactionID}, 
			this.onGetTransactionInfoSuccess, this.onGetTransactionInfoError);
	}
	catch(err)
	{
		this.triggerGenericError(err);
	}
}

function onGetTransactionInfoError(error)
{
	this.triggerGenericError(error);
}

function onGetTransactionInfoSuccess(info)
{
	try
	{
		var transactionVO = TransactionFactory.get(info);
		var currentTransactionURL = "https://sub.company.com/project/graphingcdn/" + transactionVO.id + "/?clientID=" + this.currentClientToken;
		$.get(currentTransactionURL, this.onGraphDataSuccess, this.onGraphDataError);
	}
	catch(err)
	{
		this.triggerGenericError(err);
	}
}

function onGraphDataError(error)
{
	this.triggerGenericError(error);
}

function getGraphWorker()
{
	if(this.graphWorker == null)
	{
		this.graphWorker = new Worker("GraphFactory.js");
		this.graphWorker.onmessage = function(event)
		{
			this.onGraphDataParsed(event.data);
		};
	}
	return this.graphWorker;
}

function onGraphDataSuccess(graphData)
{
	getGraphWorker().postMessage(graphData);
}

function onGraphDataParsed(graphVO)
{
	if(graphVO != null)
	{
		this.done = true;
		EventBus.trigger("GetChartingInfo:success", graphVO);
	}
	else
	{
		// parsing failed
		EventBus.trigger("GetChartingInfo:error", error);
	}
}

… to recap our problems, we now have:

1. lots of code
2. you have to scroll to find where an event is handled
3. finding event handlers for events registered within event handlers makes it even harder to follow
4. using if statements to handle potential race conditions
5. burden to ensure sequence can only start once with ability to stop it at any point
6. forced to remove event listeners to help garbage collection
7. code not fully shielded from uncaught exceptions
8. burden of error handling can make API unwieldy

WAT DO!?

Science has shown that less code equals better software. Through 1 study at least.

Enter the Promise (known as Futures in Dart). They are a way to make asynchronous code look and feel like synchronous code.

Our “use” of the above is turned into 8 lines of code (6 if you cuddle, but Machete don’t cuddle):

getGraphData().then(function()
{
	EventBus.trigger("GetChartingInfo:success", graphVO);
},
function(error)
{
	EventBus.trigger("GetChartingInfo:error", error);
});

Now, using Underscore and some refactoring using your module system of choice, you could refactor the above to be similar without Promises:

EventBus.on("GetChartingInfo:success", function(graphVO)
{
	drawIt(graphVO);
});
EventBus.on("GetChartingInfo:error", function(error))
{
	console.error("GetChartingInfo:error:", error);
});
getGraphData();

9 (7c) lines vs. 8 (6c); not bad. However, not apples to apples either. They do completely different things and the Event way doesn’t guarantee the same order in the same stack like Promises do. Subtle, but painful for n00bs to debug when they don’t know what’s going on. Heck, it’s painful for me!

The promise way only has 1 callback. The Events have multiple potential ones.

The events must be registered as listeners first before the call to getGraphData happens. If the events are not registered before getGraphData is called, you do not get the data. While getGraphData may internally cache the data, you must call it again after your listeners are registered to get this data. For smaller applications, this is easy; for larger ones, this is more challenging. It also puts a burden on those types of classes to make a caching decision sooner to make an easier API to consume.

The Promise does not care if you call it before the value(s) internally have been fetched from the server yet or not; you can call it anytime and get the same result. There is never a race condition problem using Promises as opposed to Events.

Also, by default, most Promises assume you’re caching in RAM by default based on the last return value if you follow the Promise/A spec.

API wise, it should be noted that the Event by default allows multiple responders to hear about a success or failure, whereas the Promise is only executed with 1 callback. This is neither good nor bad, just something be aware of when comparing the callback nature of Promises compared to Events.

A minor point, but the events now have a message chosen. You now need a way to track that message. If you just use a magic string, fine, but for large applications you tend to use some form of enumeration or constant class to help both management, strong-typing & code hinting for IDE’s that support it, and to ease the developer’s need to View all possible messages in a central place without a grep. Now a decision must be made; where do these enumerations go? In the class dispatching as statics/constants? Is it 1 class that supplies all global pub sub events for the entire application? Each has their pro’s and con’s, but both include more code, cognitive overhead, and crap for you to deal with.

Lastly, most Promises & promise libraries will execute on the next tick whereas most event, pub sub, and streaming libraries do not. Yes, you can add a setTimeout(func, 0), but that’s a hack on top of an API most assume works, and codes their API as, synchronously. There are a variety of use cases that aren’t described very well so I’ll cover that below regarding expecting order of execution to work, whether in synch or async code. This is why many pub sub libraries have extra error handling on removing listeners because if it’s in the middle of a dispatch, you can accidentally prevent others from getting the message.

Now, Jake’s article covers all you need to know about how Promises work, and Q’s documentation is pretty legit as well to help you create Promises as well as work with libraries that aren’t Promises out of the box.

Done Yet?

Promises, whether ES6 or Q, store the state of whether the Promise is done or not internally. You don’t have write code to do that; it’s part of them implementing the state machine part of the Promises/A spec. The Event option, however, has to allow the “done” boolean to be managed amongst the many callbacks.

This internal state management has a positive effect on sub-sequent calls with the same data, specifically getClientToken, getTransactionInfo, loading of the transaction JSON data, and receiving the parsed data from the WebWorker. For Promises, they’ll simple cache the data and re-deliver it usually immediately in the next tick (next time the stack is unwound then wound).

In the Event abstraction, you’ll have to do that yourself. In both cases, you’ll have to utilize whatever local storage option you wish to use (cookies, local storage, app cache, etc), but this functionality of locked data is built into Promises; you don’t have to code it, it’s built in + expected behavior. This includes multiple calls that could utilize multiple Promises internally. In our case, that’s about 4 Promises that each has the built-in ability to store data if it hasn’t changed.

You can also refactor asynchronous code that is not a Promise to be a Promise. That is one way developers have attempted to solve the above problems.

Solving Lots of Code

Promises are made part of async API’s. As such, instead of first registering for an event before calling something that will trigger it, you instead just call the method and the Promise is the returned value. That’s 2 lines of code to 1. Like callbacks.

This has larger ramifications, though, down the line when you’re async call is part of a series of async call allowing them to be chained as well as put into groups where they all must complete before another operation can succeed. This further cuts down on needing to define event handlers.

Scroll to Find Code

No more scrolling, it’s defined inline. The call and the responses are all next to each other.

Nested Event Handler Registration

This problem isn’t solved immediately with Promises. You can get into situations where people do not return Promises from async code, and they end up doing work in the success handlers. Through refactoring you can flatten these deeply nested chains. Thomas Burleson has a great example where he shows both.

Handling Race Conditions

Part of the spec for Promises is to solve this issue. Promises will automatically call their success or error method if the event has already occurred. This, again, reduces the amount of code you need to write when consuming async services. Additionally, it adds flexibility to call them in any order, at any time you need to; no need for orchestration/setup code to ensure everything is “ready for the data events that will fire and YOU NEED TO BE THERE FOR”. You instead react. This is part of where the “reactive” part comes in Functional Reactive Programming.

Finally, Promises have an internal state machine convention that they only ever complete once; so you don’t have to worry about your error or success methods firing more than once. They are not like events in this aspect which are known to fire many times.

This is one of the main problems that Events and Pub Sub run into when the application starts to grow in size and developers.

Aborting Sequence

Promises do not solve the abort sequence; i.e. you cannot easily start a long async Promise or chain of Promises once they’ve started. De-reffing the parent in hopes the child will get garbage collected isn’t sure fire, and adding API’s to a Promise based class is a more prudent option, although, more work in the case of many Promises through Composition.

More advanced Promise API’s such as Dart’s StreamController and RX.js’ Backpressure have the API’s you need to wrap Promises with the ability to stop async sequences, or even pause the events being emitted from the emitter. However, both incur a completely different paradigm of programming so it’s not as simple as “oh, I’ll just use this awesome library/x-compiler…”. We’ll get into that in the Streams section.

No More Cleanup Code

Promises, if they’re values are non-global and the chains are local to the function block, will get eaten by garbage collection if you de-reference them.

From a memory standpoint, this can be a blessing or a curse. Promises are ensured to only execute one time, and cache the value each additional call assuming the inputs don’t change, all while retaining the same code execution order. However, they basically act like a getter; they cache the data they are referencing as a form of Observers, albeit with 1 listener to start. Note, the Observer pattern is different than what stream API’s refer to regarding Observables. People will use the word “observable” to either refer to something that emits change events like the Observer design pattern, or a watchable property/stream in something like RX.js/Bacon.js. For large data sets, images, etc. you may not wish to leave this stuff laying around in RAM.

Check out this post (plus the comments full of lulz) from Drew Crawford talking about how brazenly just caching all data in Promises isn’t always a good idea.

Shielded From Exceptions

In the native implementations, Promises are built around the concept of success and failure. This includes built-in support for if an exception is thrown in a constructor it calls the error callback. Additionally, you can interpret successful responses from the server as a mistake and cause an error condition safely.

That said, the burden of good error handling is still on the developer in case an error does bubble up through a deep Promise chain, you want a good idea of where it came from. The same goes for good synchronous error handling through tried and true try/catch blocks within more complex Promise operations. Looking through the stack trace to work your way back down is still required in some cases which, again, can be mitigated with an effective logging solution. Theoretically yield used within a Generator function could help mitigate some of the error hunting as well.

Also, take note some Promise libraries take the positive initiative to allow you to get stack traces that are… you know… actually useful in debugging such as Q (check out longStackSupport = true) and Bluebird’s longStackTraces. these types of debugging features are NOT in Angular’s version of $q, and it’s Promise API. Do the hard work of being proactive yourself with good try/catch error messages and logging.

Also note, many Promise libraries will allow you to apply multiple catches to differentiate between the various types of async errors which is a wonderful practice you should endeavor to do.

Next Tick

Some Promise libraries will make a big deal about “nextTick”. The reason is most people when writing synchronous code expect the same order of execution no matter how many times the code is run. While Promises aren’t technically synchronous code, you write it as such.

For example, we all expect the function below to print out 1, 2, 3 no matter how many times it’s called:

function dreamMachine()
{
	console.log("1");
	console.log("2");
	console.log("3");
}

We also expect the function below to print out 1, 3, 2 no matter how many times it’s called:

function dreamMachine()
{
	console.log("1");
	Q.fcall(somePromise).then(function()
	{
		console.log("2");
	});
	console.log("3");
}

This is one feature Promises have above simple Pub Sub because your callbacks are fired in the same order regardless of how long the response takes, and if the data is cached or not.

Those who get mad about it are the ones who want faster code, typically server-side Node developers. The nextTick is often at a minimum a setTimeout(runTheThenFunction(), 0). This’ll significantly warp benchmarks that make the code look slow whereas in reality the code isn’t actually using CPU.

However, it IS a problem in that the code still takes more time and can significantly increase unit test suite test runner time. There are various Promise/A+ spec compliant libraries that will give you a flag to turn this off. This is for developers who know exactly why you’ll get 1, 2, 3 in some situations above. If you don’t know why that can happen, leave the setting off, heh!

Be aware, Node.js has a more efficient nextTick with configuration options.

Another use case is asynchronous initialization. Some objects/classes have significant amounts of asynchronous setup they have to do. You want an opportunity to be aware of some of these operations, say the loading of images in gaming Sprites, or using a server-side logging that has JSON file based configuration values is must load and parse before being used. Incidentally, this makes writing invalidation routines on top of existing GUI code a big easier.

Finally, in the case of networking operations, you don’t always want to impose knowledge of order of operations on the user. Instead, let them set and call things in any order, and let the class figure out the true order on the nextTick/frame.

For example, imagine if JQuery’s ajax didn’t have object initialization as it’s API. Instead of:

$.ajax({
  url: "http://jessewarden.com",
  success: function(data)
  {
  	console.log("loaded data:", data);
  }
});

It’d be:

var operation = $.ajax();
operation.url = "http://jessewarden.com";
operation.dataType = "html";
operation.type = "GET";
operation.success = function(data)
{
	console.log("loaded data:", data);
};
operation.load();

You not only have to learn an API, but the order in which the methods are called just to make a simple GET call? What if even more code later continues to modify, or even overwrite the operation’s parameters?

Using nextTick, calling order doesn’t matter. Internally, the actual XHR call won’t be made until the next frame anyway, so the ajax operation can handle setting everything up in the order it needs. This follows OOP encapsulation principles, and makes life easy on the developer.

Promise Pros

Promises significantly reduce the amount of code you have to write for asynchronous operations where you’d traditionally use callbacks or events.

They also localize it vs. spreading it out like events and callbacks are want to do. Just be sure you practice flattening some Promise chains so you don’t end up with the opposite problem of Promise chain hell vs. callback hell.

They also make chaining multiple asynchronous operations together much easier to read and debug.

They help reduce race conditions when the Promise already has a cached value, and merely calls the callback immediately. For low memory items, this also has the side benefit of caching data in memory for expedited calls later on, especially for those classes/Views initialized later in the application lifecycle. They also make writing asynchronous libraries easier on the consumers because they can call their setup calls in any order knowing the actual resolving of them will occur on the next frame.

Finally, you can write your code and not have to care something is synchronous or asynchronous; it looks and is written in a synchronous way.

Promise Cons

For those who are new to Promises, as I mentioned, they end up creating callback hell using Promises. Rather than creating their own operations that return Promises, they instead merely use Promises and continue to nest them. Again, check out Thomas Burlesson’s great example where he shows how you flatten deeply nested Promise chains. Once you practice, you can quickly flatten these trees as you create them. Eventually you’ll start writing methods that return Promises by default making your life easier. Like once you memorize a for loop’s syntax, you never think about it again.

For low memory situations on mobile, you could get in trouble when caching the data in Promises. While you’re supposed to follow the Promise/A spec and ensure Promise’s fullfilled state stays true and never changes the data, care must be taken to ensure you aren’t keeping these around unnecessarily, especially on mobile devices. This isn’t necessarely indicative of Promises, rather caching in general, which is a commonly hard programming problem. They are just brought up since most Promises handling async operations are often around data, your want to cache it.

Caching isn’t straightforward either. Some libraries help in this regard, but basic programming for by value vs. by reference apply. When getting values you’re basically assuming the developer doesn’t change them if they’re by ref. Cloning has performance impacts as well for deeply nested objects. Following the Promise/A spec with regards to not changing the value is often based on convention and isn’t always easily enforced. Don’t get me started on the insanity of sharing ArrayBuffers across WebWorkers; shiz is magic death.

<< Part 4 – Publish Subscribe | Part 6 – Streams >>