Type Driven Development: Pipeline

Part 19 – Pipeline

This is a series of posts I’m writing about using types as another tool in software development, Continuous Delivery, & keeping LLM’s honest. They’re also a design & refactoring tool, a communication tool, and reduce how many tests you have to write.

Parts

1. Part 1 – Branded Types
2. Part 2 – Product Types
3. Part 3 – Union & Discriminated Unions
4. Part 4 – Non-Empty Collections
5. Part 5 – Indexed Types
6. Part 6 – unknown vs any
7. Part 7 – Result
8. Part 8 – Schema
9. Part 9 – Total Function
10. Part 10 – Errors as Values
11. Part 11 – Property Tests
12. Part 12 – Type Proofs
13. Part 13 – Exhaustiveness Checking
14. Part 14 – Parse, Don’t Validate
15. Part 15 – Anti-Corruption Layer
16. Part 16 – Opaque Types
17. Part 17 – Maybe
18. Part 18 – Smart Constructors
19. Part 19 – Pipeline

Once you have total functions:

type Parse = (s:string) => Person[]
type FilterCows = (p:Person[]) => Cow[]
type Format = (c:Cow[]) => stringCode language: JavaScript (javascript)

You need some way to build larger functions. You could write imperative code, but then you have all these temporary variables w/scoping issues, nested if statements, error handling, & manual return statements which double as control flow.

function formatCows(s) {
  // can't use const, default to a Maybe
  let persons

  try {
    persons = parse(s)

  // why can't parse handle this?
  } catch(err) {

    // persons' is not undefined, trust me bro
    const cows = filterCows(persons!)
    const teamCow = format(cows)
    // stop running the code here
    return teamCow
  }

  // uh, empty I guess?
  return ''
}Code language: JavaScript (javascript)

The solution is to create pipelines; wiring your total functions together into a larger function. You may have done this w/Arrays which have a fluent API:

JSON.parse(json)
.filter( p => p.isCow )
.reduce( (acc, cow) => acc + `${cow.name}\n`, '' )Code language: JavaScript (javascript)

Pipelines are the same. Most Functional Programming languages (and PHP… wat) have the pipeline operator to make this more concise:

function pipeline(s) {
  return JSON.parse(s)
  |> JSON.parse(%)
  |> %.filter(p => p.isCow)
  |> formatNames(%)
}Code language: JavaScript (javascript)

However, you can use the native JavaScript Promise so you don’t have to build your own pipe functions like Effect/Lodash/Neverthrow do:

function pipeline(s) {
  return Promise.resolve(s)
  .then(JSON.parse)
  .then(p => p.isCow)
  .then(formatNames)
}Code language: JavaScript (javascript)

The pipeline is responsible for taking the output of the first function, then automatically calling the function it is connected to with that output as the input. It repeats this for all the pipes; what comes out is the final output.

Promise.resolve(s) // output json string
.then(JSON.parse) // output an Array of Person objects
.then(p => p.isCow) // Array of only Cows
.then(formatNames) // string with Cow names formatted on newlinesCode language: JavaScript (javascript)

You can see how it removes scoping issues, nested code, & helps w/a missing or misplaced return keyword, but still keeps visually the order of operations. You can then use this larger function in even larger functions.

For validators can use this to accumulate all errors for the form, not just what failed and exit early; helps in richer UI form validation. Streams & sinks can be built to efficiently parse lots of data. The compiler ensures these pipes “fit” correctly together, e.g. “this functions output type is the correct input type for this next function” and so on.

This is the main construct you use to build “Total Applications”. That’s why when you read the Effect-ts hello world docs, they talk about “running a program” vs. just a function.