August 23, 2017

Fullstack Academy, W3-D3: Implementing Promises From Scratch

Fullstack Academy, W3-D3: Implementing Promises From Scratch

Today we revisited promises and this time dove deep into their implementation.

Our exercises involved writing our own version of promises modeled after the ES2015 version, including the constructor function along with most of its public static and instance methods and its private internal methods. The result was a verion of promises that behaves nearly identically to the ES2015 promises, except our implementation doesn't have truly private methods to hide their essential internal mechanics like the JavaScript implementation does. We simply denoted the intention for those properties and methods to be "private" by prefacing them with ._internal.... Otherwise, it's a solid working implementation!

The point of the exercise, of course, isn't that we'll need to know how to build our own promise library from scratch (there are plenty of excellent ones already), but to help demystify how promises work in order to allow us to use them in our programs with greater intelligence and confidence. After completing this project, I can honestly say that goal has been realized. The "magic" of promises is gone–in a good way! They now seem as familiar and useful to me as my handy String and Array instance methods.

For the sake of my own review and reference, and for anyone else who might like a walkthrough of what's happening inside of JavaScript promises, I will break down the parts of a promise below and describe what they do and what the code for that behavior might look like.

The point of promises is to give us back functional composition and error bubbling in the async world.
– Domenic Denicola,
"You're Missing the Point of Promises"

Things Promises Solve

  • Callback Hell
    • Series of nested aync operations with callback
    • The callback call other async operations that take a callback
    • Every callback may need if/else blocks for error handling
  • Callback Black Hole
    • All the return values of your async callbacks get stuck in the callback's scope

Promise Public Methods

Promises don't have much of their internals exposed for developers to get creative with, but they do come with a few methods that provide useful access to some of their internal functionality.

Instance Methods

.then(successHandler, failHandler)

  • used for handling the eventual value of a promise, whether it fulfills or rejects

.catch(failHandler)

  • used for handling any uncaught rejections/errors

Static Methods

.resolve()

  • returns a Promise instance that immediately resolves to the value of whatever you pass to it

.all()

  • useful for letting promises resolve in parallel

Let's Code a Promise Constructor

So what is the final product even supposed to look like anyway?

Well if you've used the ES2015 promise constructor, then you know that making a Promise instance looks something like this:

const aPromiseForSomeValue =
    new Promise(function (resolve, reject) {
      someAsynOperation(
      'some argument', 
      function (err, result) { 
        if (err) reject(err); 
        else resolve(result);
      })
    })

So that's what it looks like to use a Promise constructor to make a promise instance, but how does it work under the hood? What happens to the anonymous function passed into the constructor? What do resolve and reject do? Where do they come from?

And what about .then()? Everyone who has interacted with promises has done this sort of thing:

somePromise
.then(doSomethingWithResult)
.then(doSomethingElseWithThat)
.catch(doThisIfError)

But how on earth does that sorcery work?

I'm going to try to walk through all of the above...below. My implementation won't be identical to JavaScript's nor Promise/A+ compliant, but it will implement almost all of the essential behavior of JavaScript promises. The idea is to help us develop a sufficiently accurate mental model for how promises work and why they behave as they do, so we can use promises more intelligently and with greater confidence, even if we don't know every low level detail of their implementation.

First let's get familiar with where Promises come from in JavaScript: the Promise Constructor.


Structure and State

The $Promise Constructor [1]

  • is a constructor function to be invoked with the new keyword
  • it requires a function as an argument
    • this function is known as the "executor" since it executes essential internal methods of the Promise, which we'll see further below
  • it throws a TypeError if called with no function argument
function $Promise ( executor ) {
  executor() // if this is not a Fn, it will throw a TypeError
}

An instance of the $Promise constructor...

  • starts with "pending" internal state
  • has an ._internalResolve instance method
  • has an ._internalReject instance method
function $Promise ( executor ) {
  this._state = 'pending';
  executor(); // we'll look at what this does momentarily
}

$Promise.prototype._internalResolve = function ( value ) {
    // this does stuff to handle a "fulfilled" promise
}

$Promise.prototype._internalReject = function ( reason ) {
    // this does stuff to handle a "rejected" or "failed" promise
}

$Promise.prototype._internalResolve()

  • can take one argument of any value
  • changes the promise instance's state to "fulfilled"
  • stores the passed-in value on the promise instance for future retreival
  • works even when passed falsey values
  • does not overwrite the status or value of an already-settled promise
function $Promise ( executor ) {
  this._state = 'pending';
  executor() // coming soon...
}

$Promise.prototype._internalResolve = function ( value ) {
  if ( this._status !== 'pending' ) return; // won't overwrite settled promises
  this._state = 'fulfilled'; 
  this._value = value;  // the value with which the promise is fulfilled
}

$Promise.prototype._internalReject = function ( reason ) {
    // this is going to look really similar to ._internalResolve
    // except for one thing
}

$Promise.prototype._internalReject()

  • can take one argument of any value, generally the return value or error thrown by a failed async operation
  • changes the promise instance's state to "rejected"
  • stores the failure "reason" to the promise as it's value to be retrieved later
  • works even when passed falsey values
  • does not overwrite the status or value of an already settled promise
function $Promise ( executor ) {
  this._state = 'pending';
  executor()
}

$Promise.prototype._internalResolve = function ( value ) {
  if ( this._status !== 'pending' ) return;
  this._state = 'fulfilled';
  this._value = value;
}

$Promise.prototype._internalReject = function ( reason ) {
  if ( this._status !== 'pending' ) return;
  this._state = 'rejected';
  this._value = value;
}

The executor function

  • gets called once, i.e. upon instantiation of a promise
  • takes two other functions as arguments: resolve and reject
function $Promise ( executor ) {
  this._state = 'pending';
  executor(
      resolve, // where'd this come from?
      reject   // and this?
  )
}

The resolve function argument

  • fulfills the promise with the value that was passed into it
  • is itself the ._internalResolve internal method
function $Promise ( executor ) {
  this._state = 'pending';
  executor(
      this._internalResolve,
      reject  // bet you can guess what this is then
  )
}

The reject argument

  • rejects the promise with the value (error reason) that was passed into it
  • is itself the ._internalReject method
function $Promise ( executor ) {
  this._state = 'pending';
  executor(
      this._internalResolve,
      this._internalReject
  )
}

Recap: Constructor, Executor, Resolve, Reject

Through the executor function, you have access to the methods that determine the fate of the promise, whether it fulfills or fails. By calling resolve() or reject() inside of the callback function of an async function wrapped in a Promise constructor, and passing them the value returned by the async function, you can have that return value stored in the promise (whenever it arrives) to be handled once it arrives.

const aPromiseForSomeValue =
    new Promise(function (resolve, reject) { // the "executor"
      someAsynOperation( // wating to get data from this
      'someArgument', 
      function (err, result) { // async callback to handle whatever gets returned
      
        if (err) reject(err); // executor-provided access to ._internalReject 
                                  // to fail the promise in case of error
        else resolve(result); // executor-provided acces to ._internalResolve
                                  // to fulfill promise is successful
      })
    })

Binding this to ._internalResolve & Reject

Knowing that resolve() and reject() in the code above are simply pointers to the actual internal instance methods of the promise (at least in our implementation), we know that they will throw an error because they are being called out of context rather than by the promise instance itself (i.e. somePromise._internalResolve()). Their this keyword is going to end up undefined (at least in strict mode). So we need to .bind() the references of their this keyword to the promise instance before they are passed into the executor inside of our $Promise constructor implementation.

It's an easy fix:

function $Promise ( executor ) {
  this._state = 'pending';
  executor(
      this._internalResolve.bind(this),
      this._internalReject.bind(this)
  )
}

Attaching Handlers

This is where Promises really begin to shine compared to old fashioned async callbacks. Handling the eventual result of an async operation with old fashioned async callback function involves some non-trivial problems and annoyances. The only way to do something with an async callback function's eventual result is to handle it inside of its callback function. But what if you need to do lots of things with that result? What if you need to do lots of other async things with that eventual result? Well now you're nesting async callbacks inside async callback inside of async callbacks and it's just an ugly, hard to read mess.

With promises, you can cleanly attach multiple callbacks to it to handle its eventual result, whenever that happens. You do this by calling the promise instance's .then() method. You can call .then() as many times as you need on the same promise, both before and after is has settled. This make promises very portable compared to traditional async callback functions.

Let' break this down:

A promise's .then() method

  • takes two functions as arguments, the first a handler for if the promise fulfills, the second a handler for it the promise fails
  • stores both handlers (callback functions) as a group in the promise so that one of them can be executed later
  • stores a group of handlers in the promise every time it's called on that promise instance
  • stores non-function handlers in the group as falsey values
  • if the promise is already settled, it immediately invokes all the handlers appropriate to the settled state of the promise (i.e. fulfilled or rejected)
function $Promise ( executor ) {
  this._state = 'pending';
  this._handlerGroups = []; // holds the handlers passed into .then()
  executor(
    this._internalResolve.bind( this ),
    this._internalReject.bind( this )
  )
}

$Promise.prototype.then = 
    function ( onSuccess, onReject ) {
      if ( !isFn( onSuccess ) ) onSuccess = null; 
      if ( !isFn( onReject ) ) onReject = null;
      const newHandlerGroup = { // create the group of handlers
        successCb: onSuccess,
        errorCb: onReject
      }; // store handlers for future use when the promise settles
      this._handlerGroups.push( newHandlerGroup );
      if ( this._isSettled() ) this._callHandlers(); // we'll look at this soon
    }

Calling the Right Handlers at the Right Time

Promises collect handlers in order to do something with the fulfillment or rejection value once it arrives. Handlers can be registered onto a promise both before and after it has settled to its value. The handlers wont be executed until the promise settles, and handlers registered after it settles need to be invoked right away. But how does a promise "know" when it has resolved so it can call its handlers? And how do they get called after it has already settled?

Well, let's take a look at these two states a promise can be in:

  1. An Unsettled Promise: Say you have a promise. It's still waiting for it's async innards to resolve. Your code continues executing and attaches a couple handlers to it. They get stored. A couple nano seconds later, your promise settles to a state of either fulfilled or rejected, depending on the outcome of the function passed into the Promise constructor (the executor). Upon settling, either ._internalResolve or ._internalReject gets invoked by the executor funtion. It sets the new state and value of the promise. The promise now holds a value and has some handlers stored. Now what? Well, it turns out that ._internalResolve and ._internalReject do more than just set the value and state of the promise. They also invoke another internal method that is responsible for invoking all the appropriate handlers for the promise's state. So that's the first way handlers can get invoked. All handlers that are registered onto the promise before it settles are executed by the internal resolve/reject methods. But what if handlers are attached after the internal resolve/reject method runs, since they only run once per promise?
  2. A Settled Promise: Say you have a promise. It settled 20 milliseconds ago and executed a couple handlers. Then your code calls .then() on it again and attaches a new handler into storage. The promise is already settled, so ._internalResolve (or ._internalReject) isn't going to run again. So they can't be responsible for invoking the new handler. That means .then() itself needs to check the state of the promise and invoke the handler(s) if it's already settled.

Let's try to break this down with some branching logic:

If a promise has not yet settled/resolved

  • it may collect handlers, but does not call any handlers yet (see line 10 below)
$Promise.prototype.then = function ( onSuccess, onReject ) {
      if ( !isFn( onSuccess ) ) onSuccess = null; 
      if ( !isFn( onReject ) ) onReject = null;
      const newHandlerGroup = { // create the group of handlers
        successCb: onSuccess,
        errorCb: onReject
      }; 
      this._handlerGroups.push( newHandlerGroup ); // store the handlers
      if ( this._isSettled() ) this._callHandlers(); // don't run this yet
    }

If a promise settles and has handlers attached

  • it calls the appropriate (success/reject) handler(s) attached earlier by .then()
  • it calls handlers in the order .then() attached them
  • it calls the handler by passing in the promise's value
  • it calls the appropriate handler from each handler group only once per attachement then discards the group.
// gets called in the executor by async callback if successfull
$Promise.prototype._internalResolve = function ( value ) {
  this._settle( 'fulfilled', value );
}
// gets called in the executor by async callback if error 
$Promise.prototype._internalReject = function ( reason ) {
  this._settle( 'rejected', reason );
}

$Promise.prototype._settle = function ( state, value ) {
  if ( this._isSettled() ) return;
  this._state = state;
  this._value = value;
  this._callHandlers(); // everything kicks off here!
}

$Promise.prototype._callHandlers = function () {
  if ( this._isPending() ) return; // if settled, do all the stuff
  this._handlerGroups.forEach( group => { // loops through handlers in their attachment order
    const handler = this._getCorrectHandler( group ); // does what is says
    handler( this._value ); // may be a success or fail handler
  } );
  this._clearHandlerQueue(); // after all handlers executed, clear
}

When new handlers get attached to an already-settled promise

  • they get stored like the rest
  • immediately the appropriate (success/reject) handler is called
  • after the handler is called, the handler group is discarded
$Promise.prototype.then = function ( onSuccess, onReject ) {
  if ( !isFn( onSuccess ) ) onSuccess = null; 
  if ( !isFn( onReject ) ) onReject = null;
  const newHandlerGroup = { // create the group of handlers
    successCb: onSuccess,
    errorCb: onReject
  }; // store handlers for immediate retrieval since already settled
  this._handlerGroups.push( newHandlerGroup );
  if ( this._isSettled() ) this._callHandlers(); // runs the just-added one
}

$Promise.prototype._callHandlers = function () {
  if ( this._isPending() ) return; // if settled, do all the stuff
  this._handlerGroups.forEach( group => {
    const handler = this._getCorrectHandler( group ); // does what is says
    handler( this._value ); // may be a success or fail handler
  } );
  this._clearHandlerQueue(); // does what it says
}

A promise's .catch method

If you've used .then(), you've certainly used .catch(). Did you know it's really just .then(null, rejectHandler)?

  • attaches the passed-in function as an error handler
  • returns the same kind of thing that .then would
$Promise.prototype.catch = function (onReject) {
    return this.then(null, onReject)
}

Promise Chaining and Transformation

Attaching multiple hanlders to a promise is handy if you want to do multiple isolated things with an individual promise's eventual result, especially if you want to do it at different points in time in your code. But what if you want to do something with the result of that handler, like pass it into another function, even an async function? And then what if you wanted to do something with the result of that handler?

This is where promises really come into their own. Not only can you call .then() on the same promise multiple times. You can also call .then(). on .then() to further handle the result of the preceding .then(). Like so:

somePromise
.then(doSomethingWithPromiseValue)
.then(doSomethingWithPreviousThenResult)
.catch(doThisIfErrorInEither)

You can chain .then() on .then() as many times as you wish, allowing you to create a sequence of functions that can handle the output of the previous one, or at least wait for its output before executing its own operations, which may be dependent on some side effect of a previous .then().

This is the true beauty of promises, because it allows us to coordinate asynchronous operations one after another and thus avoid race conditions where async operations may unpredictably happen in the wrong order. By using .then() we can basically get all the benefits of non-blocking async code, with all the predictability and reasonability of synchronous code. There are of course ways to do this without promises, the most common method being nesting aysnc function inside the callbacks of other async function. But if you've ever had to do that, then you're familiar with this shape:

router.get('/dashboard', function(req, res, next) {
  Friends.findAllFriendsOf(req.session.user, function(err, friends) {
    if (err) next(err);
    Friends.findAllFriendsOf(friends, function(err, friends_of_friends) {
      if (err) next(err);
      users = friends_of_friends.filter(function(friend) {
        return !friend.isBlockedBy(req.sesssion.user)
      })
      Stories.findAllStoriesBy(users, function(err, stories) {
        if (err) next(err);
        res.render('dashboard', {stories: stories})
      })
    })
  })
})

That's a pretty realistic example of a callback pyramid resulting from querying a database for some data, in this case "Stories" on line 9 (think Instagram/Facebook feeds), which depends on data from prior queries (like "users" on line 6) in order to perform the search. This certainly isn't a worst-case scenario of callback hell/pyramid of doom; it's about four indents deep, but it's still not the easiest to read, and it involves lots of repetitive code (like all the error handling). For even more complex queries, you'll find out that the hellishness of async callbacks increases exponentially with each new nested callback.

With promises, the same code could look like:

router.get( '/dashboard', function ( req, res, next ) {
  Friends.findAllFriendsOf( req.session.user ) // get all of user's "friends"
    .then( Friends.findAllFriendsOf ) // get all friends of user's friends
    .then( friends_of_friends => {
      return friends_of_friends.filter( friend => { // filter out blocked friends
        return !friend.isBlockedBy( req.session.user )
      } )
    } )
    .then( Stories.findAllStoriesBy ) // get stories of non-blocked friends
    .then( res.render.bind( res, 'dashboard' ) ) // render to template
    .catch( next ); // pass errors to error-handling middleware
})

And you could pile up the .then()s to do far more complex queries without ever needing to worry about your code becoming unreadable. And notice how easy error handling is too. Just one .catch() at the end. How's that for DRY code?

So how does this magic work? How is it possible to call .then() on .then()? Well, .then() is an instance method on promises, so .then() must return a promise, but where'd that come from? And how how does it end up containing the settled value of the previous promise? And how can a single .catch() (also a promise instance method) at the end of a .then() chain reliably catch an errors from a .then() any number of lines above it? So much mystery in this super useful construct.

Let's try to break this down:

Starting with a promise named promise1 (p1)

  • .then() on p1 creates a new promise p2 and stores it internally along with its handlers
  • .then() then returns p2 immediately, before it's even settled

This wasn't part of our earlier implementation of .then(). Let's add it:

$Promise.prototype.then = function ( onSuccess, onReject ) {
  if ( !isFn( onSuccess ) ) onSuccess = null;
  if ( !isFn( onReject ) ) onReject = null;
  const newPromise = new $Promise( () => {} ) // make new empty promise
  const newHandlerGroup = {
    successCb: onSuccess,
    errorCb: onReject,
    downstreamPromise: newPromise // store reference to the new promise set its
  };                              // value later when current promise settles
  this._handlerGroups.push( newHandlerGroup ); // store handler group
  if ( this._isSettled() ) this._callHandlers(); // handle if settled
  return newPromise; // this is what you actually .then() off of
}                    // when you chain .then()s

The (eventual) value of promise2 (p2) depends on the outcome of p1 and its handler

This is where the innards of promises start to get a little hairy. Once p1 is settled, ._callHandlers() will fire and run all the appropriate success/reject handlers stored in p1. But that's not all it does. .callHandlers() is also responsible for determining the settled value of p2 based on the outcome of p1 and its handlers. There are a lot of possible scenarios a settled promise can find itself in, and each scenario will produce a slightly different state of affairs for p2. So we'll need to take this bit by bit. This is probably the most important part of promises to understand in terms of every-day use, because if you aren't confident about what your .then() is going to pass into its fulfill/reject handlers, then you may find yourself deeply vexed while debugging the source of your promise's problem and afraid to do any complex .then() chaining for fear of something unexpected happening.

Let's break this down:


If p1 fulfills...

  • ...with no success handler

    • p2 should fulfill with p1's value
  • ...with a success handler that returns some value

    • p2 should then fulfill and should have whatever that handler returns as its own value
  • ...with a success handler that returns a new promise (p3)

    • then p2 should mimic/assimilate with p3, taking on p3's (possibly eventual) state and value as though it were itself p3, whether p3 fulfills or rejects
  • ...with a success handler that happens to throw an error

    • then p2 should be rejected with the error as its value

If p1 rejects...

  • ...with no rejection handler

    • then p2 should be rejected with p1's rejection reason as its value
  • ...with a rejection handler that returns some value

    • then p2 should fulfill with whatever the error handler returns[2]
  • ...with a reject handler that returns a new promise (p3)

    • then p2 should mimic/assimilate with p3, taking on p3's (possibly eventual) state and value as though it were itself p3, whether p3 fulfills or rejects
  • ...with a rejection handler that itself throws an error

    • then p2 should be rejected with the error as its value
  • still chains correctly if the promise is already settled

  • .then can be chained many times


Static Methods .resolve and .all

The static method $Promise.resolve
is a function, and not one we have already written
takes a and returns a
takes a and returns the same
demonstrates why "resolved" and "fulfilled" are not synonyms
The static method $Promise.all
is a function
takes a single array argument
converts an into a
converts an into a
converts an into a
converts an into a
converts an (fulfilling in random order) into a (ordered by index in the original array)
rejects with when one of the input promises rejects with
is not affected by additional rejections


  1. We're calling it $Promise to avoid using the JavaScript reserved word Promise. ↩︎

  2. NEED PUT THIS IN OWN WORDS: Why fulfilled? This is similar to try-catch. If promiseA is rejected (equivalent to try failure), we pass the reason to promiseA's error handler (equivalent to catch). We have now successfully handled the error, so promiseB should represent the error handler returning something useful, not a new error. promiseB would only reject if the error handler itself failed somehow (which we already addressed in a previous test). ↩︎