Functional Node.js: Working with Promises

I’ve used promises on every Node project I’ve worked on over the past year. I’ve made a lot of mistakes and done a fair bit of reading on the subject. The following is a guide to writing clear, concise functional code with the Bluebird promise library. Additionally, I’ll use the classic promise syntax with chained methods, rather than async/await. In my opinion, it’s easier to read. However, I like to think that what I have to say will transfer well to other promise libraries and the async/await syntax.

1/19/2018 – I’ve updated this article to talk a little about Bluebird’s Promise.map in the Most Useful Methods section.

Article Parts

1. For Absolute Beginners: Why Promises?
2. Most Useful Methods in Bluebird
3. Basic Structure of a Multi-Promise Call
4. Problems and Anti-Patterns (with Suggestions)
5. Separating Concerns: Readable Logic with Promise Chains
6. References

For Absolute Beginners: Why Promises?

If you’re brand new to coding or Javascript, one thing to know is that Javascript is designed to be asynchronous. Logic starts to run in the order that it’s written, but when a function doesn’t return/complete instantly, the following code runs instantly without waiting for the return/completion of its predecessor. A simple example of this is a timeout.

var willRunFirst = function() {
  console.log('bar');
}
var willRunSecond = function() {
  console.log('foo');
}

setTimeout(willRunSecond, 2000);
willRunFirst();

The output for the above is:

bar
foo

Because willRunSecond is delayed by two seconds, willRunFirst‘s console log finishes first and prints first, even though willRunSecond appears in the code before it.

So in traditional Javascript, code that must run after a value is returned are placed inside of callbacks, which don’t run until the value is returned. Often, but not always, the value that the callback needs comes from an HTTP request (asking for data from another resource, like the server or an external API). Here’s an example of a traditional browser Javascript (jQuery) callback requesting an external resource:

$.ajax({
  url: "/send/stuff",
  method: "POST",
  data: {
    name: "Sally",
    dob: "1925/05/29"
  },
  success: function(response) {
    console.log('success: ' + response)
  },
  error: function(err) {
    console.log('error: ' + err);
  }
});

Pretty clear, right? But what if you need to make another HTTP request using a value that you calculate based on the result from the first HTTP request? Because callbacks are nested, they can get messy pretty quickly. It can be unclear what your logic is actually doing, and closing parentheses are hard to keep track of. Awesome code not only runs efficiently, but is also easily readable by other humans. So something like the function below really sucks for everybody — for you writing it, and for others reading it.

$.ajax({
  url: "/send/stuff",
  method: "POST",
  data: {
    name: "Sally",
    dob: "1925/05/29"
  },
  success: function(response) {
    console.log('success: ' + response)
    $.ajax({
      url: "/send/more/stuff",
      method: "POST",
      data: {
        name: "Sally",
        birthCity: response.city
      },
      success: function(nextResponse) {
        console.log('success: ' + nextResponse)
      },
      error: function(err) {
        console.log('error: ' + err);
      }
    });
  },
  error: function(err) {
    console.log('error: ' + err);
  }
});

This is where promises come in handy. jQuery supports promises in ajax calls based on its wrapper around native Javascript promises. Suddenly, the above can be refactored like this:

$.ajax({
  url: "/send/stuff",
  method: "POST",
  data: {
    name: "Sally",
    dob: "1925/05/29"
  }
})
.then(function(response) {
  console.log('success: ' + response)
  return $.ajax({
    url: "/send/more/stuff",
    method: "POST",
    data: {
      name: "Sally",
      birthCity: response.city
    }
  })
})
.done(function(nextResponse) {
  console.log('success: ' + nextResponse)
})
.fail(function(err) {
  console.log('error: ' + err);
});

Much more readable, right? With native Javascript promises, you must create/define a new Promise instance (promise-returning function) before calling it, like the following vanilla XHR request:

var request = function(method, path, data) {
  return new Promise(function(resolve, reject) {
    var req = new XMLHttpRequest();
    req.open(method, path);
    req.setRequestHeader("Content-Type", "application/json;charset=UTF-8");
    req.onload = function() {
      if(req.status == 200) {
        resolve(req.responseText);
      } else {
        reject(new Error(req.statusText));
      }
    }
    req.onerror = function() {
      reject(new Error('unknown error'));
    }
    return req.send(JSON.stringify(data));
  });
}

//with some function renderMessage
var message;
return request('POST', '/mail', data)
.then(function(result) {
  if(result == 'true') {
    message = 'Your message was successfully sent. ';
  } else {
    message = "Sorry, we couldn't send your message. Try again later. ";
  }
  renderMessage(message);
})
.catch(function(error) {
  message = "Sorry, we couldn't send your message. Try again later. ";
  renderMessage(message);
});

With the Bluebird library, we can “promisify” callback-returning functions and construct promise chains on demand without explicit definitions. This is not the only way to use a promise library, but I find it useful and concise, especially for the server. The rest of this article will focus on writing functional promises using the Bluebird library as a substitute for the native Javascript Promise class. Node packages expose a wide variety of asynchronous functions that can be “promisified”, including writing to/reading from the file system, querying a database, and sending emails and text messages. Some Node packages even expose promise-returning functions, but including Bluebird or another promise library will help you smoothly call an asynchronous function, wait for its result, transform it, and call the next asynchronous function in a chain of virtually any length, while handling errors through a single channel. I’ll talk more about why this is good later on.

Most Useful Methods in Bluebird

To get started with Bluebird on the server, replace the native Promise class with Bluebird in your file:

const Promise = require('bluebird');

What can Bluebird do for you? Here are some of the methods I use the most in my code. I’ll give an example using all of them at the end of the section.

Promise.promisify(singleFunction) and Promise.promisifyAll(methods)

“Promisification” is the best because it allows you to use virtually any traditional callback-taking function or method (possibly from a library) and transform it into a promise-returning machine. For example, using Promise.promisifyAll(require('fs')), you can read and write from the file system and handle promises instead of callbacks. Ditto for async functions almost any library. Bluebird appends “Async” to the end of every promisified method name, which helps me personally keep track of which traditional library methods are returning promises. More from the Bluebird docs…

Promise.resolve(value)

You can chain promise after promise… but eventually, you need to return a value. Promises, however, are designed to return promises, not values. Enter Promise.resolve, which allows you to return a value from a promise. The returned value can ‘fulfill’ the promise in one of two ways: it can either exit the promise chain (returning a plain value for non-promise functions to read) or become the value returned to the next .then method block in the promise chain. If there are further .then method blocks in the promise chain, Promise.resolve will pass the returned value to the next .then method block. Only a Promise.resolve called in the last .then method block (or occasionally from a .catch method block) will return a plain value into the wild. More from the Bluebird docs…

Promise.reject(error)

Promise.reject will reject the promise and return an error object. Traditionally, the .catch method block rejects any errors, but Promise.reject allows you to kick any error (even one that your code generates explicitly) to the next available .catch method block to handle as you see fit within that .catch method block. Again, Promise.reject does not exit the promise instantly, but passes the rejected error to the next .catch method block to be dealt with as an error there. In later versions of Node, promises must be rejected with a Javascript error object. To create your own errors, be descriptive and call return Promise.reject(new Error(`some string or object`)); More on creating your own errors below, and  more on Promise.reject from the Bluebird docs…

Promise.all([promiseReturningFunction1, promiseReturningFunction2])

Promise.all allows you to call any number of promise-returning functions simultaneously, as long as you don’t depend on the return of one to call another. This can be useful for speeding up multiple database or api calls that don’t depend on each other but all return promises. However, it’s important to note that if any one of the promises in the array is rejected, none of the other promises resolve. The parameter of the .then method block that follows a Promise.all call is an array where the resolved objects are at the same indices (indexes) as the functions that produced them. More from the Bluebird docs…

Promise.map(initialArray, (item, index) => {});

Promise.map allows you to map an existing array and return an array of promise-returning functions.  In a native map, you call return to pass a value into the new array and move to the next item in the initial array.  However, if you want to generate an array of promises (perhaps you intend to call them with Promise.all), a native map has the unintended consequence of executing each promise immediately and returning nothing into the array. Promise.map to the rescue! More from the Bluebird docs…

For a sandbox example of all these methods, clone the repo for this post and run experiment.js in node. This sandbox example is FULL of problems and anti-patterns, but it gives some idea of the use and limitations of each of these handy methods using just Bluebird and the fs module.

Basic Structure of a Multi-Promise Call

Most promise chains should take a single .catch method at the end to handle and reject any errors the async functions throw. You’ll notice that the parameter referencing the returned value from each promise-returning function is scoped to the .then method block — that is, you can theoretically call the returned value the same thing in every .then method block. Always return your Promise.methods and promise-returning functions! Any promise method or promise-returning function called without being explicitly returned will hang and return nothing. The exceptions to this are functions and methods from promise-returning libraries (such as axios, mongoose, sequelize). However, for clarity at even moderate scale, these calls should be wrapped in their own helper functions and thus will have to be returned on call in the “master promise”.

return thisFunctionAsync(argument)
.then((returnedValue) => {
  //transform returnedValue and pass calculatedValue to the next promise
  return nextFunctionAsync(calculatedValue)
})
.then((returnedValue) => {
  //transform returnedValue and extract what you want, resolving the promise
})
.catch((error) => {
  //handle error, reject Promise
});

Problems and Anti-Patterns (with Suggestions)

Please see the references section below for great descriptions of anti-patterns from other developers.

Nesting Promises

The whole point of promises is to flatten the “callback waterfall” for multiple nested callbacks and handle all errors in a single channel. Promises should be easy to read and never indented more than 3-4 tabs over, period. If you find yourself indenting a ton, see the section on Master Promises below.

But! you may say — I have to chain the promise directly after the function that returns it! Not true! You can perfectly well call a promise-returning function within a .then method block from a previous promise-returning function — then close the first .then method block and open the next one in series, not nested. As long as the promise-returning function is the last thing called in the .then method block, Bluebird/Javascript knows that the next .then method block in series handles its return value.

return thisFunctionAsync(argument)
.then((returnedValue) => {
  //transform returnedValue and pass calculatedValue to the next promise
  return nextFunctionAsync(calculatedValue)
  .then((nextReturnedValue) => {
    //handle the return value from nextFunctionAsync
  })
})
.catch...

return thisFunctionAsync(argument)
.then((returnedValue) => {
  //transform returnedValue and pass calculatedValue to the next promise
  return nextFunctionAsync(calculatedValue)
})
.then((returnedValue) => {
  //handle the return value from nextFunctionAsync
})
.catch...

But! you may say. The scoping! I need access to the return value of the first promise-returning function still in the .then method block of the second returning function! Okay: define a variable outside the promise chain and set the return value or some transformation of it equal, so you can access it in the second .then method block.

return thisFunctionAsync(argument)
.then((returnedValue) => {
  //transform returnedValue and pass calculatedValue to the next promise
  return nextFunctionAsync(calculatedValue)
  .then((nextReturnedValue) => {
   //I need access to returnedValue here but I'm not supposed to nest! Help!
  })
})
.catch...

let someVariable;
return thisFunctionAsync(argument)
.then((returnedValue) => {
  someVariable = returnedValue;
  //transform returnedValue and pass calculatedValue to the next promise
  return nextFunctionAsync(calculatedValue)
})
.then((returnedValue) => {
  //now you have access via someVariable!
})
.catch...

Returning a Bunch of Plain Objects Back into the Promise Chain

This anti-pattern is an abuse of Promise.resolve and usually happens when you spread promise logic over several functions instead of compartmentalizing promise-returning functions and functions to map the data they return separately.

//using mongoose, a promise-returning mongodb ORM, to get all the users using a room that the first user is assigned to
const callUsersDb = function(id) {
  Users
  .findById(id)
  .select('name roomId')
  .then((result) => {
    const roomId = result.roomId;
    return Promise.resolve(roomId);
  })
  .catch((err) => {
    return Promise.reject(err);
  });
}

const callRoomsDb = function(id) {
  Rooms
  .findById(id)
  .select('users')
  .then((result) => {
    const roomUsers = {
      roomId: id,
      usersArray: result
    }
    return Promise.resolve(roomUsers);
  })
  .catch((err) => {
    return Promise.reject(err);
  });
}

//function called by controller
return callUsersDb(1234567890)
.then((result) => {
  return callRoomsDb(result);
})
.catch...

It is needlessly verbose and logically indirect to resolve the roomId value in callUsersDb just to pipe it to another promise. If these functions are spread out across files, it’s hard to get a sense of what is actually happening in the function called by the controller. For an alternative philosophy, see the section on Master Promises below.

Creating Non-Specific Errors or Rejecting Without A Native Error Object

Lots of developers will advise you to never, ever create your own Error objects. I disagree. Throwing self-created errors intentionally in promises can quickly reveal problems with database and API calls and allow you to exit a promise chain without too much bullshit (more on that under …Exiting the Chain below. But your code will quickly turn to undebuggable shit if you don’t get super specific with the errors you intentionally throw. Additionally, make sure you wrap your string/object in an instance of the Javascript Error constructor, or Node 8+ will be unhappy, and so will you.

return thisFunctionAsync(argument)
.then((returnedValue) => {
  if(!returnedValue) {
    return Promise.reject(`it didn't work, :(`);
  } else {
    return Promise.resolve(returnedValue);
  }
})
.catch...

return thisFunctionAsync(argument)
.then((returnedValue) => {
  if(!returnedValue) {
    return Promise.reject(new Error(`it didn't work, :(`));
  } else {
    return Promise.resolve(returnedValue);
  }
})
.catch...

return thisFunctionAsync(argument)
.then((returnedValue) => {
  if(!returnedValue) {
    return Promise.reject(new Error({Error: `thisFunctionAsync didn't throw an error but returnedValue returned ${returnedValue}`}));
  } else {
    return Promise.resolve(returnedValue);
  }
})
.catch...

Including an Error Parameter in a .then Method Block

This anti-pattern is also filed under “why bother with promises, then?” One of the best features of promises is that all your errors flow through the single catch statement. If you treat promises as glorified callbacks and include the error parameter in the .then method block, you’re messing with function-thrown error handling logic in your .then. Why do this to yourself? It’s hard to read, harder to debug, and chances are you aren’t exiting (resolving/rejecting) the promise properly.

Using defer, Ever

I don’t have much to say about this except don’t do it. I haven’t heard an example yet where the benefits of doing this outweighed the costs. If you have, please, enlighten me in the comments. For more on what this is, see the Bluebird Docs on Anti-Patterns.

Express: Failing to Return a Promise on Send/Render

This is a common mistake. In fact, almost no examples of promises with Express on Stack Overflow and like get this one right. After you render and or send your response from Express, resolve or reject the promise. It seems like a small detail, but your tests will randomly fail, and you will cry. Never leave your promises hanging.

Express: Sending/Rendering Early & Exiting the Chain

I’m going to suggest something that could be considered controversial. There’s a known problem with exiting a promise chain with n .then method blocks after executing (n – 1) or (n – 2) of them. The tedious (and test error-causing way) is to pass shit like return Promise.resolve('already rendered') to each successive block and skip the logic of that block to pass the same statement until the promise is over. I don’t like it. It’s messy, ugly, and verbose. It requires manual adjustment of every single .then method block with (more) if blocks after you render. So why not Promise.reject essentially the same thing, and handle the ifs once in the .catch method block? Sneaky, I know. But until there’s a fix for this (Promise.exit, Bluebird/ES people?), I like it better.

var expressController = (req, res) => {
  return thisFunctionAsync(argument)
  .then((returnedValue) => {
    if(!returnedValue) {
      return Promise.reject(new Error({Error: `thisFunctionAsync didn't throw an error but returnedValue returned ${returnedValue}`}));
    } else if(returnedValue === 'foo') {
      return Promise.resolve(returnedValue);
    } else {
      return Promise.resolve("first value isn't foo");
    }
  })
  .then((returnedValue) => {
    if(returnedValue === "first value isn't foo") {
      return Promise.resolve(returnedValue);
    } else {
      //do something else
    }
  })
  .then((returnedValue) => {
    if(returnedValue === "first value isn't foo") {
      res.json({
        failure: "value isn't foo"
      });
      return Promise.resolve(true);
    } else {
      //do something else
    }
  })
  .catch...
}

var expressController = (req, res) => {
  return thisFunctionAsync(argument)
  .then((returnedValue) => {
    if(!returnedValue) {
      return Promise.reject(new Error(`thisFunctionAsync didn't throw an error but returnedValue returned ${returnedValue}`));
    } else if(returnedValue === 'foo') {
      return Promise.resolve(returnedValue);
    } else {
      res.json({
        failure: "value isn't foo"
      });
      return Promise.reject(new Error("already sent value isn't foo"));
    }
  })
  .then((returnedValue) => {
    //do more stuff
  })
  .then((returnedValue) => {
    //and more stuff
  })
  .catch((err) => {
    if(err.message === "already sent value isn't foo") {
      return Promise.resolve(true);
    } else {
      //handle real errors
    }
  });
}

Separating Concerns: Readable Logic with Promise Chains

Master Promises: a Locus of Logic

I’m going to use Express as an example here, but my points about abstraction probably hold for any framework. This is my opinion, and as you’ll see below, there are some qualified exceptions.

Express was the first back-end framework I learned. In all the example projects I did, it was hard for me to see why the server logic needed to be abstracted from the connection, why routers needed to be abstracted from the server logic, and why the controller functions needed to be abstracted from the routes. My first apps were a hot mess because I didn’t grasp the concept of readability at scale. No one wants a single thousand line Javascript file with tons of repeating code blocks unless it’s compiled by Webpack. But the truth is that I hadn’t gone far enough. Your controller function should be merely a road map of the logic that powers sending your response. In the land of async functions, that roadmap should be one crystal clear promise chain without any smaller promise chains embedded in it. For the “don’t” example, see the Returning a Bunch of Plain Objects Back into the Promise Chain anti-pattern above.

The key to readable master promises is to wrap every promise-returning function and each transformation of the returned data in its own helper function. You might have 10-15 of these tidbits per helper file, and you can organize them however you want (database queries in a models subdirectory, etc). When you’re done, the master promise is a summary of what you’re doing, and the actual logic of it is tucked away into helper functions, which can be reused (and reused) depending on how abstract you can readably make their parameters.

Here’s an Express example using sessions. It collects some data about a potential assignment for some user from the client, and queries or writes to the users, assignments, and rooms collections or tables. This example only reads/writes from the database, but you can use this same approach to execute/return any async function.

const controller = (req, res) => {
  let user;
  let assignment;
  return queryUser(req.body.id)
  .then((userResult) => {
    console.log(userResult);
    if(userResult) {
      user = userToRenderInfo(userResult);
      const getAssnInfo = userToAssn(userResult, req.body.someThing);
      return writeAssignment(getAssnInfo)
    } else {
      req.session.error = `No user exists with id ${req.body.id}`;
      return Promise.reject(new Error(req.session.error));
    }
  })
  .then((newAssn) => {
    console.log(newAssn);
    if(newAssn) {
      assignment = assnToRenderInfo(newAssn);
      const getRoomInfo = assnToRoom(newAssn);
      return queryRoom(getRoomInfo)
    } else {
      req.session.error = `No assignment was created for user id ${req.body.id} with some detail ${req.body.someThing}`;
      return Promise.reject(new Error(req.session.error));
    }
  })
  .then((roomResult) => {
    console.log(roomResult);
    if(roomResult) {
      const rooms = roomToRenderInfo(roomResult);
      req.session.success = `Assignment scheduled and available rooms successfully retrieved`;
      res.json({
        success: req.session.success,
        user,
        assignment,
        rooms
      });
      return Promise.resolve(req.session.success);
    } else {
      req.session.error = `No rooms returned for assignment ${assignment} for user id ${req.body.id}`;
      return Promise.reject(new Error(req.session.error));
    }
  })
  .catch((err) => {
    if(!req.session.error) {
      req.session.error = `Failed to schedule user assignment for user id ${req.body.id} with some detail ${req.body.someThing}`;
    }
    res.json({
      error: req.session.error
    });
    return Promise.reject(err);
  });
}

//variables scoped to the controller function are declared and later assigned to the objects holding the info we want to render/send:
user
assignment 

//helper functions

//plain-object-returning functions that map db responses to objects to render/send
userToRenderInfo
assnToRenderInfo
roomToRenderInfo

//plain-object-returning functions that map db responses to extract data for another CRUD operation (eg, the next promise-returning function you want to call)
userToAssn
assnToRoom

//promise-returning functions that CRUD our db (see mongoose, sequelize, etc)
queryUser
writeAssignment
queryRoom

Obviously there are more ways that you might want to map data, but the idea is to define all transactional logic in helper functions with return values and call them in your controller. Some of these helper functions will return promises and some plain objects. All of these helper functions will be located in other files. This also helps hierarchicalize your requires, reducing your chances of having to untangle circular dependencies. Model files are only included in that model’s query files; only needed queries and mapping functions need be required in the controller. If you’re defining functions in your controller, it’s time to rethink that.

When and How To Nest

Given all of the above, is there every an appropriate time to nest? I argue that there can be, when a promise chain has two very clear loci (locuses) of logic. What I mean by nesting is this section is including one inner promise chain as a promise-returning function called in an outer promise chain that contains it. I ran into a situation recently where I found that I preferred using a nested promise: validating a user with the Passportjs local strategy.

On the Passport docs site (as of October 2017), they show a local strategy querying/writing to a database with a callback and returning done. However, in practice, the logic of validation is invariably more complicated. For example, on this project, the steps to signup a user were

1. Check if the signup token is valid (if not, return);
2. Check to see if the user already exists (if so, return);
3. If the new user is associated with a new business, create the new business;
4. Add the new business ID to the user, if any, and save the new user;
5. Update all associated businesses, new or not, with the new user’s ID;
6. Mark the signup token used

So in this case, I decided to use two promise chains to show two main focuses of the logic — doing the database operations to save the user, and using Passport to authenticate. Here’s what the passport strategy ended up looking like:

const signupStrategy = new LocalStrategy({
  //options object
  //pass the request object from express to the auth function (needed to access the values the user put in)
  passReqToCallback: true,
  //tell the auth function that req.body.email is the username
  usernameField: 'email',
  //tell the auth function that req.body.password is the password
  passwordField: 'password'
}, (req, username, password, done) => {
  //signup strategy logic function
  console.log(req.body, req.params);
  //look up the signup token in the db to verify that the token exists, is authorized, and is not expired
  if(password.length < 8) { 
    req.session.error = 'Sorry, your password is too short. Please make it longer.'; 
	//return false for registration back to the signup strategy return 
	done(null, false); 
  } else { 
    //FIRST MAJOR PROMISE CHAIN handles passport auth logic 
    //SECOND MAJOR PROMISE CHAIN handles signup db logic 
    return signup(req)
	.then((sessionUser) => {
      console.log(`registration result ${sessionUser}`);
      //get a user session object or false from the registration function
      if(sessionUser) {
        //if the the user was successfully returned from saving, log the success message and pass true to done to authorize (log in) the user
        console.log(req.session.success);
        return Promise.resolve(done(null, sessionUser));
      } else {
        //if the user obj is false and there is an error message, log the error message and reject promise to catch method below
        if(!req.session.error) {
          req.session.error = "there's no session user from signup logic";
        }
        return Promise.reject(new Error(req.session.error));
      }
    })
    .catch((err) => {
      //if something goes wrong with the auth process, log the error and pass false to done to say that the user wasn't successfully signed up/logged in
      console.log(err);
      if(!req.session.error) {
        req.session.error = 'some unknown error signing you up';
      }
      console.log(req.session.error);
      return Promise.resolve(done(null, false));
    });
  }
});

Note how I’m returning Promise.resolve from the .catch method block. That’s because I want to reject authorization by calling done with a false user, but to do that I need to fulfill the promise with that false done call, not reject the promise itself. Thank you, Bluebird!

That about wraps it up! More reading below.

Resources

• Bluebird Docs on Anti-Patterns
• Bluebird Github Thread on Early Returns
• Nolan Lawson for PouchDB on Understanding Promises
• David Walsh on Promises in the Browser
• Dave Atchley on Promise Patterns and Anti-Patterns
• Matt DesLauriers’ Promise Cookbook
• Dr. Axel Rauschmayer on ES6 Promises for Asynchronous Programming
• Terry Mun on Writing Better Ajax