Meteor has always had an ambiguous relationship with the rest of the Node.js ecosystem.

One one hand it runs on Node.js and follows a lot of its conventions, yet at the same time Meteor also adopts a different approach for a number of things.

Nowhere is this more apparent than when it comes to packages. From the start, people have asked (sometimes quite loudly) why Meteor didn’t just use NPM out of the box.

It turns out that there are some very good reasons for this. And due to Meteor’s dual client-and-server architecture as well as Meteor’s tendency to avoid callbacks, making a vanilla NPM packages more “meteoric” can require a bit of work.

In this article, we’ll explore the different techniques available to do just that, from painful, manual wrapping all the way to using third-party packages to smooth things out.

This is the second article in a three-part series focusing on sync/async in Meteor:

Sending Emails

In the first part of this series, we saw that Meteor made use of Fibers in order to enforce “stylistic” synchronicity. So in order to stay consistent, it make sense to write our libraries using a similar pattern.

Let’s pick a concrete example. We’ll write a simple Meteor package to expose a Meteor method that does two things:

  1. Send out an email to a list of addresses.
  2. Add a log entry to keep track of which user triggered the method call.

Here’s how we could call this package from a Meteor method:

Meteor.methods({
  sendEmailMethod: function(emails, content) {
    try {
      var emailSent = sendEmail(emails, content);
      console.log('Emails successfuly sent');
    } catch (e) {
      console.log('Something bad happened!');
    }
  }
});

You would then call this method from the client as usual with:

Meteor.call('sendEmailMethod', ['tom@test.com', 'sacha@test.com'], 'Hello world!');

Please note that apart from the above line of code, all the code examples in this article will run on the server.

We’ve also uploaded a simple app containing the code for this article to GitHub.

Keeping it In-House

The easiest way to write this sendEmail function would be using Meteor’s very own Email.send():

// on the server
sendEmail = function(emails, content) {
  Email.send({to: emails, from: "hello@myapp.com", text: content});
  EmailLogs.insert({time: new Date(), userId: Meteor.userId()});
}
Version 1: using Email.send()

Note that Email.send() has a synchronous API, which means that if it fails, a catchable exception will be thrown straight away. And conversely, if the code that comes afterwards gets executed, we know it means the call succeeded, so we can log the event with confidence.

Wrapping the Mailgun NPM Package

This works because behind the scenes, Meteor uses Fibers to provide Email.send() (and by extension, our own sendEmail()) with a synchronous API, without us needing to specify it explicitly.

However, if you ever need to deal with non-Meteor code you’ll soon find out that this behind-the-scenes work hasn’t been done, and that it’s up to you to convert these APIs into something that looks more “meteoric”.

First, let’s start with a typical, out-of-the-box NPM package. The standard NPM pattern is to provide async APIs that take a callback as a final argument, and that callback expects arguments of the form (error, result).

We’ll go back to our previous newsletter example, but this time instead of using Meteor’s own Email.send() method, we’ll send our emails through Mailgun’s API, using the mailgun-js package.

Mailgun.

The code that you might initially write looks something like this:

// Mailgun API Keys
var API_KEY = 'abc123';
var DOMAIN = 'xyz.mailgun.org';
var mailgun = Npm.require('mailgun-js')({apiKey: API_KEY, domain: DOMAIN});

sendEmail = function(emails, content, callback) {

  var data = {to: emails, from: "hello@myapp.com", text: content};

  mailgun.messages().send(data, function(error, body) {
    if (error)
      return callback(error);

    EmailLogs.insert({time: new Date(), userId: Meteor.userId()});

    callback(null);
  });
};
Version 2: using Mailgun

The first thing you’ll notice when running this code is that the EmailLogs.insert() command fails. This is because all Meteor code expects to run in a Fiber. We can get around that by wrapping our callback using bindEnvironment, which takes care of setting up a Fiber for us:

sendEmail = function(emails, content, callback) {
  var future = new Future;
  var data = {to: emails, from: "hello@myapp.com", text: content};

  mailgun.messages().send(data, Meteor.bindEnvironment(function(error, body) {
    if (error)
      return callback(error);

    EmailLogs.insert({time: new Date(), userId: Meteor.userId()});

    callback(null);
  }));
};
Version 3: wrapping with bindEnvironment

More on bindEnvironment

Why exactly did we need to use bindEnvironment? To understand the issue at hand, we have to dig a little deeper into how Meteor.userId() works.

Behind the scenes, all Meteor server code is running on a single thread. A single user’s method isn’t going to lock up the whole server, as the fibers that they run in yield to each other. So how can something like Meteor.userId() work when a different fiber’s callbacks could run at any time?

The answer is that Meteor attaches a set of variables to the running fiber itself (that set is called the “environment”), one such variable being the userId of the caller of a method. Meteor.bindEnvironment() will look at the set of environment variables defined at the time it’s set up, and attached them to the new fiber it creates when the callback is run.

As we’ve naively written our code in an async “callback” style, our function call gets a little more complex:

Meteor.methods({
  sendEmailMethod: function(emails, content) {
    sendEmail(emails, content, function(error, emailSent) {
      if (error) {
        console.log('Something bad happened!');
      } else {
        console.log('Emails successfuly sent');
      }
    });
  }
});

Compared to our first example, we had to change the way we call our sendEmail function, requiring us to rewrite any code that calls it. And it’s not quite callback hell yet, but we can already see how our code will grow more and more complex with each new API call.

So let’s see how we can go back to our simple, sync-style function call while still using the Mailgun API.

Back to the Future

The lowest-level way to achieve this is by using Future, a Fibers sub-library:

var Future = Npm.require('fibers/future')

sendEmail = function(emails, content) {
  var future = new Future;
  var data = {to: emails, from: "hello@myapp.com", text: content};

  mailgun.messages().send(data, Meteor.bindEnvironment(function(error, body) {

    if (error)
      return future.error(error);

    EmailLogs.insert({time: new Date(), userId: Meteor.userId()});

    future.return(null);
  }));

  return future.wait();
};
Version 4: using Future

Here, we create a future object and call wait() on it to “synchronously” wait until return() is called inside the callback.

Future can do a few other things, but this is more or less all you need to know about it for most use cases.

Pictured: the face people usually make when trying to understand this stuff.

Now we can use our library nicely again:

Meteor.methods({
  sendEmailMethod: function(emails, content) {
    try {
      var emailSent = sendEmail(emails, content);
      console.log('Emails successfuly sent');
    } catch (e) {
      console.log('Something bad happened!');
    }
  }
});

WrapAsync: An Easier Way

Still, that was a lot of boilerplate, and we wouldn’t want to duplicate this for every single one of our library’s functions.

This is exactly where Meteor.wrapAsync comes in. This utility takes care of wrapping and “futurizing” node-style functions while binding their environment.


var messages = mailgun.messages();
var wrappedSend = Meteor.wrapAsync(messages.send, messages);

sendEmail = function(emails, content) {

  wrappedSend({to: emails, from: "hello@myapp.com", text: content});

  EmailLogs.insert({time: new Date(), userId: Meteor.userId()});
};
Version 5: using wrapAsync

A Bit of a Bind

You might be wondering why we had to provide messages as a second argument to wrapAsync. This is because mailgun.messages().send() is an object method that expects this to be the mailgun.messages() object. So we need to specify the context (this argument) when we wrap it.

Arunoda to the Rescue

Finally, if you are doing a lot of this, you should check out Arunoda’s async package, which provides even more utilities for wrapping callbacks easily.

Wrapping Up

Sync/async concepts are by no means intuitive or easy to understand, all the more so when it comes to getting NPM packages to play nice with Meteor.

Yet even if packages like async can make our lives much easier and mask some of that complexity, it’s still important to understand what’s going on behind the scenes.

Hopefully this article will help you navigate these tricky situations next time you write a Meteor package.

Be sure to sign up for our newsletter below to know when we publish the last part of our Sync/Async series, about Latency Compensation!