Meteor was born out of a simple desire to make building web apps more approachable. As the Meteor documentation explains:

Meteor is an ultra-simple environment for building modern websites. What once took weeks, even with the best tools, now takes hours with Meteor.

But a side effect of making the complex simple is that the simple can sometimes become unexpectedly complex. Today, we’ll see an example of just that as we take a look at pagination in Meteor apps.

Pagination Without Pages

When you hear about “pagination”, you probably think about splitting content into pages. After all, that’s what the word literally means.

But when it comes to web apps (and Meteor apps in particular), the concept of “pages” isn’t as relevant as it used to be.

To understand why, imagine you’re on the third page of a paginated list of search results, with 10 results per page. Now Meteor being reactive, this list would presumably be updated in real time.

What happens if someone adds an item to the first page of the list? Should it “push down” all the other results in the list, thus changing the content of the page you’re currently looking at?

If so, what if you were just about to click on result number 30, and –now being number 31– it’s suddenly disappeared to page 4?

You can see how adopting traditional pagination can be a source of UX headaches in real-time apps. For that reason, they tend to use another pattern: incremental pagination (i.e. a “load more” button or infinite pagination).

An early prototype of infinite pagination. Photo credit: Lawrie Cate

But while incremental pagination may solve a lot of UX issues, it’s not without its own technical problems.

Problem 1: Limited Visibility on the Client

The first issue lies with one of Meteor’s fundamental principles: database everywhere.

One of Meteor’s key innovations was coming up with a way to copy a subset of the database to the client, and keep that subset in sync.

The key word here is subset: you obviously don’t want to simply send over your entire database to the client. That would be a very bad idea, both for performance and security reasons.

The issue here is that while the client knows what’s in that subset, it’s not aware of what isn’t in it. In other words, the client may know about 10 posts, but it has no clue whether the actual server database contains 10, ten thousand, or ten millions posts in total.

The client has no way of knowing how many documents it’s missing.

This becomes a problem when implementing incremental pagination: how do you know when you’ve loaded every possible post, and you don’t need to show that “load more” button anymore?

Solution 1: Ask And You Shall Receive

A relatively simple solution (and the one we implement in Discover Meteor) is comparing how many documents you asked for with how many you actually received.

In other words, imagine you ask for 10 posts:

Meteor.subscribe("posts", 10);

But you only actually received 5:

Posts.find().count(); // 5

You can safely deduct that this means you’ve displayed everything there was to display, and you can safely hide that “load more” button.

On the other hand, if you asked for 10 posts and you received 10, there might still be more to come on the server, and you can keep showing “load more”.

Of course, the issue here is that our algorithm doesn’t work for the case where there are exactly 10 posts on the server: we’ll ask for 10 posts and receive 10, and erroneously keep showing the “load more” button despite having reached the end of the collection.

Solution 2: Publishing Counts

A better (though more complex) solution would be to separately provide a count of total documents for any given cursor.

For example, you could imagine a getCursorCount Meteor method that you can call with a cursor as argument. The method would run the cursor on the server, then return the result to the client.

But a method wouldn’t be reactive, so your total count for any given cursor would quickly get out of date as items are added to your database. So a better way to accomplish the same thing is to use a dedicated subscription/publication.

This is exactly what the publish-counts package does. Inside a publication, you can set an observer on a cursor with:

Counts.publish(this, "name-of-counter", Posts.find());

And you’ll then be able to get the total count for that cursor by calling Counts.get('name-of-counter') on the client.

The package can even handle trickier situations, like calculating the total number of documents from a nested sub-property.

Wishlist: Total Cursor Counts

But hacks and third-party packages aside, the ideal solution would be for total cursor counts to be supported out of the box by Meteor.

You can already find out how many documents correspond to a cursor on the client with:

Posts.find({author: "Sacha"}).count() // 10

So it’s not hard to imagine an API for figuring out how many documents correspond to the same cursor on the server:

Posts.find({author: "Sacha"}).totalCount() // 173

Of course, as with many seemingly simple features, this might turn out to be harder than it looks to implement. But if the Meteor Development Group is reading, here’s hoping they consider it!

Problem 2: Subscription Ownership

The second issue is a bit trickier to explain, but it boils down to the fact that cursors and publications/subscriptions are not aware of each other.

So when you call Collection.find() on the client, it couldn’t care less which publication is actually publishing the documents available in the browser’s memory.

Why should this matter? After all, even if you have multiple publications all publishing data from the same collection, these documents are all coming from the same database. So this should in no way impact the result of Collection.find()?

To help visualize this issue, let’s do a small thought experiment.

Buying Comics

Imagine a peculiar comic book store stand that specializes in programming comics. Being an avid collector, you buy the latest issue of Meteor Comics (#6), as well as the original launch issue (#1) to complete your collection.

Buying comics and bringing them home.

Once you’re back home, you show off your newly acquired collection to your roommate. She starts browsing the first issue, and pretty soon she’s hooked and she wants to read the rest, so she asks you for the first two issues of Meteor Comics.

So you go back to the store, buy issue #2, and bring it back. You now have issues #1, #2, and #6 available at home:

Growing our collection.

But here’s the key part: until you come back from that second trip, the “first two issues” your roommate has at her disposal technically consist of issues #1 and #6!

Now of course, your roommate is smart enough to deduct that there must be more issues between #1 and #6 back at the store. But on the other hand, a really dumb roommate (or a software program) would have no way of knowing this unless you specifically told it. So unless you did so, it would probably assume that issue #6 directly follows issue #1 and wonder why the story wasn’t making any sense.

Missing issues.

Now let’s bring things back to Meteor land while this little story is still fresh in your mind.

Disappearing Issues

In Meteor terms, the store would be the server, making the latest or earliest issues available for anybody wanting to acquire them via these two publications:

Meteor.publish("latestIssues", function(limit) {
  return Issues.find({}, {limit: limit, sort: {date: -1}});
});

Meteor.publish("earliestIssues", function(limit) {
  return Issues.find({}, {limit: limit, sort: {date: 1}});
});

Your initial trip to the comic book store would be the act of subscribing to both publications to get one document from each:

Meteor.subscribe("latestIssues", 1);
Meteor.subscribe("earliestIssues", 1);

And your roommate’s request for more issues would be the equivalent of incrementing one of those subscription’s limit:

var subscriptionLimit = 2;
Meteor.subscribe("earliestIssues", 2);

Finally, her sitting down and reading the “first two issues” would be the equivalent of a MongoDB query:

var cursorLimit = 2;
var earliestIssuesCursor = Issues.find({}, {limit: cursorLimit, sort: {date: 1}})

Remember how I said a dumb computer would have no way of knowing about the extra issues left back at the store? This is exactly the problem we run into when we want to increment the limit on the client.

Let’s say we naively increment both subscriptionLimit and cursorLimit as soon as the user clicks “load more”.

Here’s the thing: if we do this, there will be a small window of time between that incrementation and the time when the subscription has returned its results where the results of the cursors will also include the latest issue we obtained from the subscription to latestIssues.

As soon as the correct results are loaded though that issue will stop being one of the “first two issues” and disappear, resulting in a confusing content flicker and bad overall UX.

Pagination flicker (pay attention to the item numbering).

You can also see the problem live and play around with the code in this MeteorPad (see further down the page for a version of the same code with a fix).

Solution 1: Two-Step Process

The first solution is to implement a two-step process. So instead of changing both limits at once, we’ll change the subscription limit first, and only change the cursor limit once the subscription is done loading.

By doing this, we’re making sure that during that problematic window of time we’re still only asking for one document, not two. Only once the new, correct data is loaded do we expand our client-side cursor to show two documents.

This is the solution we implement in Discover Meteor, by the way, and you can also check it out in combination with the template-level subscription pattern in this MeteorPad.

Compared to the previous implementation, this time we’re only incrementing the limit if the subscription is ready:

// if subscription is ready, set limit to new limit
if (subscription.ready()) {
  console.log("> Received "+limit+" posts. \n\n")
  instance.loaded.set(limit);
} else {
  console.log("> Subscription is not ready yet. \n\n"); // do nothing
}

Solution 2: Find From Publication

You could avoid the issue entirely if there was a way to limit results returned by Collection.find() to documents published by a specific publication.

In that scenario, you’d simply specify that you want to display documents from the latestIssues or earliestIssues publications, without worrying about cross-contamination from one publication to the other.

This is exactly what the Find From Publication package offers.

You first need to define a special publication using FindFromPublication.publish instead of Meteor.publish:

FindFromPublication.publish('latestIssues', function() {
  return Issues.find();
});

And once you’ve subscribed to that publication on the client, you’re then able to limit your queries to its documents by passing its name as the first argument of Collection.findFromPublication:

var issuesCursor = Issues.findFromPublication('latestIssues', {}, {limit: 2});

Wishlist: Native Find From Publication

Ideally, this “find from publication” feature would be available natively in Meteor.

You can already pass an options object to Collection.find(), so you could easily imagine a syntax like this:

var issuesCursor = Issues.findFromPublication({}, {limit: 5, publication: 'latestIssues'});

That being said, while this feature might seem simple on paper, it would probably require quite a few changes to the way Meteor works under the hood.

Conclusion

Pagination is by no means a make-or-break problem, and these issues both have existing work-arounds. Still, every little thing counts. And since Meteor’s mission is to make developers everywhere more productive, it’d be a shame to make them spend time figuring this out.

So we’re hoping Meteor gets some new pagination features in the near future, and that this blog post becomes obsolete soon!