Episode 02: Publications & Subscriptions
In this episode, we’ll cover Meteor’s publications and subscriptions, used to synchronize data between the server and the client.
- 00:53: Traditional web apps vs Meteor
- 02:29: Minimongo
- 04:58: Publishing
- 07:00: Subscribing
- 09:04: Querying
- 11:10: Advanced publications & subscriptions
Sacha: Today we’re talking about publications and subscriptions. Publications and subscriptions are basically a set of tools that gives you a way to control the flow of data from the server to the client.
A lot of traditional Web apps and Web apps framework kind of sidestep that problem, but more and more Web developers are finding that they need to deal with it. Today we’ll see how Meteor deals with it.
Traditional Web Apps VS Meteor
Tom: Let’s start by talking about how a traditional framework such as Rails deals with the problem of getting data from the server to the client. From the docs running in the server file somewhere into your browser.
In a Rails app, since your data doesn’t ever actually move from the server, it stays there, what happens is that the server converts that data, or the data that you need to see, through the page that you’re currently looking at, into a string of HTML and then it pumps that over the wire to you.
Your browser, even though it’s seeing a view of the data, doesn’t really deal with the data in any pure form. Then in a Rails app, or a traditional app, the next time you go to the next page it will just send possibly the same data in a different rendered form in the context of the new page.
Sacha: If we wanted to maybe diagram the flow of data in a Rails app, it would basically go from the database to the Rails code on the server, but it would stop there. Meteor works a little bit differently.
In the previous episodes we’ve talked about how the Meteor came out in order to create a way to build faster more responsive Web apps, and part of that was moving the app’s logic to the client, so that whereas in a traditional Rails app, for example, you click a link it sends a request to the server, the server sends back data, you refresh the browser.
In a Meteor app or any single page Web app, this is all handled by the client, but it’s no use moving the app’s logic if it still has to ask the server back for the data every time, if it still has to make that round trip it’s still going to be slow. Meteor’s work around is to actually bring in some of that data to the client.
Tom: The place that it puts that data on the client is in Minimongo. Minimongo is an example of what Meteor calls a mini database with the future view of having other types of mini database.
What is Minimongo? Minimongo is a client side version of the Mongo database. What that means is that you get the same API that you would on the server, but on the client all you’re doing is really querying your client-side cache of the server’s data.
Sacha: One question you might have is, well how do you get your data from the server to the client? Or in other words, from the real Mongo to Minimongo. There’s multiple ways. One answer, and the one that Meteor provides out of the box is called autopublish. Basically what autopublish does is it takes your entire server-side database and then clones it over to the client.
Tom: This approach is pretty good when you’re just getting started on your app, and you don’t really want to think about publications and subscriptions, because it really just takes the problem away. You can work around the client just as if you’re on the server, and you don’t really need to think about how to bridge that gap. But of course it doesn’t work in practice. You can’t, if you’re Facebook, you can’t publish your entire database to every browser that’s connected to you.
Sacha: There’s performance reasons like it would just take too long to download. There are security reasons, not maybe but definitely you don’t want to publish all your users’ emails.
Tom: You don’t want to send the same thing to each client either, you might want to send one set of data to a user, and another set of data to another.
Sacha: We need a way to control the flow of data, we need to control which subset of our database is actually being copied to a client. That’s exactly where publications and subscriptions come in.
Tom: There are sort of three interesting phases in the process of the data getting from the client to the server. The first is publishing, the second is subscribing, and the third is querying. We’re cover them all in this podcast.
Sacha: Publishing is basically specifying which part of your data you want to make public.
Tom: To do this, you call the media.publish function, in which you just collect a set of data that you want to send back to the client, and you can base that selection on who the user is, and perhaps some arguments they’ve provided to you.
Sacha: It might help to use a simple analogy. For example, imagine you are a grocery store manager. While some items might be on sale, but some maybe are not, and there’s also items that you don’t want to sell to certain people.
For example you don’t want to sell alcohol to minors. Another example could be there’s a certain sale in an item, and the sale is limited to five items per customer. So you have these rules in place that govern which items can be sold and how to sell them. That’s actually a really good illustration of what publication can do.
Tom: Let’s talk about something you might do in a real app.
Suppose you’re running a news aggregator app, and you have a collection of data called posts. You might choose to mark some posts as deleted, and those would be posts you’d never want to publish to any user. That would be analogous to items in the store that are not for sale. Or perhaps you’d want to publish those posts, but only to admin users. So that would be another rule that you might have based on the user ID.
Sacha: Or you might want to control how many posts a user can ask for in one go, and you might want to set a hard limit, for example, to prevent denial of service attacks.
Tom: Finally you might want to control which part of the posts you want to send down. So a more subtle question of, “Hey, we want you to be able to see these posts, but maybe there are some fields on the posts which are private, or that just aren’t relevant to you.”
Sacha: In that sense publications are a way to control which subset of your data is made public and under which conditions.
Tom: But how do you grab that data? How do you use that publication when you’re in the client?
Sacha: Just because all posts are made public doesn’t mean that you want to grab all of them every single time. The client needs a mechanism to ask for the subset of data it actually needs, and that was subscribing does. To go back to our grocery store example, a good analogy would be a customer who walks in the door and asks for a six-pack of beer.
So he’s asking for a specific subset of the items that are for sale in the store, and it’s up to the store, in other words the server, to listen to that request and then either give back the items requested or if it’s a minor, deny the request.
Tom: To take that back to our practical real world example, we might use a post-publication to paginate through 10, and then 20, and then 30 posts as we scroll down the list. We might go to the user’s home page and get a list of all the posts they’ve published, and these would all be subscriptions that we would open on the client as we browse around our app.
The second one listing the user’s posts would take in an ID as an argument for that user, and perhaps would check to see if the currently logged in user is allowed to see that user’s posts.
Sacha: In other words, we will have this post list publication on the server, and if it doesn’t take any arguments it will return the same set of documents to any client that connects to it. But if it does take arguments, well then the clients have the option of passing in these arguments such as the number of posts that they want, a user ID, or maybe a way of sorting the results.
Then according to these arguments, the publication will return the correct subset of documents.
So far we’ve seen how to make posts public and then load them up on the client, but if we just stop there nothing would really be happening. We still need a way to select our data and make use of it. That’s where querying comes in. Querying is done through the collection.find function which is used to select the subset of data you actually want to use.
Tom: If we go back to our grocery store analogy, when you get home from the store and you’ve got your bag of food, you might pick a recipe and select a subset of that food that you need to actually make the recipe.
Sacha: Yeah, you’re not necessarily using all the ingredients that you have at your disposal, you might just use some of them, you might use if you bought salt you might use that in two or three recipes or none of them.
Tom: If we go back to our practical example on the posts home page, we might want to call out the top post in a special area of the page. We might need one query to find that, and then we might need a different query to find the rest of the posts, so that’s built around the page.
Sacha: At this point a question that a lot of people have is why are we writing the same code twice. That’s because on the server you’ll often write code that looks like post.find and then some parameters. Then you end up doing the same thing on the client. But it’s important to understand that those two lines of code, even though they might look similar or sometimes even be similar, have very different roles.
So on the server you’re defining what should be made publicly available, and on the client you’re defining what you will actually use and that’s the specific query. Sometimes that’s the same thing, for example, if you just want to display the last 10 posts and that’s what your publication always returns, well you would have a find of post and a limit of 10 in both cases, but sometimes it’s different. It’s important to understand why that is.
Advanced Publications & Subscriptions
Tom: Perhaps we hinted at it in this episode but you can have multiple publications for a single collection. You might have one publication that publishes a list of posts with a limited set of properties, and another publication that does it one post at a time and publishes a lot more properties, that is very common.
Sacha: Or, you could have multiple prescriptions for one publication. For example, you can have a single publication that publishes a list of posts, and then you can subscribe multiple times with different arguments to control how the data is sorted.
Tom: Finally another thing to consider is where exactly to open your subscriptions. This is an area of debate in the Meteor community. Some people think you should load it all at the top level before doing anything else.
Others thinks the browser should be in charge of loading subscriptions, and another position is the templates themselves should load the data they need as they’re created. There are arguments in favor and against each of those ideas.
Sacha: That’s probably best left for another episode. In this episode we’ve tried to outline the basic concepts that you need to master in order to have a good understanding of publications and subscriptions. Those concepts are not that complex, but it’s true that when you add more layers to your app and more features, publications and subscriptions can get a bit more involved.
It’s important to have a good mastery of the basics, and also to remember that throughout all this you also have to pay close attention to security.
Tom: Because publications are the primary way that you make data from your database available to clients. You really want to make sure you get them right from a security perspective. We’ll go into a lot more detail on this in our next episode which will be on the subject of security.
Sacha: In the meantime if you want to learn more about publications and subscriptions, we have quite a few blog posts on this topic on our blog at discovermeteor.com. Finally, if you enjoyed this episode don’t forget to subscribe on iTunes and maybe even leave us a review to tell us what you think.