Episode 05: Collections
In this episode of the Discover Meteor Podcast, we tell you everything you need to know about Collections.
- 00:23: What’s a Collection?
- 04:05: Creating a Collection
- 06:57: Collection operations
- 09:00: Optimistic Updates
- 10:30: Using Cursors
- 12:09: From Collections to Models
- 13:36: A Note About Data Structure
- 14:42: Other Useful Packages
- 17:26: Wrapping Up
- Publish Counts package
- Collections documentation
- SimpleSchema package
- Collection2 package
- Collection Hooks package
- Collection Helpers package
- Migrations package
Collections: Meteor’s Core Feature
Sacha: What are we talking about in this episode?
Tom: In today’s episode we’re going to talk about probably the core feature of Meteor, which is the collection.
Sacha: Collections. In my experience it’s something that’s both really core to Meteor. But at the same time, you don’t really think about it that much. You don’t really think about all the cool things that collections do behind the scenes, and you can take it for granted in a way.
Tom: That’s right, because they’re doing so much magic that you really have to interact with it. You don’t even really realize exactly what you’re getting when you create a collection.
It’s probably worth understanding what we mean by a collection and what a collection in Meteor is.
Sacha: First of all, the term collection itself, it comes from MongoDB originally. If you’re familiar with other databases like MySQL or even Excel spreadsheets, you have these tables where you store data. Collection is basically the MongoDB equivalent of a table.
Tom: I guess they use the terminology collection, rather than table because it’s documents that you’re storing, rather than rows.
Sacha: Exactly. Because Mongo doesn’t have a schema, it doesn’t enforce a schema on its data, every document contained in the collection, can follow a different data structure. We’ll talk about this a bit more later, but that’s why they’re collections and not table.
Sacha: One of Meteor’s big features which we talked about a lot in previous episodes is the way it handles and syncs both client and server. But there’s still differences between both environments.
Sacha: On the client, there’s this thing called MiniMongo. As the name implies, it’s a simpler and lighter version of MongoDB that actually runs in your browser whenever you’re interacting with a Meteor app.
Tom: It’s actually like a database in your browser. The reason this is is it acts as a cache of the real database. You have a copy of some subset of the real database in the browser, and you can run queries against it. You can even perform modifications of it right there in the browser, without having to go back to the server.
Sacha: Broadly speaking, a Meteor collection is something that exists in these two environments at the same time, the client and the server.
There are some differences. Some operators, some operations are not available on the client. But, yeah, basically, Meteor’s job is to make all this transparent and keep both sides of the equation in sync.
Tom: You can hear more about how Meteor keeps the client and server collections in sync with each other, in our publications and subscriptions episode. But in this episode, we want to talk more of the API of collections, and what they do exactly. To get us started, let’s talk about how you create one.
Sacha: In practice, to create a new collection, you call “new Mongo.Collection()” and you pass one argument which is going to be the name of the collection inside MongoDB. Just a note here, you often see things like “post = new Mongo.Collection(post).” So you have post twice in there.
Tom: Typically, you write that line when you create the collection in a file that’s available both on the client and the server, which means that that code will execute both on the client and the server.
You will get both a server collection, which is pretty much straight database access, and a client collection, which is a MiniMongo cached version of that Mongo collection.
Sacha: This actually brings up an interesting point, which is what happens when you leave out the MongoDB collection name?
Tom: What happens there on both client and server is that creates what we call a local collection, which is actually just a MiniMongo collection that’s not backed by anything. Ordinarily, when you create a name to MiniMongo collection, you’re implying that there’s a collection on the server that’s pumping data into this collection.
When you create an unnamed MiniMongo collection or a local collection, you’re just making a data store to store data in. But that data isn’t persistent, doesn’t correspond to anything in the database. It’s purely a convenience for storing some data in and running some queries against.
Sacha: This actually points to something that’s pretty interesting, which is that although collections can start as MongoDB collections, there’s nothing preventing you from using them just as regular datastores.
For example, you might use a collection to store enough error messages or any data that you need to query. But that doesn’t need to persist, like you said. Or you can even use them to send data from the server to the client.
Without going too much into details, the “publish-counts” package is a good example of this pattern.
Tom: That’s right. That’s data that which on the client looks like a real collection with a real database collection behind it, but in fact is not. It’s a view into the database. It’s an aggregate of what’s actually in the database.
Sacha: Now that we know how to create a collection, let’s talk about how you interact with one, how you use it in your code base. There’s a bunch of API functions, which are actually very similar to the original MongoDB API, starting with Find.
Tom: Find is simply the way you query the collection for data. You call collection.find, you give it a query, which is a selector with picking data out. You might give it some options about the order that you want data in, or which fields you want from the data.
It returns something called a cursor, which is a description of the query, which you can then use to get data out of them. We’ll talk a bit more about cursors in a moment.
The other querying API for collections is findOne, which short-circuits a cursor and just directly gives you the first document in the collection that matches the argument that you’ve passed in.
Sacha: It’s important to remember that find always returns a cursor, so even if find only returns a single document, it will return a cursor pointing to that single document, unlike findOne that just returns the documents.
There’s also insert, which inserts a document and then returns the ID of the newly inserted document. There’s an update also, very useful. I know that update only modifies a single document by default, unless you pass the multi option.
Tom: There’s upsert, which is a combination of insert and upsert. This is a Mongo operation which will either update a document if it already exists, or insert a new one if it does not. It’s very useful. Finally, there’s remove, which is just the way you delete documents.
Sacha: All these operators can be called both on the client and the server. If you call them on the server, it does what you expect. It modifies the data in the database. But the whole point of Meteor of collections is enabling all these operations on the client as well.
What happens when you do these operations on the client is something that we call optimistic updates, or optimistic UI, or sometimes, a latency compensation. Basically, the client will do the operation as best as it can without waiting on the server to confirm or affirm the result.
When the server does come back with the actual result of the real operation that happened in the database, then the client compares that with whether it came up with on its own, and if needed, then make the correction at that point.
It’s called optimistic updating because the client assumes that everything will just work fine. The point is to make the UI a lot more responsive and a lot faster, because you don’t need to wait for the result of that round trip anymore.
Tom: When you think about this a bit more, you can see why Meteor need to have MiniMongo. It’s to have a client side cache of the data, that behaves pretty much the same as the real Mongo.
Now that we understand the basic API of collections, let’s talk a little bit more about cursors, because their APIs is also really important. Like I said before, a cursor is a description of a query against a database.
It’s not until you actually execute something on the cursor itself that Meteor goes to the database, or, starts fetching from Minimongo to get the data out.
Cursors have a set of basic operations that do this, that maybe look like you might expect. They have a forEach which that runs over every document that matches, they have a Map which does the same, and returns something for each one.
They have a fetch which simply returns an array of every document. They have a count which gives you the number of matching documents.
What’s interesting about all four of those APIs is that on the client, they’re all reactive data sources. If you call them from a reactive context, like a template helper, they will rerun every time the documents that match change. You can hear more about reactive sources in context in our episode on reactivity.
Sacha: I just want to point out that a lot of the time you don’t even need to use these API functions, because Meteor can cause them transparently for you when you use Spacebars, which is the templating language used by Meteor.
If you called each in Spacebars it will just loop over the cursor, and you don’t need to do a forEach or Map beforehand.
From Collections to Models
Earlier, we talked about MongoDB and how it’s a schema-less database, unlike MySQL. But that doesn’t mean that you can’t use a schema with it. It just means that schema is not part of MongoDB. It has to be enforced in your code base.
There’s actually quite a few ways to do that in Meteor. The most widespread is probably a package called SimpleSchema.
Tom: You can use simple-schema to enforce a consistent format for all the data that goes into a given collection.
That means that all your reading code, all your view code that takes the data and does something with it, can have confidence that it will always be dealing with the same shape of data, and doesn’t need to have a lot each cases and guards to deal with, different data formats that could exist.
Sacha: It can also be useful just to vow the data for security reasons, or maybe you want to implement a character limit on the field, or things like that. In fact, there’s another package called Collection2, which takes your schema and then adds that validation layer automatically for you.
Tom: It’s probably worth using them if you want to try and keep consistency in your data.
Sacha: Speaking of collections and collection schemas, one question people often have is why they’re to store everything on the same collection, or try and split things out in multiple collections. In other words, denormalization versus normalization.
This is a big debate. Without going too much into details, one thing we can say is you probably want to avoid relying too heavily on nested documents. To give you a practical example, let’s say you have a photo gallery collection, and each gallery can contain a certain number of photos.
You probably don’t want to just have the photo’s array be a property of the gallery document. Instead, you would have one gallery collection and then one photo collection. Then create relationships between those two collections. The reason why is because Meteor’s publish and subscribe system doesn’t handle nested documents very well.
Tom: That’s right. One way to make it easier to deal with those relationships between different collection is to use the “collection-helpers” package, which enables you to add functions to the prototype of documents that come out of a collection.
So you would either make post.user as a function that returns the user of the current post, which might be handy.
Sacha: It is actually quite handy. It’s a big help in turning collections into something a bit more like models. Even though Meteor isn’t really an MVC, a Model-View-Controller framework, collections are probably the closest thing to models.
If you put together simple-schema, Collection2, collection-helpers, and all these other tools, you get something pretty useful.
Tom: If you want to short-circuit that a little bit and use something that’s a little bit more like a model to begin with, you can take a look at the Astronomy package, which is a heavier layer of abstraction on top of collections.
Sacha: We’ve talked about simple-schema, collection-helpers, Astronomy. There’s a few other useful tools and packages when it come to dealing with collections.
One of them is the “collection-hooks” package, which provides hooks that you can use to trigger callbacks, before or after any collection operations, such as insert, update, remove, and so on. It’s certainly very useful, and one of the packages that you will find in a lot of Meteor apps.
Tom: Another packages that you might want to take a look at is the Percolate’s “migrations” package. Which is something you would use if you ever want to change the structure of the documents that you’re storing in the database.
You have one schema that you defined, using simple-schema, and now you decided to change something like add a field, or rename it, or move a piece of data from one collection to another.
In that case, you probably want to not only write a new schema, but also transform all the existing data that’s in the database into that new schema. Which would involve running a loop into every document in the database, and updating it to the new format.
percolate:migrations package helps you do that. It basically lets you define a series of migrations which you would slowly document as your app evolves over time. It enables you to step forward and backwards through those migrations, so you can reach the state of the application at any point in time throughout its development.
Sacha: Today we talked a lot about collections. On one level, collections are a fairly simple feature. It just works out of the box. It uses the same API as Mongo, so it might seem there isn’t much there, it’s not that innovative.
But I actually think it’s one of Meteor’s core features, if not the core feature of the whole framework.
Tom: Yeah, definitely. We’ve seen a few packages that build off these features that collections give you. Really, you end off with quite a powerful system where the underlying technology seems quite simple, but really isn’t. There’s really not any other framework out there that quite does what collections do for you.
Sacha: Lately we’ve seen the rise of the Facebook stack with things like React for the rendering and GraphQL, Relay, and so on. If you look at GraphQL, although it’s a really awesome piece of technology, it’s not actually realtime and reactive like collections are.
Collections were part of Meteor’s original launch in 2012. Even now, there still isn’t really another system that’s quite in performance and as easy to use.
Tom: I think it’s fair to say that if there’s one thing about Meteor that you should understand, it’s really collections. You don’t need to understand in detail how they work behind the scenes, but it’s really the core of the platform.
Sacha: As always, thanks for listening, and thanks also to everybody who filled out our survey. We asked you guys how we could improve our podcast, and we got a lot of great feedback.
Some of you told us the podcast was sometimes a little bit dull, and could use some lightening up. Some of you complained about our weird accents. There’s not that much we can do about the accents, but we’ll definitely try to make the podcast a little bit more fun and livelier, whilst still covering all bases and being very methodical.
Which actually brings me to the topic of the next episode, which will be all about Meteor methods.