Tailable Cursors

Tailable cursors are only allowed on capped collections and can only return objects in natural order. Tailable queries never use indexes.

A tailable cursor "tails" the end of a capped collection, much like the Unix "tail -f" command. They key idea is that if we "catch up" and have reached the end of the collection, our position is remembered rather than the cursor being closed. Thus, after new objects are inserted, we can resume retrieving from where we left off – which is then very inexpensive.

If the field you wish to "tail" is indexed, do not use tailable cursors; instead simply (re)query for { field : { $gt : value } } where value is where you last left off. This is normally quite efficient. Tailable cursors are for cases where having an index would be prohibitive (extremely high write collections). If performance is not problematic, use a normal query and cursor, tailable adds some complexity.

As no index will be used for the query, the initial scanning to find the first object to return will likely be quite costly. However once found, retrieving additional data from new inserts then becomes very inexpensive.

The cursor may become invalid if, for example, the last object returned is at the end of the collection and is deleted. Thus, you should be prepared to requery if the cursor is "dead". You can determine if a cursor is dead by checking its id. An id of zero indicates a dead cursor (use isDead in the c++ driver). In addition, the cursor will be in "dead" state after a query which returns no matches.

MongoDB replication uses tailable cursors to follow the end of the primary's replication op log collection. Writes to the oplog would be slower with an index. The tailable feature eliminates the need to create an index for replication's use case.

C++ example:

#include "client/dbclient.h"

using namespace mongo;

/* "tail" the namespace, outputting elements as they are added. Cursor blocks
 * waiting for data if no documents currently exist. For this to work something
 * field -- _id in this case -- should be increasing when items are added.
 */
void tail(DBClientBase& conn, const char *ns) {
  // minKey is smaller than any other possible value
  BSONElement lastId = minKey.firstElement();

  Query query = Query().sort("$natural"); // { $natural : 1 } means in forward
                                          // capped collection insertion order
  while( 1 ) {
    auto_ptr<DBClientCursor> c =
      conn.query(ns, query, 0, 0, 0,
                 QueryOption_CursorTailable | QueryOption_AwaitData );
    while( 1 ) {
      if( !c->more() ) {
        if( c->isDead() ) {
          // we need to requery
          break;
        }
        // No need to wait here, cursor will block for several sec with _AwaitData
        continue; // we will try more() again
      }
      BSONObj o = c->next();
      lastId = o["_id"];
      cout << o.toString() << endl;
    }

    // prepare to requery from where we left off
    query = QUERY( "_id" << GT << lastId ).sort("$natural");
  }
}

Javascript example:

var coll = db.some.capped.collection;
var lastVal = coll.find().sort({ '$natural' : 1 })
                         .limit( 1 ).next()[ 'increasing' ];
while(1){

  cursor = coll.find({ 'increasing' : { '$gte' : lastVal } });

  // tailable
  cursor.addOption( 2 );
  // await data
  cursor.addOption( 32 );

  // Waits several sec for more data
  while( cursor.hasNext() ){
    var doc = cursor.next();
    lastVal = doc[ 'increasing' ];
    printjson( doc );
  }

}

Tailable Cursors

See Also

Labels