Moving Chunks

At any given time, a chunk is hosted at one mongod server. The sharding machinery routes all the requests to that server automatically, without the application needing to know which server that is. From times to time, the balancer may decide to move chunks around.

It is possible to issue a manual command to move a chunk, using the following command:

db.runCommand( { moveChunk : "test.blog.posts" ,
                 find : { author : "eliot" } ,
                 to : "shard1" } )

Parameters:

• moveChunk: a full collection namespace, including the database name
• find: a query expression that falls within the chunk to be moved; the command will find the FROM (donor) shard automatically
• to: shard id where the chunk will be moved

The command will return as soon as the TO and FROM shards agreed that it is now the TO's responsibility to handle the chunk.

Moving a chunk is a complex but under-the-covers operation. It involves two interconnected protocols. One, to clone the data of the actual chunk, including any changes made during the cloning process itself. The second protocol is a commit protocol that makes sure that all the migration participants – the TO-shard, the FROM-shard, and the config servers – agree that the migration has completed.

Example

Suppose we are sharding the test.foo collection using the x field as the shard key. Suppose that we want to move a chunk that looks like:

> db.chunks.find({ns : "test.foo", min : {x : 4}})
{ "_id" : "test.foo-x_17", "lastmod" : { "t" : 43000, "i" : 0 }, "ns" : "test.foo", "min" : { "x" : 4 }, "max" : { "x" : 9 }, "shard" : "bar" }

This chunk is currently on the shard called "bar" and we want to move it to the shard called "baz". We can do this by choosing some value v in the range 4 <= v < 9, so we'll choose 5:

> db.adminCommand({moveChunk : "test.foo", find : {x : 5}, to : "baz"})
{ "millis" : 146880, "ok" : 1 }

MongoDB will find the chunk with {x:5} in its range and move it to shard baz. The command will return when the move is complete.

If someone is already doing something to that particular chunk, it will be locked and you will not be able to migrate it until the other operation is complete.

> db.adminCommand({moveChunk : "test.foo", find : {x : 5}, to : "baz"})
{
        "cause" : {
                "who" : {
                        "_id" : "test.foo",
                        "process" : "ip-10-114-74-220:1299610147:182607800",
                        "state" : 1,
                        "ts" : ObjectId("4d768c5d6bf858cd08a80ac8"),
                        "when" : ISODate("2011-03-08T20:06:53.687Z"),
                        "who" : "ip-10-114-74-220:1299610147:182607800:conn22:510531961",
                        "why" : "migrate-{ x: 5 }"
                },
                "errmsg" : "The collection's metadata lock is already taken.",
                "ok" : 0
        },
        "ok" : 0,
        "errmsg" : "move failed"
}

Troubleshooting

Under certain circumstances, a migrate can fail in a way where the shard cannot figure out what to do next. If this occurs, the shard will terminate (instead of serving possible inconsistent data) with a message such as:

Tue Dec 10 16:15:49 [conn123] ERROR: moveChunk commit failed: version is at 124|1 instead of 125|1
Tue Dec 10 16:15:49 [conn123] ERROR: TERMINATING

If this occurs, make sure:

• All of your config servers are reachable by the shards and mongos processes.
• All of the config servers have the same chunk information for that chunk that failed the migrate.

If the chunk is different, you can restart your mongos servers to reconcile the differences, then restart your shard.