Recovering Deleted MongoDB Data with mongorestore

As a professional full-stack developer and database engineer with over 7 years of MongoDB experience, I‘ve helped dozens of companies recover lost data from faulty database drops or corruptions. In this comprehensive 3200+ word guide, I‘ll share everything I‘ve learned about successfully restoring MongoDB data from backups.

We‘ll cover:

• Core concepts for MongoDB backup & restore
• Step-by-step mongorestore usage cases
• Restoring entire deleted databases
• Recovering dropped collections
• Merging backups with existing collections
• Controlling parallelism and index recreation
• Implementing advanced incremental backup strategies

And plenty of hard-won pro tips scattered throughout! Buckle up for in-depth knowledge on MongoDB restores.

Why Backup & Restore Matter

Recent surveys on causes of application outages reveal an unfortunate truth:

Despite modern infrastructure resiliency, accidentally running the wrong script or clicking the wrong button can still happen. When that involves precious production databases, backup & restore is your safety net.

Having worked with over 50 companies suffering data issues, I always stress the critical nature of usable, tested backups. Let‘s overview core concepts before diving into restores.

Types of Backups

MongoDB supports two main backup types:

Logical backups with mongodump
Human-readable BSON documents dumped from MongoDB instances. Allows granular restores down to a single collection.

Snapshot backups with filesystem snapshots or plugins
Low-level snapshots of data files, capturing exact on-disk state. Faster full restores but less flexible.

This guide focuses solely on logical mongodump backups. Testing both backup types is ideal for comprehensive recoverability.

Backup Cadence Strategies

Base cadence: Frequent logical backups (daily or twice daily)
Supplementary cadence: Periodic on-demand logical & snapshot backups

Shorter logical backup intervals minimize potential data loss. Larger periodic snapshot backups provide longer history and guard against data corruption.

Storage Location & Validation

Store backups externally from database servers! Backups do no good on a corrupted machine.

Cloud object stores like S3 work excellently – just ensure database instances have outbound connectivity.

Post-backup, validating restores before needing them is vital. What seems backup up may not actually be restoreable…

Now let‘s dive into making those backups usable!

mongorestore In Action

The mongorestore utility handles replaying MongoDB backups created by mongodump. It:

• Restores logically into a running MongoDB instance
• Recreates databases, collections and indexes
• Insert, overwrite or merge with existing data

Its basic syntax is:

mongorestore --db <targetdb> <dump files location>

But real-world usage gets considerably more advanced. Let‘s walk through example cases.

Restoring Accidentally Dropped Databases

Situation: A tired sysadmin accidentally runs mongo mydb --eval "db.dropDatabase()" late one Friday night…

Reaction: Panic, multiple expletives, frantic checking if backups exist (phew!)

Fix: Simply restore the most recent backup:

mongorestore /backups/mydb-2023-02-23

This will:

1. Recreate the mydb database
2. Iterate through the bson files, recreating those collections with data
3. Rebuild indexes post-insert

Assuming good data in the source dump, mydb now looks good as new!

Pro Tip: When restoring an entire database, you can pass any collection dump filepath instead of the top level. MongoDB uses metadata to determine original DB/collections.

Restoring Dropped Collections

Dropping a critical collection loses data just as quick. Let‘s consider:

use blog; 

db.posts.drop(); // oops!

We want to restore blog.posts collection specifically from backup:

mongorestore -d blog -c posts /backups/blog/posts.bson

However, that alone won‘t recreate necessary indexes!

We must include --drop to purge remnants and rebuild indexes:

mongorestore -d blog -c posts --drop /backups/blog/posts.bson  

And data now restores safely and completely.

Pro Tip: Always consider index recreation when restoring collections directly.

Merging Collection Data from Backup

Say we receive updated legacy data that needs merging into an existing collection.

We extract the relevant docs to a bson file and want to insert any changes, but retain existing documents.

mongorestore can handle this with --maintainInsertionOrder:

mongorestore -d reporting -c purchases --maintainInsertionOrder /updated_purchases.bson

This smartly inserts new docs and overwrites any exact _id matches. Result: merged updated content!

Pro Tip: Specify --objcheck to validate all inserted docs against the collection schema too.

Controlling Parallel Restores

Insertion speed during larger restores matters when aiming for rapid recovery.

By default mongorestore linearly inserts docs from the dump files. Adding some parallelism improves throughput significantly.

Parallel Collections

Consuming multiple dump files at once happens automatically. If restoring many collections, mongorestore will insert into each concurrently.

Parallel Documents

Restoring huge single collections can enable the --numInsertionWorkers option:

mongorestore -d archival -c events --numInsertionWorkers 8 --drop big_events.bson

This utilizes 8 threads to parse bson and insert documents simultaneously.

In my testing on beefy servers, anywhere from 8-16 threads helps optimize batch document insertion speed.

Disable Index Recreation

Automatic index builds after large inserts also hinder performance.

Postpone index builds by using --noIndexRestore, the restore documents much quicker:

mongorestore -d logs -c events --noIndexRestore /backups/logs

Then later run normal index building commands.

Pro Tip: Dropping indexes first aids rebuilding, preventing duplication conflicts.

Incremental Backup & Restore Patterns

Weekly full backups plus daily incremental backups form a common cadence. Mongodb supports this well.

Incremental Backups

The key to incremental dumps lies in the --query flag for mongodump:

# Initial full backup
mongodump -d archival -o /backups/archival 

# Daily incremental dumps
mongodump -d archival -o /backups/archival-inc --query ‘{ $and: [ {_id: {$gt: ObjectId(...) } }, {"updatedAt": { $gt: ISODate(...) }} ] }‘

This queries for docs with _id values higher than the last dump, plus any that updated after a timestamp. Just those documents get written incrementally.

Pro Tip: Include an updatedAt timestamp field on mutable docs to enable incremental patterns.

Incremental Restores

Simply apply the backups sequentially:

mongorestore /backups/archival  

mongorestore /backups/archival-inc-20230224
mongorestore /backups/archival-inc-20230225  

Mongo logically handles inserting, overwriting and skipping docs correctly per incremental dump.

Easy rollback or point-in-time recovery! Just stop restoring incrementals at a usable data state.

Pro Tip: Delete old backups from disk as they age out of usefulness.

Final Thoughts

That wraps my guide to mastering MongoDB restores! We covered:

• Backup types, cadence strategies and storage
• Common mongorestore usage cases
• How to merge document data safely
• Tuning parallel insertion and indexes
• Enabling incremental backups

Reliably recovering from disaster is every bit as important as prevention. I hope walking through these tangible mongorestore scenarios helps demystify rehydrating precious MongoDB data.

Questions or restoration war stories? Share them below!