I've proposed this several times in the past (mostly on grabbit), but I think it's time to take the plunge and use a relational DB for all the indexing/querying operations. This will solve a bunch of our metadata-related problems (see e.g., #398, #397).

My plan is to use dataset, which wraps SQLAlchemy. The DB itself will be in-memory SQLite; I don't anticipate running into many BIDS datasets so large that memory becomes an issue. Switching to a DB-based approach will also make it much easier to save/load the entire index to avoid the re-indexing process.

If anyone has any objections/suggestions, now is a good time to raise them, otherwise I plan to work up something in the next few days.