Ok here it is.

The basic ideas are:

• Metabase drivers (and in the future, other modules) are now separate Leiningen projects under the modules subdirectory.
• To build drivers, we build Metabase as a Leiningen/Maven dependency and install it locally, and the driver subprojects have Metabase as a dependency; when building them they bundle up everything they need (including dependencies) in a driver JAR. New script ./bin/build-driver.sh does all of this for you
• Built drivers are bundled inside the Metabase uberjar as separate JARs under the modules directory
  • When Metabase launches, all of these "core" modules are extracted from the uberjar (or resources directory if running with lein) into your plugins directory
• System modules and any other JARs in your plugins directory are added to the classpath at runtime at launch
  • In order to get this all working I became a JVM ClassLoader wizard and spent days studying Clojure and OpenJDK source to figure out how to get everything to work. Lots and lots of magic including proxy JDBC Drivers and DataSources
  • because of my wizardry you in fact no longer need to pass a -cp argument when using Java 9+, just stick everything in plugins
  • New stuff is also Java-11 compatible, 🆒
• Plugins can/should include a Metabase "plugin manifest" file, named metabase-plugin.yaml, which tells Metabase how to initialize the plugin. See the SQLite one for an example
  • This can do everything from telliing Metabase which namespaces to load to telling Metabase it provides a driver and that it can be lazy-loaded
• Drivers supplied as separate plugins now have lazy-loading capabilities, meaning they are not actually loaded until doing something like connecting to a database of that type. Drivers for databases we're not actually using never get loaded.
  • In my tests not loading classes for drivers we're not using should shave ~200 MB off of memory usage, because having a bunch of unused but loaded classes sitting around eats a ton of memory. This should make running Metabase on Heroku hobby dynos great again™
• To run tests for these drivers in subprojects I wrote a Leiningen middleware plugin, available now on Clojars that adds the dependencies/source paths/test paths of the driver[s] in question to the Metabase project which lets you run tests for that driver like a total champion (i.e., without any complicated/slow build/install cycles)

Future Steps 🚀

While working on this I was thinking a lot about how this could do more than just save a lot of memory usage. All of the following are now real possibilities:

• People who want to ship 3rd-party drivers can now actually do so easily. They can follow the pattern for the drivers we ship as part of the core Metabase project, bundle their driver up as a JAR, and all other people would need to do is drop it in their plugins directory. (This was supposed to work in the past but it never actually quite worked 100%. So it is very exciting that it does now.)

A Metabase plugins store

• It is now 100% feasible to add the capability to download plugins while Metabase is running and install them without restarting.
  • Let the possibilities of that blow your mind for a second
  • Yes, that means it would be totally doable to add something like a Metabase plugins store and let people add a bunch of wacky plugins to implement wacky features to their instance without having to wait for the core team to implement it or running a fork of the project
  • I'm still thinking a lot about how to properly secure 3rd-party plugins, especially if a Metabase store is involved. In a past life I had the misfortune of a being an Android developer, and one of the things I did like about the platform was the way your application asked for permissions: app manifests include a list of required permissions. I envision something like this in the future for Metabase plugins, in the manifest: a list of plugins permissions, like "add API endpoints" or "add a new chart type" or "add a new driver". However, this could be tricky with Clojure being the dynamic language it is -- it's trivial to swap out functions at runtime, or evaluate forms and do nasty things. A bad actor could write a plugin that says it does adds stickers but actually steals your database credentials. 🤢
    • Thus I think we'd want to curate the store and code review anything before adding new plugins it, which would negate some of the benefits of not having to wait for the already-backlogged core team but would keep people from shooting themselves in the foot
    • Alternatively, we could either try to sandbox things, perhaps using something like https://github.com/flatland/clojail. This sounds tricky to get right
    • Or we could tell people to install plugins at their own risk. Sounds like a disaster waiting to happen
    • Curating plugins probably makes sense because otherwise we're basically reinventing package registries and I don't want to find myself in 6 months having to put out a left-pad-gate fire

Plugins admin page

• It would be super simple to add a application database table to track the plugins people have installed and add an admin page to let people see what plugins they have installed and enable/disable them. We could also track a URL to check for updates and let you update a plugin without restarting, or even update plugins automatically. We could even automatically download plugins again if needed when you restart (e.g. if running on Docker or some other situation where the filesystem isn't persisted)
  • With this possibility in mind I added the version and description stuff to the plugins manifest
  • One thing to consider is where the madness ends. Loading tons of plugins can get tricky if certain plugins conflict with others, depend on others, or need to be loaded in a certain order. There is a reason tools like LOOT exist for modding games and "modding" Metabase wouldn't be any different. At some point we will proably have to add some of those sorts of features to the plugins admin page.

Shipping more of Metabase as separate modules

• Shipping more stuff as separate modules is worth considering in my opinion. Breaking stuff into subprojects could help break what is officially considered a "very large" codebase (according to Quora) -- we're sitting at close to 100k LOC of Clojure code alone, and we all know a line of Lisp is worth 20 lines of Java or something like that.
  • Unofficial or unsupported drivers can be spun out into separate GitHub projects instead of shipping as part of Metabase core
  • We could do something like ship the entire API as a separate (versioned) module, which would let you run a different version if needed (or perhaps run different versions of the API side-by-side, under different routes?). This would make versioning the API easier for people that use it to do magical things
  • Shipping custom enterprise modules or other things not part of the core Metabase project would be a lot easier if they were simply implemented as plugins, especially if it meant you could add them to any Metabase instance without having to restart
• If we add logic to update plugins automatically we could ship bugfixes to people without having to ship a new version of Metabase. Now wouldn't that be cool
• It's time to start thinking of Metabase not as a monolithic product but instead as a platform upon which your dreams may come true

Open Questions

• I'm still not sure how plugins will work for JS stuff. Would we simply add another <script> tag to index.html responses? We'd probably need to expose some interfaces for doing stuff since you wouldn't be able to resolve stuff at runtime and just do whatever you wanted the way you could with Clojure because things would be minified/etc? @tlrobinson thoughts?