Rollup only had a very minor build time improvement over webpack and it now sits at around 5 seconds.
Meanwhile I created the BYTM Plugin Template which uses vite and it compiles within half a second.
Migration would encompass a rework of the build process but I think it would still be worth it.