Description
The kubernetes driver currently only supports creating/using deployments for buildx nodes. This makes it difficult if we want to use local caching - as we can't attach a separate volume to each replica in a deployment (buildkit doesn't support sharing a cache between nodes). A statefulset allows us to attach a volume to each node to store cache, but the kubernetes driver only looks for deployments when looking for existing nodes to use.
So we either have to have a single buildkit node, or store cache in emptydirs - neither of which is ideal.
Description
The kubernetes driver currently only supports creating/using deployments for buildx nodes. This makes it difficult if we want to use local caching - as we can't attach a separate volume to each replica in a deployment (buildkit doesn't support sharing a cache between nodes). A statefulset allows us to attach a volume to each node to store cache, but the kubernetes driver only looks for deployments when looking for existing nodes to use.
So we either have to have a single buildkit node, or store cache in
emptydirs - neither of which is ideal.