It seems that the fp16 setting is not effective. I tried to use fp16 manually, and offload the autoencoder and CLIP to cpu memory during ddim denoising, and can run a 45x512x768 video on 12G GPU memory.