Is your feature request related to a problem? Please describe.
The issue is related to #5620 and #6011. The new offload_states API works only with FusedAdam GPU optimizer. Currently there is no way to offload a trainable model that is using a CPU-based optimizer like DeepSpeedCPUAdam.

Describe the solution you'd like
Extend #6011 to support offloading of a model configured with CPU-based DeepSpeedCPUAdam optimizer.

Thanks,