Now this is just a thought, but would it be possible to save the optimizer state in a separate file next to the hypernetwork? That would remove the need to prune the file afterwards.

Optimizer "sidecars" might also enable reuse of an optimizer state when restarting from scratch? Please disregard if that would not work, I'm not familiar with the technical details.
[Edit: This comment was invisible until 2022-11-11 as my fresh account was flagged at the time of writing, thankfully @aria1th considered it after I emailed them]

is there any demonstration of the beneft this brings

@AUTOMATIC1111 Yes, AdamW (and Momentum based optimizers) uses adaptive learning rate, which is estimated from its momentum.

If we try to start from zero, AdamW will use given pure learning rate, and observe the effect.

If we resume properly, AdamW uses given trajectory, generally lower learning rate.

Loading optimizer state does not make training deterministic, since it still uses random when its trying to avoid local minima.

Multiple users in discord reported that HN training continuing from saved checkpoints does not works well, especially at beginning, HN sometimes tends to 'die' quickly for some reason. Also its is known that generally lowering learning rate down at 'resuming' helps it.

This should be an optimizer's fault - from not knowing anything about its previous trajectory.

Here's a general discussion about loading optimizers.

With this patch, I never observed drastic style change or transition of preview images like before. But this might mean that someone might want to nuke optimizer state to intentionally trigger style change / transition?

Someone suggested me that if we can have separate file for optimizer, which can be distributed separately.
I think its a good idea, so I'll try working on it.

Finished implementing and testing.

Files will be saved separately as *.pt and *.pt.optim file.

To use valid optimizer, it will now use hash value itself.

Closes #4048

Temporarily closed for resolving conflict

Finished testing again:
HNs file itself won't contain optimizer in any case.

1. Tested if HN can be saved and loaded with separate optimizer file.
2. Tested - HNs itself are properly saved without hash in name
3. Tested - HNs will not use optimizer file if its saved hash is different.
4. Tested - HNs won't save optimizer file if the option is disabled.

I think there isn't a scenario where the user would want to put optimizer opts into checkpoint itself when saving to separate file exists. I don't want to have useless options so please remove that and the code to support it.

Yeah I removed that option (merging optimizer in checkpoint itself). Now its only saving it into separate *.optim file.
I'll change explanation in shared.py

@aria1th Whenever you load a hypernetwork that also has a .optim file in the hypernetwork directory, the UI says that it not only loads the hypernetwork, but also the optimizer. Is this intended? I'm not trying to resume training with the optimizer, I'm just selecting the hypernetwork for normal inference.