There’s a private parameter in the SGD optimizer, sgd.h:

private:
  // Counts how often `step()` is called, for dampening.
  size_t iteration_{0};

in sgd.cpp:

      const auto dampening = iteration_ == 0 ? 1 : 1 - options.dampening_;
      auto& momentum = buffer_at(momentum_buffers, i);
      momentum = (options.momentum_ * momentum) + (dampening * update);

I don’t see how to restore the state of a previously run SGD optimizer
with out access to get/set member: iteration_
restoring the buffers will still result in the dampening factor being 1,
as though the optimizer is being run for the first time.
(it looks like the iteration count could also just be a flag indicating whether the optimizer has run once or not)

cc @ezyang @gchanan @zou3519