HasMany::set() should be optimized to avoid unnecessary removes & adds. From the long term, we could also delegate some handling via storage, but for now only the PHP runtime comparison could be utilized.

We should fetch the data, count the diff and add and remove only the needed.