As far as I can tell, the local time is never synced back up to the remote time, except when a rather large threshold is reached. You might want to read this article by riotgames how they solved this issue in League of Legends.
https://technology.riotgames.com/news/determinism-league-legends-unified-clock#Clock%20Sync

The main point is:

We don’t simply snap to a time, as that can create jarring in-game artifacts. Instead we gradually filter the client clock times to slowly approach server time.

Also, you might want to look into using a sliding sum instead of starting the synchronization process from scratch every time