I think that I might have spotted a mistake in release mode @Piinks @HansMuller of #64101:

The _unregister line will not remove the state from the hash set in release mode:

  void _unregister(ScaffoldState scaffold) {
    assert(_scaffolds.remove(scaffold));
  }