SemanticBuilder's already resolved reference usage but never doesn't set reference_flag for IdentifierReference.

Although we can get reference_flag by reference_id, would be great if we could directly reference_flag.