For other types, deserialize_graph populates a self.serialized_name_to_meta table, which we then use in sync_fx_node to fill in the correct val. Why are custom objects taking a different path here?

This is assuming that our val is the custom obj itself, if we serialize to serialized_name_to_meta, then the serialized custom object will live in both serialized_name_to_meta, which is attached to the serialized exported program, and in constants, which is saved at a different location from the exported program. I was thinking it's probably not good to save one thing in two places, and especially if it's a custom object blob..?

serialized_name_to_meta is a field on GraphModuleDeserializer and isn't persisted to the final ExportedProgram, as far as I can tell. It's just bookkeeping to map an IR name to meta['val'], which seems like what we're doing here. So tracking it there should not affect the serialized format.

I think the more fundamental issue is that we don't have a serialized table mapping names to custom objs in schema.Graph like we do for every other dynamic value type (tensors, symints, symbools). We are representing it implicitly through placeholder arguments/inputs to nodes.

The asymmetry of how we handle these nodes is a good signal that there's a missing piece in our design. The missing piece is that we don't have a metadata object for ScriptObjects: Tensor : TensorMeta :: ScriptObject : ???.

If we had such a piece (call it ScriptObjectMeta), the right fix would be:

1. schema.Graph gets a custom_object_values: Dict[str, ScriptObjectMeta] field, analogous to tensor_values.
2. We deserialize that and populate serialized_name_to_meta like everything else in GraphModuleDeserializer.deserialize_graph.
3. meta["val"] should contain ScriptObjectMeta, not the ScriptObject itself.

Okay, so what should ScriptObjectMeta look like? If we were being faithful to the Tensor : TensorMeta analogy, it should be something that we can deserialized into a FakeScriptObject that will work when invoking the node in FakeMode. So that's probably the right thing.

That's out of scope for this diff. But we should try to design things with a structure as close as possible to the right thing, so that when we go fix it later it will be easier (and we don't create weird divergences in the meantime).

My suggestion for a temporary ScriptObjectMeta would be a dataclass that looks like:

@dataclass
class ScriptObjectMeta:
  # key into constants table to retrieve the real ScriptObject.
  # Optional because not all ScriptObject nodes are from constants.
  Optional[constant_name]: str  

  typename: str  # fully qualified type of the ScriptObject. 

What do you think?

From your comment I realized I was implementing it stupidly, so I updated the code, but not exactly what you said. Let me know what you think! Basically I just populated serialized_name_to_meta based on the signature, since the signature stores the custom obj's node name and fqn to the constants dictionary.

The reason why I didn't make another Dict[str, ScriptObjectMeta] is because I felt that the signature already stores something similar to that already InputSpec(kind=InputKind.CUSTOM_OBJ, arg=CustomObjArgument(node_name), target=custom_obj_fqn).

Based on what we discussed, I added a ScriptObjectMeta, which stores just the name of the constant in the constants table.