Thanks, this will certainly help enable autocasting, at least unless/until a fallthrough mechanism is implemented (proposed by Ed in #28386, updated proposal in #29548).

There are some issues I'm confused about, though:

Is it essential that my fallback function be a "boxed function" and that I pass it to registerBackendFallbackKernel via makeFromBoxedFunction? This makes some sense to me because the catchall kernel must be able to handle any arguments, right?

In the current README for op_registration, your example shows registering an catchall kernel with arguments manually specified (ie, not wrapped in some generic container). Is that example misleading? Will it fail as a fallback for ops that accept different arguments?

All I need is fallback function that declares an RAII exclude guard and forwards the arguments to the current op. It would be nice if a single catchall function could serve as a fallback for any op. Is leaning on the jit operator registry, as your test does, the only way to do so? Is that what you guys are working on/what Ed was alluding to here?

This backend_fallback_kernel looks promising, but it doesn't actually call into op.

For my non-fallback (individually registered) ops, I also have to cast some of the arguments, but this is straightforward, because each wrapper function can accept and handle the op's explicit signature. I wrote a fun metaprogram to handle arbitrary functions as part of my tentative autocasting PR. So I think I'm in reasonable shape on the non-fallback front. A single generic catchall function would enable me to make my PR feature-complete, at least on the C++ side (Python API still under discussion). An eventual implementation of fallthrough will be helpful for performance (avoids an unnecessary lap through the dispatcher) but not essential for functionality.

Is it essential that my fallback function be a "boxed function" and that I pass it to registerBackendFallbackKernel via makeFromBoxedFunction? This makes some sense to me because the catchall kernel must be able to handle any arguments, right?

That's correct. You cannot write an unboxed kernel in a way that works with different operator signatures.

In the current README for op_registration, your example shows registering an catchall kernel with arguments manually specified (ie, not wrapped in some generic container). Is that example misleading? Will it fail as a fallback for ops that accept different arguments?

Catch-all kernels and backend-fallback kernels are different things. A catchall kernel is bound to a specific op and called if it's called with a tensor for which this op doesn't have a concrete backend registered. A backend-fallback kernel is not bound to an op but to a backend type and is called when any op is called with a tensor of that backend type.

All I need is fallback function that declares an RAII exclude guard and forwards the arguments to the current op. It would be nice if a single catchall function could serve as a fallback for any op. Is leaning on the jit operator registry, as your test does, the only way to do so? Is that what you guys are working on/what Ed was alluding to here?

As mentioned in the comment there, the right way would be op.callBoxed() and you should use that whenever it works (i.e. doesn't crash). We'll enable this for more and more ops over time while we fix the system. For the ops that don't work yet, you'll have to lean on the jit operator registry.

This backend_fallback_kernel looks promising, but it doesn't actually call into op.

That is the same API that this PR uses too.

A catchall kernel is bound to a specific op and called if it's called with a tensor for which this op doesn't have a concrete backend registered

But in #29548 we propose to get rid of it, FYI.

Catch-all kernels and backend-fallback kernels are different things. A catchall kernel is bound to a specific op and called if it's called with a tensor for which this op doesn't have a concrete backend registered. A backend-fallback kernel is not bound to an op but to a backend type and is called when any op is called with a tensor of that backend type.

That makes sense. For a second I thought a given catchall kernel was intended for all ops and TensorTypeIds. Thanks for clarifying, I should have looked at the code more closely.

As mentioned in the comment there, the right way would be op.callBoxed() and you should use that whenever it works (i.e. doesn't crash). We'll enable this for more and more ops over time while we fix the system. For the ops that don't work yet, you'll have to lean on the jit operator registry.

I'd like to write a fallback that correctly accommodates your ongoing work. Let's say I imitate the op_registration_test.cpp version, void backend_fallback_kernel(c10::OperatorKernel*, const c10::OperatorHandle& op, c10::Stack* stack). So my function would look like

void autocast_mode_fallback(c10::OperatorKernel*, const c10::OperatorHandle& op, c10::Stack* stack)
{
  c10::impl::ExcludeTensorTypeIdGuard guard(TensorTypeId::AutocastTensorId);
  op.callBoxed(stack);
}

right? If my code snippet here looks reasonable, I'll add it to my PR as a commented-out reminder, so I don't have to bother you again when your stack of PRs is merged. callBoxed does not appear to be a member of OperatorHandle in master but I guess you're adding it in a separate PR.

Out of curiosity, why does the backend_fallback_kernel need to accept both an OperatorKernel* and an OperatorHandle? The former seems unused/unnecessary.

This is the same API but it changed in some of the PRs stacked below this PR.

// old API
void autocast_mode_fallback(c10::OperatorKernel*, c10::Stack* stack)
void autocast_mode_fallback(c10::OperatorKernel*, const c10::OperatorHandle& op, c10::Stack* stack)

// new API
void autocast_mode_fallback(const c10::OperatorHandle& op, c10::Stack* stack)

see #29337 and #29201

Thanks! This will certainly unblock me until fallthrough is available.