Some changes occurred in compiler/rustc_codegen_llvm/src/builder/autodiff.rs

cc @ZuseZ4

r? @davidtwco

rustbot has assigned @davidtwco.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

If this code works, then you should also be able to remove some more code, as per:

Is rustc_monomorphize::collector::autodiff::collect_autodiff_fn even necessary in the first place. The catch_unwind intrinsic also needs the function arguments to be codegened, yet it doesn't have any special casing in the monomorphization collector. Enzyme needs the function to differentiate to be in the same module, but I would expect fat LTO to run early enough to already cause that to happen. If you remove collect_autodiff_fn then I would assume this special case in cross_crate_inlineable isn't necessary either.

I marked the locations as fixme in https://github.com/rust-lang/rust/pull/149033/files

Also, did you verify that the test cases still pass with your changes? Here are the instructions to do so:
https://rustc-dev-guide.rust-lang.org/autodiff/installation.html#build-instructions

r? me

Sounds good, I will update the code and ensure it will pass the test cases.

Hi there, I checked out this commit, which is the last commit before I made the commits in this PR.

I tried to follow the build steps in this link you gave: https://rustc-dev-guide.rust-lang.org/autodiff/installation.html#build-instructions, but the build fails with errors due to being unable to compile core and rustc-demangle. Both errors are caused by "process didn't exit successfully" and have this at the end: exit code: 0xc0000005, STATUS_ACCESS_VIOLATION.

Could you please help with this issue?

That's unfortunately missing too much information to be able to help. Can you create a separate issue and fill out the form? E.g. OS, stacktrace, logs, etc.

Hi there, I have created an issue here, could you take a look? If there is any other information you need, please let me know. Sorry for the inconvenience.

I was able to sucessfully create the build using WSL. When I try to run the tests on this commit, the test case with the command ./x test --stage 1 tests/codegen-llvm/autodiff fails with failures at the given tests:

failures:
    [codegen] tests/codegen-llvm/autodiff/batched.rs
    [codegen] tests/codegen-llvm/autodiff/generic.rs
    [codegen] tests/codegen-llvm/autodiff/identical_fnc.rs

test result: FAILED. 8 passed; 3 failed; 0 ignored; 0 measured; 955 filtered out; finished in 325.49ms

The other 3 test cases has their tests either passed or ignored.

In this commit (my most recent one) 9beb95d, it fails at the same tests and succeeds on everything else.

How should I proceed?

I need to look at what to do with the batching feature, since it's fragile. For the other two, it's likely just rustc improvements leading to better codegen and failing tests. As long as the majority passes you should be good to continue. So just try to remove the mentioned code and see if you can still get it to pass the same tests.

I removed the code in the fixme's from the files in https://github.com/rust-lang/rust/pull/149033/files.

However, I get these failures with this command ./x test --stage 1 tests/codegen-llvm/autodiff

failures:
    [codegen] tests/codegen-llvm/autodiff/identical_fnc.rs
    [codegen] tests/codegen-llvm/autodiff/trait.rs
    [codegen] tests/codegen-llvm/autodiff/void_ret.rs
    [codegen] tests/codegen-llvm/autodiff/generic.rs
    [codegen] tests/codegen-llvm/autodiff/batched.rs

test result: FAILED. 0 passed; 5 failed; 6 ignored; 0 measured; 955 filtered out; finished in 163.10ms

and these failures with this command ./x test --stage 1 tests/ui/autodiff

failures:
    [ui] tests/ui/autodiff/no_lto_flag.rs#with_lto
    [ui] tests/ui/autodiff/no_lto_flag.rs#no_lto
    [ui] tests/ui/autodiff/incremental.rs#DEBUG
    [ui] tests/ui/autodiff/incremental.rs#RELEASE

test result: FAILED. 0 passed; 4 failed; 10 ignored; 0 measured; 20138 filtered out; finished in 377.54ms

There seems to be another issue that causes these errors, but I am unsure where it comes from.

@wilbertbw sorry I had lost a bit track of this one. Did you figure out how to get the codegen tests to pass, with the suggested changes? Also (I guess you noticed), we improved how we distribute autodiff, so I hope main should now build on wsl without changes?

I just pulled for main and reran the tests on WSL. On the main branch, all of them except ./x test --stage 1 tests/codegen-llvm/autodiff passes the tests. Here are the errors for that test (the only difference from before is the failure of [codegen] tests/codegen-llvm/autodiff/abi_handling.rs#release):

failures:
    [codegen] tests/codegen-llvm/autodiff/generic.rs
    [codegen] tests/codegen-llvm/autodiff/identical_fnc.rs
    [codegen] tests/codegen-llvm/autodiff/abi_handling.rs#release
    [codegen] tests/codegen-llvm/autodiff/batched.rs

test result: FAILED. 7 passed; 4 failed; 0 ignored; 0 measured; 962 filtered out; finished in 270.86ms

The same occurs with this commit (does not have the changes mentioned in the files) 41703e0.

I am also unable to find a way to get the tests to pass with the suggested changes.

The following are the failures, the only difference from before is the failure of [codegen] tests/codegen-llvm/autodiff/abi_handling.rs#release.

./x test --stage 1 tests/codegen-llvm/autodiff

failures:
    [codegen] tests/codegen-llvm/autodiff/identical_fnc.rs
    [codegen] tests/codegen-llvm/autodiff/generic.rs
    [codegen] tests/codegen-llvm/autodiff/abi_handling.rs#release
    [codegen] tests/codegen-llvm/autodiff/trait.rs
    [codegen] tests/codegen-llvm/autodiff/void_ret.rs
    [codegen] tests/codegen-llvm/autodiff/batched.rs

test result: FAILED. 5 passed; 6 failed; 0 ignored; 0 measured; 962 filtered out; finished in 257.44ms

./x test --stage 1 tests/ui/autodiff

failures:
    [ui] tests/ui/autodiff/no_lto_flag.rs#with_lto
    [ui] tests/ui/autodiff/incremental.rs#RELEASE
    [ui] tests/ui/autodiff/incremental.rs#DEBUG
    [ui] tests/ui/autodiff/no_lto_flag.rs#no_lto

test result: FAILED. 6 passed; 4 failed; 4 ignored; 0 measured; 20297 filtered out; finished in 5.97s

Sorry for the inconvenience.

I also created a commit (da0ad99) for the suggested changes in https://github.com/rust-lang/rust/pull/149033/files

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

In general, tests/ui/autodiff is run in CI, so that indeed should work on main.
I'm a bit confused why it fails with these patches.
codegen-llvm/autodiff are the only really interesting tests. A few of them failing on mine is fine since we don't test it in CI yet. If more things are failing after following the fixme's we should investigate why though.

In general, does autodiff still work with all commits applied (especially the ones that delete the extra code)?
It's best to verify that outside of the rust build folder or the test subdirectory. Just create a new cargo project somewhere else and try to differentiate some simple functions there. If that works we can have a closer look at the tests.