Skip to content

Improve Fuzz CI#664

Merged
TheBlueMatt merged 6 commits intorust-bitcoin:masterfrom
RCasatta:fuzz_ci
Sep 29, 2021
Merged

Improve Fuzz CI#664
TheBlueMatt merged 6 commits intorust-bitcoin:masterfrom
RCasatta:fuzz_ci

Conversation

@RCasatta
Copy link
Copy Markdown
Collaborator

My original plan was to run fuzz testing in CI separately with both x86_64 and i686 arch.

Unfortunately, I got issues with the i686 fuzzing https://github.com/RCasatta/rust-bitcoin/actions/runs/1265153484 and apparently specifying different targets is not supported https://github.com/rust-fuzz/honggfuzz-rs/blob/f45aef1b59736fcfc215eca09aca42e49bbf36db/src/bin/cargo-hfuzz.rs#L23

Nevertheless, I think there are some advantages in doing fuzzing in CI separately (faster, better report), however, there are some disadvantages (what launched from the CI is not exactly the same script launched locally). For this reason, I am not sure if 3e310d3 is desirable and if yes, if we want to remove fuzzing in the nightly test.

In case we keep the fuzzing in the nightly test I think 9049eef is desirable because it saves about 30 seconds for other tests that don't need those deps.

strategy:
fail-fast: false
matrix:
fuzz_target: [deser_net_msg, deserialize_address, deserialize_amount, deserialize_block, deserialize_psbt, deserialize_script, deserialize_transaction, outpoint_string, uint128_fuzz]
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can't generate this from the file list, can CI somewhere fail if this misses a fuzz target?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, check done in 1aefc1c

steps:
- name: Install test dependencies

- name: Install fuzz dependencies
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't this be dropped entirely now?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, done in 9b6b50a

run: ls -R
- run: find executed_* -type f -exec cat {} + | sort > executed
- run: ls fuzz/fuzz_targets | sort > expected
- run: diff expected executed
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you want some kind of if statement that if there's a diff we fail the test.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The if isn't necessary, diff returns not 0 if files are different. See my test where I explicitly removed one fuzz target https://github.com/RCasatta/rust-bitcoin/runs/3698414243?check_suite_focus=true

Copy link
Copy Markdown
Member

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, mod one comment.

@dr-orlovsky dr-orlovsky added this to the 0.28.0 milestone Sep 25, 2021
@dr-orlovsky
Copy link
Copy Markdown
Collaborator

Why do we have such a big diff with all temporary files (which should never get into git) removed?

@RCasatta
Copy link
Copy Markdown
Collaborator Author

RCasatta commented Sep 27, 2021

which should never get into git

Those files are fuzzing test cases stimulating different program paths. Fuzzers could recreate those by running and searching for various inputs, but it could take a long time. Providing those to the CI ensures many program paths are tested even if the fuzzing runs for just 30 seconds, that's why they are committed into git.

Why removed?

fuzz targets deserialize_decimal and deserialize_udecimal are not there anymore (maybe renamed?) so that files are not used anymore.

@RCasatta
Copy link
Copy Markdown
Collaborator Author

Using rust stable toolchain since honggfuzz is supporting it -> #667 (comment)

Copy link
Copy Markdown
Member

@apoelstra apoelstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 2bbf63c

Copy link
Copy Markdown
Collaborator

@dr-orlovsky dr-orlovsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK 2bbf63c

Awaiting for @TheBlueMatt confirmation that he is satisfied on the answer to his comment and after that will merge (or he can merge) to get CI fixed and unlock further merging of other PRs

Copy link
Copy Markdown
Member

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, sorry, yes, my comment was resolved.

@TheBlueMatt TheBlueMatt merged commit 23ccc58 into rust-bitcoin:master Sep 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants