Add ability to use virtual dispatch to stagesTuple by radl97 · Pull Request #801 · AFLplusplus/LibAFL

radl97 · 2022-09-26T14:40:31Z

I often find myself trying out different configurations on every aspect of the fuzzing. Especially ObserversTuple, StagesTuple, but I think there could be a discussion over more generic parts of the code.

With this change, I could configure stages at run-time (with CLI flags in my case):

let concrete_stages = tuple_list!(
  ...
);

let mut stages = Box<dyn StagesTuple<_,_,_,_>>;

if args.is_present("flag") {
  // Con-style stages
  stages = Box::new((AdditionalStage::new(), concrete_stages));
else {
  stages = Box::new(concrete_stages);
}

fuzzer.fuzz_loop(stages.as_mut(), &mut executor, &mut state, &mut mgr);

I would like to add this to more parts of the code, because I want to run quick sanity-checks.

Note that my main issue is that I have to write out full types if I have to e.g. extract a function which passes or returns ObserversTuple. I am trying to wrap my head around alternative solutions, but this is the first which seems manageable, and does not seem to impact performance.

What do you think? Also, should I extract the broader issue (about other types) for discussion?

domenukk · 2022-09-26T14:49:39Z

This solution seems good, but for flags you can actually just wrap your stages in SkippableStages

LibAFL/libafl/src/stages/mod.rs

Line 284 in d17269d

pub struct SkippableStage<CD, E, EM, S, ST, Z>

That way they will be called at runtime, the flag will be checked, and they will be skipped.
Not including them at all in the tuple seems pretty good too

radl97 · 2022-09-26T15:33:42Z

Oh, thanks! I did not know that!

Clarifying my point, I have not found solutions for most of these problems (below and above), which does not result in code duplication (due to the restriction of types), and does not result in explicit type annotations (near e.g. function result/parameters). I would also allow some perf suffering to enable fast prototyping (interchangeable functions, etc.), but I understand that Rust's performance is one of the main focus of this project.

I've looked around my code, and found some other cases I would like runtime configurations:

ObserversTuple (already mentioned)
StagesTuple (already mentioned, alternative is SkippableStage)
LLMPEventManager vs SimpleEventManager. For me, this is the main difference with prod vs testing. It seems a lot more involved (due to e.g. lambda functions, which is the thing that annoys me the most.)
CommandExecutor was seamless, but I could not pinpoint why the difference.
Input, would be useful for prototyping and testing out different representations of the input (needed by different structured mutations)

On request, I would try to extract parts of the code to show concrete examples. Probably I missed some more specialized structures, like the SkippedStage mentioned above.

domenukk · 2022-09-29T00:32:09Z

I don't see any downsides to removing the sized requirement here, so accepting this for sure, thx :)
Let's discuss in an issue or discord what else we need to do

andreafioraldi · 2022-09-29T12:48:14Z

The main issue with using dynamic dispatching, apart the perf hit, it is the restriction of these types to be owned and contain no references.

Atm, we support dynamic dispatching of the observers (assuming they are owned, that is not the case for many observer like mapobserver) only when using rust nighly because we need specialization to do that.
I you use the unstable feature then you have ObserversOwnedMap to play with at https://github.com/AFLplusplus/LibAFL/blob/main/libafl/src/observers/owned.rs

andreafioraldi · 2022-09-29T12:49:22Z

Dynamic stages are supported too with StagesOwnedList https://github.com/AFLplusplus/LibAFL/blob/main/libafl/src/stages/owned.rs , and this works on Rust stable too

* Add ability to use virtual dispatch to stagesTuple * Fix lint

* Mac OS Autotokens (#723) * mac_tokens * more * win fix * fmt * fmt c * Use nightly fmt (#728) * Fix compilation for aarch64 qemu (#731) Typo lead to fail to compile for arm64 * Simd Fix (#729) * simd fix * fmt * Fixing readme & docs (#730) * fix * fix * add * add * fmt * 0.8.1 (#732) * New Pass Manager Arguments (#724) * new pm arguments * enable abgeana's code * Fix tui with 1 client (#734) * unbreak tui with 1 client * clippy * Add core affinity support for FreeBSD (#736) * NYX Executor (GSoC '22) (#693) * Add ccache * Update codecov.yml * Add libnyx * Fix * Add nyx build script * Fix build.sh && init executor.rs * Fix commit * Fix code * initialize `exector.rs` * refine API in `nyx_bridge.rs` * initialze `run_target` * add `test_nyxhelper` * initize `test_executor` * remove `nyx_beidge.rs` * make `test_executor` compile * Improve test * refine code * update version * fix docker * fix docker * Fix clippy * Fix build * fix build && add `set_timeout` * Fix and refine CI * fix CI * Fix CI * Add platform restrict * cargo fmt * add parallel mode * add example `nyx_libxml2_parallel` * fix fuzzer example * fix CI * add README * fix CI * fix CI * fix CI * remove unwrap and NyxResult * code format fix * add libnyx's rev * fix format * change Duration format && Fix CI * caego fmt * fix CI * fix CI * Add doc * test CI * Update test_all_fuzzers.sh * Update test_all_fuzzers.sh * Update test_all_fuzzers.sh * add cache for apt and cargo-install * Update build_and_test.yml * Update build_and_test.yml * tmp test CI * fix CI * remove debug cmd * remove test * code refine * code refine * code refine * code refine * add Makefile * fix example doc for nyx * add `NyxHelper::new_with_initial_timeout` * fix `NyxHelper::new` * fix curl parameter * code refine * add check for setup script * use afl-clang-fast in nyx * fix logic * fix makefile * fix CI * Update build_and_test.yml * Update build_and_test.yml * remove debug cmd Co-authored-by: syheliel <syheliel@gmail.com> Co-authored-by: Dominik Maier <dmnk@google.com> * Fix spelling error (#745) * OSX force_load option (#743) * Update clang.rs * fmt * Add continous JSON Logging monitor (#738) * Add simple JSON Monitor * Add documentation * Log global state * Fix formatting * Save state depending on closure outcome, have file opened all the time * Make OnDiskJSONMonitor cloneable * Switch to FnMut to allow stateful closures * Use &mut M: Monitor for the closure * Fix documentation of Rand::below (#747) * Netopenbsd build fix (#746) * core affinity netbsd implementation. * openbsd build fix * Fix autotokens doc (#751) * fix * remove wrong doc * Simplification for netbsd-specific code (#750) the cpuset api is already present in libc... * Add test case minimising stage (tmin) (#735) * add test case minimising stage * general purpose minimiser impl, with fuzzer example * reorganise, document, and other cleanup * correct python API return value * correct some docs * nit: versioning in fuzzers * ise -> ize * Implement a corpus minimiser (cmin) (#739) * initial try * correct case where cull attempts to fetch non-existent corpus entries * various on_remove, on_replace implementations * ise -> ize (consistency), use TestcaseScore instead of rolling our own * oops, feature gate * documentation! * link c++ * doc-nit: correction in opt explanation don't write documentation at 0300 * better linking * Skippable stage, generator wrapper for Grimoire (#748) * Skippable stage, generator wrapper for Grimoire * more fancy wrapper * MapFeedback: Adding support for with_name() (#752) * Adding support for with_name() * Adding with_name() function description * dragonflybsd build fix for core affinity. (#753) supporting most of linux sched api here. * CI for FreeBSD (#754) * CI for FreeBSD * rustup -y? * fixed path, switched to clippy * bsd don't source * added llvm * clippy * more yml * ? * testing ci * llvm? * llvm?? * more llvm, more tests * fixed testcase' * mem limits * more sudo * reenable all the CI * Fixes for new Clippy (#755) * New Clippy fixes for QEMU (#757) * Core affinity for FreeBSD pinning task to the wanted cpu (#756) * Do not zero-init struct in QEMU (#758) * New Clippy fixes for QEMU * no need to 0-initialize mem * clippy * Add doc for libafl_nyx (#759) Co-authored-by: syheliel <syheliel@gmail.com> * Adjust NyxExecutor trait bound to HasTargetBytes from HasBytesVec (#760) * adjust NyxExecutor trait bound to HasTargetBytes from HasBytesVec * oops actually use HasTargetBytes instead * libafl_frida: ASan hook adding Apple's memset_pattern* api. (#761) * Fix cargo doc on windows (#762) * add doc cfg * fix nostd docs * ignore CommandConfigurator doc test execution on non-unix platform * add cargo doc step pipeline on windows platform * Enable memset_patter ASan hooks for Apple on libafl_frida (#763) * Fix forkserver options (#771) * Stability improve (#773) * initial * add * fmt & fix * dbg remove * clp * clp * more * clippy * del * fix * remove unused * fix * doc * Fix doc (#780) * Add track_stability option to CalibrationStage (#781) * add * Update gramatron.rs * Update emu.rs * try * clp * Dump registers on freebsd x86_64 (#779) * Illumos support (#775) implementing core affinity too. * Reduce clang warnings for version output in libafl_cc. (#778) * Extend gramatron recursive mutator (#783) * Dump registers on NetBSD amd64 (#786) * Add support for ARMBE8 (#768) * Changes to build QEMU out-of-tree so that we don't need to clone the repo for each feature combination we build * Add be support to libafl_qemu * More config tweaks Co-authored-by: Your Name <you@example.com> * [AFLplusplus/LibAFL] dump registers on OpenBSD amd64 (PR #787) * dump registers on openbsd * write_crash implementations * Windows gdiplus (#789) * Initial steps * Harness code cleanup * don't panic on linux in order not to break the CI * formatting once again * restored cfg unix to unbreak linux build * Remove clang download from windows CI (#791) * Attempt to remove clang 12 setup * frida_gdiplus added to CI * Gdiplus comments (#792) * Attempt to remove clang 12 setup * frida_gdiplus added to CI * Redundancy note * formatting again :\ * mistake of directory name * Fix len miscalculation in grimoire string replace (#794) * Fix len miscalculation in grimoire string replace * ok Rust i was writing JS these days Co-authored-by: Andrea Fioraldi <andrea.fioraldi@trellix.com> * Fix doc typos (#796) * Fix CI (#798) * bump (#799) * Support for write_crash on netbsd (#788) * Support for bolts::cpu::read_time_counter on arm64 (#790) * Add ability to use virtual dispatch to StagesTuple (#801) * Add ability to use virtual dispatch to stagesTuple * Fix lint * Adding CPSR register for arm qemu (#800) * trying to add in observer * writing test * got up to running with instrumentation but i still need to get the map * fixing fuzzer code * adding tinyinst fuzzer * adding ffi to store all the map data into vec. * adding some new things * adding somewhat state of how i would like it should work * fixing some things * alot of false positives. * fixing before adding args * updated to use FileInput! * adding build script to pull tinyinst * fixing git issue * writing instruction to run how to run tinyinst fuzzer Co-authored-by: Dongjia Zhang <tokazerkje@outlook.com> Co-authored-by: Dominik Maier <dmnk@google.com> Co-authored-by: Phan Thanh Duy <phanthanhduypr@gmail.com> Co-authored-by: Nicholas Lang <97475577+nicklangsysdig@users.noreply.github.com> Co-authored-by: David CARLIER <devnexen@gmail.com> Co-authored-by: syheliel <45957390+syheliel@users.noreply.github.com> Co-authored-by: syheliel <syheliel@gmail.com> Co-authored-by: Aiden Hall <AidenRHall@users.noreply.github.com> Co-authored-by: Sönke <eknoes@users.noreply.github.com> Co-authored-by: Sirui Mu <msrlancern@gmail.com> Co-authored-by: Addison Crump <me@addisoncrump.info> Co-authored-by: Patrick Gersch <gersch.patrick@gmail.com> Co-authored-by: Teddy Heinen <teddy@heinen.dev> Co-authored-by: Vincent <space_white@yahoo.com> Co-authored-by: Andrea Fioraldi <andreafioraldi@gmail.com> Co-authored-by: WorksButNotTested <62701594+WorksButNotTested@users.noreply.github.com> Co-authored-by: Your Name <you@example.com> Co-authored-by: expend20 <36543551+expend20@users.noreply.github.com> Co-authored-by: Andrea Fioraldi <andrea.fioraldi@trellix.com> Co-authored-by: Ben Davis <ben@thebendavis.net> Co-authored-by: radl97 <radl97@users.noreply.github.com>

* step1 for tinyinst * step2: minimal executor * updated libafl * Tinyinst Update (#853) * Mac OS Autotokens (#723) * mac_tokens * more * win fix * fmt * fmt c * Use nightly fmt (#728) * Fix compilation for aarch64 qemu (#731) Typo lead to fail to compile for arm64 * Simd Fix (#729) * simd fix * fmt * Fixing readme & docs (#730) * fix * fix * add * add * fmt * 0.8.1 (#732) * New Pass Manager Arguments (#724) * new pm arguments * enable abgeana's code * Fix tui with 1 client (#734) * unbreak tui with 1 client * clippy * Add core affinity support for FreeBSD (#736) * NYX Executor (GSoC '22) (#693) * Add ccache * Update codecov.yml * Add libnyx * Fix * Add nyx build script * Fix build.sh && init executor.rs * Fix commit * Fix code * initialize `exector.rs` * refine API in `nyx_bridge.rs` * initialze `run_target` * add `test_nyxhelper` * initize `test_executor` * remove `nyx_beidge.rs` * make `test_executor` compile * Improve test * refine code * update version * fix docker * fix docker * Fix clippy * Fix build * fix build && add `set_timeout` * Fix and refine CI * fix CI * Fix CI * Add platform restrict * cargo fmt * add parallel mode * add example `nyx_libxml2_parallel` * fix fuzzer example * fix CI * add README * fix CI * fix CI * fix CI * remove unwrap and NyxResult * code format fix * add libnyx's rev * fix format * change Duration format && Fix CI * caego fmt * fix CI * fix CI * Add doc * test CI * Update test_all_fuzzers.sh * Update test_all_fuzzers.sh * Update test_all_fuzzers.sh * add cache for apt and cargo-install * Update build_and_test.yml * Update build_and_test.yml * tmp test CI * fix CI * remove debug cmd * remove test * code refine * code refine * code refine * code refine * add Makefile * fix example doc for nyx * add `NyxHelper::new_with_initial_timeout` * fix `NyxHelper::new` * fix curl parameter * code refine * add check for setup script * use afl-clang-fast in nyx * fix logic * fix makefile * fix CI * Update build_and_test.yml * Update build_and_test.yml * remove debug cmd Co-authored-by: syheliel <syheliel@gmail.com> Co-authored-by: Dominik Maier <dmnk@google.com> * Fix spelling error (#745) * OSX force_load option (#743) * Update clang.rs * fmt * Add continous JSON Logging monitor (#738) * Add simple JSON Monitor * Add documentation * Log global state * Fix formatting * Save state depending on closure outcome, have file opened all the time * Make OnDiskJSONMonitor cloneable * Switch to FnMut to allow stateful closures * Use &mut M: Monitor for the closure * Fix documentation of Rand::below (#747) * Netopenbsd build fix (#746) * core affinity netbsd implementation. * openbsd build fix * Fix autotokens doc (#751) * fix * remove wrong doc * Simplification for netbsd-specific code (#750) the cpuset api is already present in libc... * Add test case minimising stage (tmin) (#735) * add test case minimising stage * general purpose minimiser impl, with fuzzer example * reorganise, document, and other cleanup * correct python API return value * correct some docs * nit: versioning in fuzzers * ise -> ize * Implement a corpus minimiser (cmin) (#739) * initial try * correct case where cull attempts to fetch non-existent corpus entries * various on_remove, on_replace implementations * ise -> ize (consistency), use TestcaseScore instead of rolling our own * oops, feature gate * documentation! * link c++ * doc-nit: correction in opt explanation don't write documentation at 0300 * better linking * Skippable stage, generator wrapper for Grimoire (#748) * Skippable stage, generator wrapper for Grimoire * more fancy wrapper * MapFeedback: Adding support for with_name() (#752) * Adding support for with_name() * Adding with_name() function description * dragonflybsd build fix for core affinity. (#753) supporting most of linux sched api here. * CI for FreeBSD (#754) * CI for FreeBSD * rustup -y? * fixed path, switched to clippy * bsd don't source * added llvm * clippy * more yml * ? * testing ci * llvm? * llvm?? * more llvm, more tests * fixed testcase' * mem limits * more sudo * reenable all the CI * Fixes for new Clippy (#755) * New Clippy fixes for QEMU (#757) * Core affinity for FreeBSD pinning task to the wanted cpu (#756) * Do not zero-init struct in QEMU (#758) * New Clippy fixes for QEMU * no need to 0-initialize mem * clippy * Add doc for libafl_nyx (#759) Co-authored-by: syheliel <syheliel@gmail.com> * Adjust NyxExecutor trait bound to HasTargetBytes from HasBytesVec (#760) * adjust NyxExecutor trait bound to HasTargetBytes from HasBytesVec * oops actually use HasTargetBytes instead * libafl_frida: ASan hook adding Apple's memset_pattern* api. (#761) * Fix cargo doc on windows (#762) * add doc cfg * fix nostd docs * ignore CommandConfigurator doc test execution on non-unix platform * add cargo doc step pipeline on windows platform * Enable memset_patter ASan hooks for Apple on libafl_frida (#763) * Fix forkserver options (#771) * Stability improve (#773) * initial * add * fmt & fix * dbg remove * clp * clp * more * clippy * del * fix * remove unused * fix * doc * Fix doc (#780) * Add track_stability option to CalibrationStage (#781) * add * Update gramatron.rs * Update emu.rs * try * clp * Dump registers on freebsd x86_64 (#779) * Illumos support (#775) implementing core affinity too. * Reduce clang warnings for version output in libafl_cc. (#778) * Extend gramatron recursive mutator (#783) * Dump registers on NetBSD amd64 (#786) * Add support for ARMBE8 (#768) * Changes to build QEMU out-of-tree so that we don't need to clone the repo for each feature combination we build * Add be support to libafl_qemu * More config tweaks Co-authored-by: Your Name <you@example.com> * [AFLplusplus/LibAFL] dump registers on OpenBSD amd64 (PR #787) * dump registers on openbsd * write_crash implementations * Windows gdiplus (#789) * Initial steps * Harness code cleanup * don't panic on linux in order not to break the CI * formatting once again * restored cfg unix to unbreak linux build * Remove clang download from windows CI (#791) * Attempt to remove clang 12 setup * frida_gdiplus added to CI * Gdiplus comments (#792) * Attempt to remove clang 12 setup * frida_gdiplus added to CI * Redundancy note * formatting again :\ * mistake of directory name * Fix len miscalculation in grimoire string replace (#794) * Fix len miscalculation in grimoire string replace * ok Rust i was writing JS these days Co-authored-by: Andrea Fioraldi <andrea.fioraldi@trellix.com> * Fix doc typos (#796) * Fix CI (#798) * bump (#799) * Support for write_crash on netbsd (#788) * Support for bolts::cpu::read_time_counter on arm64 (#790) * Add ability to use virtual dispatch to StagesTuple (#801) * Add ability to use virtual dispatch to stagesTuple * Fix lint * Adding CPSR register for arm qemu (#800) * trying to add in observer * writing test * got up to running with instrumentation but i still need to get the map * fixing fuzzer code * adding tinyinst fuzzer * adding ffi to store all the map data into vec. * adding some new things * adding somewhat state of how i would like it should work * fixing some things * alot of false positives. * fixing before adding args * updated to use FileInput! * adding build script to pull tinyinst * fixing git issue * writing instruction to run how to run tinyinst fuzzer Co-authored-by: Dongjia Zhang <tokazerkje@outlook.com> Co-authored-by: Dominik Maier <dmnk@google.com> Co-authored-by: Phan Thanh Duy <phanthanhduypr@gmail.com> Co-authored-by: Nicholas Lang <97475577+nicklangsysdig@users.noreply.github.com> Co-authored-by: David CARLIER <devnexen@gmail.com> Co-authored-by: syheliel <45957390+syheliel@users.noreply.github.com> Co-authored-by: syheliel <syheliel@gmail.com> Co-authored-by: Aiden Hall <AidenRHall@users.noreply.github.com> Co-authored-by: Sönke <eknoes@users.noreply.github.com> Co-authored-by: Sirui Mu <msrlancern@gmail.com> Co-authored-by: Addison Crump <me@addisoncrump.info> Co-authored-by: Patrick Gersch <gersch.patrick@gmail.com> Co-authored-by: Teddy Heinen <teddy@heinen.dev> Co-authored-by: Vincent <space_white@yahoo.com> Co-authored-by: Andrea Fioraldi <andreafioraldi@gmail.com> Co-authored-by: WorksButNotTested <62701594+WorksButNotTested@users.noreply.github.com> Co-authored-by: Your Name <you@example.com> Co-authored-by: expend20 <36543551+expend20@users.noreply.github.com> Co-authored-by: Andrea Fioraldi <andrea.fioraldi@trellix.com> Co-authored-by: Ben Davis <ben@thebendavis.net> Co-authored-by: radl97 <radl97@users.noreply.github.com> * fix * fmt * Submodule * Submodule? * Tinyinst Update V2 (#905) * updated to lastest libafl * going to replace tinyinst to more like jackalope with tinyinstrumentation * fixing clippy * keep working on cpp ffi. sad * updating litecov to tinyinst. also start making our own litecov * revert to map instead of list. not sure why its not working * making fuzzer listobserver * working with listobserver!: * cleaning up * adding cargo make run * updating cargo for tinyinst * updating readme * readme, clippy * fmt * fmt * fix * fix * docker * fix * fmt Co-authored-by: Dominik Maier <dmnk@google.com> Co-authored-by: biazo <eric.l.biazo@gmail.com> Co-authored-by: Phan Thanh Duy <phanthanhduypr@gmail.com> Co-authored-by: Nicholas Lang <97475577+nicklangsysdig@users.noreply.github.com> Co-authored-by: David CARLIER <devnexen@gmail.com> Co-authored-by: syheliel <45957390+syheliel@users.noreply.github.com> Co-authored-by: syheliel <syheliel@gmail.com> Co-authored-by: Aiden Hall <AidenRHall@users.noreply.github.com> Co-authored-by: Sönke <eknoes@users.noreply.github.com> Co-authored-by: Sirui Mu <msrlancern@gmail.com> Co-authored-by: Addison Crump <me@addisoncrump.info> Co-authored-by: Patrick Gersch <gersch.patrick@gmail.com> Co-authored-by: Teddy Heinen <teddy@heinen.dev> Co-authored-by: Vincent <space_white@yahoo.com> Co-authored-by: Andrea Fioraldi <andreafioraldi@gmail.com> Co-authored-by: WorksButNotTested <62701594+WorksButNotTested@users.noreply.github.com> Co-authored-by: Your Name <you@example.com> Co-authored-by: expend20 <36543551+expend20@users.noreply.github.com> Co-authored-by: Andrea Fioraldi <andrea.fioraldi@trellix.com> Co-authored-by: Ben Davis <ben@thebendavis.net> Co-authored-by: radl97 <radl97@users.noreply.github.com> Co-authored-by: Dominik Maier <domenukk@gmail.com>

Add ability to use virtual dispatch to stagesTuple

0f14c1e

Fix lint

e5e73a0

domenukk merged commit 30f143c into AFLplusplus:main Sep 29, 2022

khang06 pushed a commit to khang06/LibAFL that referenced this pull request Oct 11, 2022

Add ability to use virtual dispatch to StagesTuple (AFLplusplus#801)

a0ed52a

* Add ability to use virtual dispatch to stagesTuple * Fix lint

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add ability to use virtual dispatch to stagesTuple#801

Add ability to use virtual dispatch to stagesTuple#801
domenukk merged 2 commits intoAFLplusplus:mainfrom
radl97:patch-1

radl97 commented Sep 26, 2022

Uh oh!

domenukk commented Sep 26, 2022 •

edited

Loading

Uh oh!

radl97 commented Sep 26, 2022

Uh oh!

domenukk commented Sep 29, 2022

Uh oh!

andreafioraldi commented Sep 29, 2022

Uh oh!

andreafioraldi commented Sep 29, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

radl97 commented Sep 26, 2022

Uh oh!

domenukk commented Sep 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

radl97 commented Sep 26, 2022

Uh oh!

domenukk commented Sep 29, 2022

Uh oh!

andreafioraldi commented Sep 29, 2022

Uh oh!

andreafioraldi commented Sep 29, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

domenukk commented Sep 26, 2022 •

edited

Loading