Continuous Fuzzing Integration with Fuzzit by yevgenypats · Pull Request #7509 · envoyproxy/envoy

yevgenypats · 2019-07-09T18:16:16Z

Please do not merge yet. This is work in progress.
The contribution was discuss in the mailing-list with @htuch , @mattklein123 .

Description:
This will introduce another platform (apart from oss) fuzz that
will run the long-running fuzzers as well as will introduce
"sanity fuzzers" that will run the accumlated corpus and crashes
on every Pull-Request to detect bugs early-on in the development
cycle.

Risk Level:
Low - as this will introduce only another step in CircleCI where the fuzzers will be uploaded to Fuzzit and the heavy lifting will be there.

Testing:
No code is added just a CI code in Circle

Docs Changes:
Will be updated later.

Release Notes: None
[Optional Fixes #Issue] None
[Optional Deprecated:] None

htuch · 2019-07-09T19:28:43Z

ci/run_fuzzit.sh

One key issue here is build performance; if this takes a significant amount of time (which anecdotally it does from local runs), e.g. > 1 hour, it will become a CI bottleneck. Many of our CI jobs now benefit from Bazel caching and (in the future) RBE enablement, this one is going to be inherently stuck without this, since the oss-fuzz image doesn't have any plumbing for this. CC @lizan

@htuch This is exactly what I'm trying to do just wanted to see if it works with the current oss-fuzz images. Anyway I was able to compile the targets with bazel for example

using bazel build //test/server:server_fuzz_test the build completes successfully but it seems that it doesn't link against libFuzzer. what would be the right command to link it against libfuzzer (i.e -fsanitize=fuzzer) as I don't wont to use the oss docker both to use the caching and also not to use the nested docker in Circle.

Yeah, I'm afraid you're going to have to do some gnarly Bazel hacking to get it to link against libfuzzer under Bazel. We would be super appreciative of this work, but it will involve diving into https://github.com/envoyproxy/envoy/blob/master/bazel/envoy_test.bzl#L66 and figuring out how to cleanly offer both the corpus run (using the existing test driver) and a way to on the Bazel CLI configure a link against libfuzzer (and somehow getting libfuzzer included in the Envoy build).

Thanks! I'll find the way to do that. I just wanted to make sure that I'm not doing double work and it's not implemented yet.

htuch · 2019-07-09T19:29:14Z

ci/run_fuzzit.sh

Are we doing nested Docker here? I.e. running oss-fuzz Docker build inside a CircleCI Docker env?

I'm exactly working on this to make it work not inside the oss-fuzz docker.

yevgenypats · 2019-07-16T12:37:13Z

@htuch looks like I get some no-credits error at CircleCI? any particular reason for this to happen? I can test my changes.

htuch · 2019-07-16T13:52:31Z

@yevgenypats this is a general Envoy CI issue, we are working to resolve.

htuch · 2019-07-16T22:23:50Z

/retest

repokitteh-read-only · 2019-07-16T22:23:54Z

🔨 rebuilding ci/circleci: Build Error (failed build)

🐱

Caused by: a #7509 (comment) was created by @htuch.

see: more, trace.

yevgenypats · 2019-07-17T05:14:43Z

@htuch thx. just an update. look like it works now. I'll update when the PR will be ready for review. should be soon. cheers.

yevgenypats · 2019-09-17T09:14:44Z

@htuch @lizan sorry for the delay. I was a bit overloaded with our release for Java Support:)

As far as I can tell there are some problems integrating with azure pipelines. The current one that I'm strangling with is that it looks like the output from the build inside the docker are not available on the host. i.e I can't access the fuzzing binaries...

+ ls -la /home/vsts/work/1/a/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/common/access_log/access_log_formatter_fuzz_test_with_libfuzzer.runfiles/envoy/test/common/access_log/access_log_formatter_fuzz_test_with_libfuzzer
lrwxrwxrwx 1 vsts docker 167 Sep 17 07:53 /home/vsts/work/1/a/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/common/access_log/access_log_formatter_fuzz_test_with_libfuzzer.runfiles/envoy/test/common/access_log/access_log_formatter_fuzz_test_with_libfuzzer -> /build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/common/access_log/access_log_formatter_fuzz_test_with_libfuzzer
++ realpath /home/vsts/work/1/a/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/common/access_log/access_log_formatter_fuzz_test_with_libfuzzer.runfiles/envoy/test/common/access_log/access_log_formatter_fuzz_test_with_libfuzzer
realpath: /home/vsts/work/1/a/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/common/access_log/access_log_formatter_fuzz_test_with_libfuzzer.runfiles/envoy/test/common/access_log/access_log_formatter_fuzz_test_with_libfuzzer: No such file or directory

Is there any easy way to solve this or should we go back to Circle where it's already worked?

lizan · 2019-09-17T20:27:50Z

@yevgenypats just run everything in docker like it did in Circle? i.e. ci/run_envoy_docker.sh 'ci/do_ci.sh bazel.fuzz && ci/run_fuzzit.sh local-regression' as the step?

yevgenypats · 2019-09-17T20:30:37Z

I can try that but regression also run the test in a docker so the tests will be reproducible. Will I have access from inside the docker?

…

On Tue, Sep 17, 2019, 11:28 PM Lizan Zhou ***@***.***> wrote: @yevgenypats <https://github.com/yevgenypats> just run everything in docker like it did in Circle? i.e. ci/run_envoy_docker.sh 'ci/do_ci.sh bazel.fuzz && ci/run_fuzzit.sh local-regression' as the step? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7509?email_source=notifications&email_token=AD52CDUSDXDEATHNQIPVJT3QKE4WHA5CNFSM4H7ICZEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD65ZSSI#issuecomment-532388169>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD52CDWIRX6R7JIJMUWEBF3QKE4WHANCNFSM4H7ICZEA> .

lizan · 2019-09-17T21:59:18Z

it should have same access, lmk if you hit any issue.

yevgenypats · 2019-09-18T10:44:02Z

@lizan I think something was changed. what would be the right (automated) way to find the path to the fuzzing binary. I get this https://dev.azure.com/cncf/envoy/_build/results?buildId=11190&view=logs&jobId=d205ed5c-f2f1-5f43-d907-fe4cb51a95a5 now.

lizan · 2019-09-18T11:09:41Z

@yevgenypats Oh yeah bazel test no longer generate binary with RBE, so you need to modify ci/do_ci.sh to do bazel build in addition to bazel test (in this line: https://github.com/envoyproxy/envoy/blob/master/ci/do_ci.sh#L261), maybe just add a fuzzit CI_TARGET in the do_ci.sh script?

yevgenypats · 2019-09-18T12:07:12Z

so I've add a CI_TARGET and looks like it progressing but I don't have acces to docker daemon. https://dev.azure.com/cncf/envoy/_build/results?buildId=11205&view=logs

where will be the right way to add access to the docker daemon?

yevgenypats · 2019-09-18T19:36:19Z

@lizan good news, I think this almost works. Now I want to do the following I want to run regression tests on every PR/commit and fuzzing only on pushes to master (or daily cron from master). what would be the right way to configure this in the AzurePipelines

lizan

Great job! Thanks

lizan · 2019-09-18T19:45:50Z

ci/run_envoy_docker.sh

nice catch, thanks

lizan · 2019-09-18T19:47:12Z

.azure-pipelines/linux.yml

Now this job can be rolled in to the parameterized job above?

Oh yes, that would make sense for the regression but how do I make the fuzzing only run on pushes to master / daily cron (by fuzzing I mean building the fuzzers and uploading them to fuzzit so they can run asynchronously and I want this to happen only for the master branch and not every PR /branch etc...)

ah, then probably not rolling into is fine. You can add a conditions field to the job, condition: ne(variables['Build.Reason'], 'PullRequest'), to exclude PRs? See https://docs.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml for details and tune it to what you want to do.

ok I'll try that. I'll try to roll regression into that parametrised and the fuzzing into it's own step with condition

that's fine too, thanks!

yevgenypats · 2019-09-19T09:31:06Z

@lizan good news! Looks like we are on track. Now you or @htuch need to sign up at https://app.fuzzit.dev so I can add you to envoy account so you can take over. Also once I add you to envoy account you need to copy FUZZIT_API_KEY to AZP environment variables.

yevgenypats · 2019-09-20T09:10:24Z

@lizan you should now have access to https://app.fuzzit.dev/orgs/envoyproxy/settings

lizan · 2019-09-20T09:56:16Z

ci/do_ci.sh

no need --test_arg

lizan · 2019-09-20T09:56:26Z

ci/do_ci.sh

lizan · 2019-09-20T09:58:06Z

.azure-pipelines/linux.yml

are you intended to do this in every PR? This doesn't provide anything in addition to bazel.fuzz above, does it?

yeah. I think it is and this was one of the main reasons integrating fuzzit in addition to OSS Fuzz. this downloads the current corpus+fixed_crashes from Fuzzit Servers and runs the fuzzers through those test-cases which is stronger then just a run for 10 seconds with empty corpus (which is fine but that just checks if the fuzzers compiled successfully)

can we merge those two jobs into one?

lizan · 2019-09-20T09:59:51Z

.azure-pipelines/linux.yml

Add FUZZIT_API_KEY env (it can be in FUZZIT_API_KEY=$(FuzzitApiKey) before ci/do_ci.sh, or env below and add propagate it in ci/run_envoy_docker.sh.

lizan · 2019-09-20T10:26:01Z

ci/do_ci.sh

update comment as well.

lizan · 2019-09-20T10:26:59Z

.azure-pipelines/linux.yml

can we merge those two jobs into one?

yevgenypats · 2019-09-20T17:16:32Z

@lizan fixed the review:)

lizan · 2019-09-20T21:05:26Z

@htuch?

htuch

/wait

htuch · 2019-09-23T15:05:49Z

.azure-pipelines/linux.yml

Can you add some comments here explaining when this runs? Does this happen on every PR, does it block PRs, is this contributing to the PR critical path on CI? I think this PR generally looks awesome, but this needs some clarification, preferably in source comments. Thanks!

Done, as explained in the comments there are two workflows:

Fuzzing - This will run on every push/merge to master, will build the fuzzers and will upload them to Fuzzit where they will run asynchronous. This will ensure the latest version of the code is always being fuzzed and new bugs are found as new code is added.

Regression - This will run on every commit/PR and will run the fuzzers inline in the CI together with the corpus generated on Fuzzit as well as previous fixed crashes. This will ensure bugs are found BEFORE merge.

Signed-off-by: Yevgeny Pats <yp@fuzzit.dev>

yevgenypats · 2019-09-23T16:22:28Z

@htuch why this fails now ?https://circleci.com/gh/envoyproxy/envoy/279964?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

yevgenypats · 2019-09-23T16:59:53Z

/retest

repokitteh-read-only · 2019-09-23T16:59:57Z

🔨 rebuilding ci/circleci: docs (failed build)

🐱

Caused by: a #7509 (comment) was created by @yevgenypats.

see: more, trace.

yevgenypats · 2019-09-23T17:27:00Z

@htuch passing!:)

htuch

Thanks @yevgenypats. Appreciate your patience and work on iterating to find the best fit with Envoy CI and fuzzing. Also thanks @lizan for all the help with the CI aspects.

yevgenypats · 2019-09-23T21:29:17Z

@htuch @lizan sure thing, thank you! and feel free to ping me if there are any issues/changes you need.

also @lizan are you sure the FUZZIT_API_KEY is available because I see that the script didn't find it (I couldn't check it before merge as this is environment variable is not available for forked-PRs)

Also looks there are some crashes already https://app.fuzzit.dev/orgs/envoyproxy/dashboard

and of course feel free to RT https://twitter.com/fuzzitdev/status/1176246876821217283

Thanks!

htuch · 2019-09-23T21:45:28Z

@yevgenypats are these crashes limited in visibility? We would like to maintain embargo on any zero days and only have https://github.com/google/oss-fuzz/blob/master/projects/envoy/project.yaml#L3 see these.

yevgenypats · 2019-09-23T21:48:31Z

@htuch yes of course, like we discussed. currently only @lizan has access I can add you to the email notifications but you need to sign-up at https://app.fuzzit.dev so I can add you to the envoy account so you will have access to the crashes and other data.

lizan · 2019-09-23T21:54:38Z

I set the api key right now, should be fixed in future builds

yevgenypats · 2019-09-23T21:55:58Z

Great.

…

On Mon, Sep 23, 2019, 5:55 PM Lizan Zhou ***@***.***> wrote: I set it right now, should be fixed in future builds — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7509?email_source=notifications&email_token=AD52CDUYCVHBV4TWPG5AYHTQLE3LVA5CNFSM4H7ICZEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7MMOZY#issuecomment-534300519>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD52CDRJ4YHQ2IFSKL2UCEDQLE3LVANCNFSM4H7ICZEA> .

lizan · 2019-09-24T04:33:25Z

@yevgenypats now it is failing with

+ ./fuzzit create target --skip-if-exists --public-corpus envoyproxy/access-log-formatter
2019/09/23 23:20:23 Creating target...
2019/09/23 23:20:23 target can only contain lowercase characetrs, numbers and hypens

yevgenypats force-pushed the fuzzit_integration branch 4 times, most recently from 13d63ca to 206ec5f Compare July 9, 2019 18:35

htuch reviewed Jul 9, 2019

View reviewed changes

htuch added the waiting label Jul 9, 2019

alyssawilk assigned htuch Jul 9, 2019

yevgenypats force-pushed the fuzzit_integration branch from 206ec5f to 5594a38 Compare July 9, 2019 20:40

repokitteh-read-only bot removed the waiting label Jul 9, 2019

yevgenypats force-pushed the fuzzit_integration branch 3 times, most recently from e1b4af6 to 1178f97 Compare July 10, 2019 06:22

htuch added the waiting:any label Jul 12, 2019

repokitteh-read-only bot removed the waiting:any label Jul 12, 2019

htuch added the waiting label Jul 12, 2019

yevgenypats force-pushed the fuzzit_integration branch from 1178f97 to a0cb5b4 Compare July 16, 2019 12:27

repokitteh-read-only bot removed the waiting label Jul 16, 2019

yevgenypats force-pushed the fuzzit_integration branch from a0cb5b4 to b704a74 Compare July 16, 2019 12:36

yevgenypats force-pushed the fuzzit_integration branch from b704a74 to 958e2c6 Compare July 16, 2019 13:40

htuch added the waiting label Jul 16, 2019

yevgenypats force-pushed the fuzzit_integration branch from 958e2c6 to 3c3ec56 Compare July 17, 2019 05:13

repokitteh-read-only bot removed the waiting label Jul 17, 2019

yevgenypats force-pushed the fuzzit_integration branch 3 times, most recently from 59876ab to e1b1881 Compare July 17, 2019 07:56

lizan reviewed Sep 18, 2019

View reviewed changes

lizan reviewed Sep 20, 2019

View reviewed changes

lizan previously approved these changes Sep 20, 2019

View reviewed changes

htuch suggested changes Sep 23, 2019

View reviewed changes

Migrate to AZP

54dc342

Signed-off-by: Yevgeny Pats <yp@fuzzit.dev>

htuch approved these changes Sep 23, 2019

View reviewed changes

yevgenypats mentioned this pull request Sep 24, 2019

update fuzzit CLI to v2.4.57 #8342

Merged

Conversation

yevgenypats commented Jul 9, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yevgenypats commented Jul 16, 2019

Uh oh!

htuch commented Jul 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

htuch commented Jul 16, 2019

Uh oh!

repokitteh-read-only bot commented Jul 16, 2019

Uh oh!

yevgenypats commented Jul 17, 2019

Uh oh!

yevgenypats commented Sep 17, 2019

Uh oh!

lizan commented Sep 17, 2019

Uh oh!

yevgenypats commented Sep 17, 2019 via email

Uh oh!

lizan commented Sep 17, 2019

Uh oh!

yevgenypats commented Sep 18, 2019

Uh oh!

lizan commented Sep 18, 2019

Uh oh!

yevgenypats commented Sep 18, 2019

Uh oh!

yevgenypats commented Sep 18, 2019

Uh oh!

lizan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yevgenypats commented Sep 19, 2019

Uh oh!

yevgenypats commented Sep 20, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

htuch commented Jul 16, 2019 •

edited

Loading

lizan commented Sep 23, 2019 •

edited

Loading