From e4ebd70e369156b948bd96cc8d6da3b96562f2f2 Mon Sep 17 00:00:00 2001 From: Stevan Andjelkovic Date: Tue, 24 Sep 2024 10:53:48 +0200 Subject: [PATCH] More on structure. --- README-unprocessed.md | 47 ++++++++++++++++++++++---------------- README.md | 53 +++++++++++++++++++++++++++++-------------- 2 files changed, 63 insertions(+), 37 deletions(-) diff --git a/README-unprocessed.md b/README-unprocessed.md index 10a81c7..c2ce9be 100644 --- a/README-unprocessed.md +++ b/README-unprocessed.md @@ -48,8 +48,7 @@ thinking](https://lcamtuf.coredump.cx/afl/historical_notes.txt) about how fuzzing can be combined with [evolutionary algorithms](https://en.wikipedia.org/wiki/Evolutionary_algorithm). -the idea -being that +XXX: The idea being that... * AFL (2013), @@ -77,11 +76,12 @@ etc. * The idea of combining coverage-guidance and PBT -* Now let's have a look at what has happend since Dan wrote his post. +* Now let's have a look at what has happend since Dan's post. * First off, at some point he added an update to his post where he explicitly mentiones: + Go-fuzz? + - https://adalogics.com/blog/structure-aware-go-fuzzing-complex-types + Hypothesis - Has notion of coverage: https://hypothesis.readthedocs.io/en/latest/details.html#hypothesis.event) @@ -89,33 +89,36 @@ etc. * When you search for coverage guided property-based testing -* [FuzzChick](https://dl.acm.org/doi/10.1145/3360607)? Not released, lives in +* [FuzzChick](https://dl.acm.org/doi/10.1145/3360607) (2019). Not released, lives in an [unmaintained branch](https://github.com/QuickChick/QuickChick/compare/master...FuzzChick) that [doesn't compile](https://github.com/QuickChick/QuickChick/issues/277)? - coverage info is [same as in AFL](https://youtu.be/RR6c_fiMfJQ?t=2226) +* FuzzChick, related work mentions: + +* [JQF + Zest: Coverage-guided semantic fuzzing for + Java](https://github.com/rohanpadhye/jqf)? + +* [Crowbar](https://github.com/stedolan/crowbar) + - [extended abstract from OCaml workshop](https://github.com/ocaml/ocaml.org-media/blob/086fc25105cbccb188c28ec74126d72962921ff8/meetings/ocaml/2017/extended-abstract__2017__stephen-dolan_mindy-preston__testing-with-crowbar.pdf) (2017) + - Uses fuzzing indirectly to generate the data? * [libfuzzer](https://llvm.org/docs/LibFuzzer.html) and it's successor [FuzzTest](https://github.com/google/fuzztest) ("It is a first-of-its-kind tool that bridges the gap between fuzzing and property-based testing") (2022?) - * [honggfuzz](https://github.com/google/honggfuzz) - -* [Crowbar](https://github.com/stedolan/crowbar) + - open PR to add it to cargo fuzz: https://github.com/rust-fuzz/book/pull/14 +* [Structure-aware fuzzing using libfuzzer-sys in + Rust](https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-fuzzing.html) * [MUTAGEN: Reliable Coverage-Guided, Property-Based Testing using Exhaustive Mutations](https://www.mista.me/assets/pdf/icst23-preprint.pdf) (2023) - - -* Shae "shapr" Erisson's post [*Run property tests until coverage stops - increasing*](https://shapr.github.io/posts/2023-07-30-goldilocks-property-tests.html) - (2023) and [trynocular](https://github.com/shapr/trynocular) library. - -* [JQF + Zest: Coverage-guided semantic fuzzing for - Java](https://github.com/rohanpadhye/jqf)? - -* [Structure-aware fuzzing using libfuzzer-sys in - Rust](https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-fuzzing.html) + - https://github.com/OctopiChalmers/mutagen/ + - Uses GHC + [plugin](https://github.com/OctopiChalmers/mutagen/blob/main/src/Test/Mutagen/Tracer/Plugin.hs) + to annotate source code with coverage information of: function clauses, + case statements, multi-way ifs, and each branch of if-then-else + expressions ## Examples and the main idea of coverage-guidance @@ -156,6 +159,8 @@ reduce the problem to only need $O(2^8 + 2^8 + 2^8 + 2^8) = O(2^8 \cdot 4) = O(2^{10}) = 1024$ tries. With other words coverage-guidence turns an exponential problem into a polynomial problem! +## Prototype implementation + Great, but where do we get this coverage information from? AFL and `go-fuzz` both get it from the compiler. @@ -196,11 +201,9 @@ very first version[^2]! So the question is: can we implement coverage-guided property-based testing using the internal notion of coverage that property-based testing already has? -## Prototype implementation * QuickCheck as defined in the appendix of the original [paper](https://dl.acm.org/doi/10.1145/351240.351266) (ICFP, 2000) - - Extended with shrinking - Extended monadic properties * Edsko de Vries' @@ -272,6 +275,10 @@ The full source code is available * [AFL "whitepaper"](https://lcamtuf.coredump.cx/afl/technical_details.txt) * [AFL mutation heuristics](https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-works.html) +* Shae "shapr" Erisson's post [*Run property tests until coverage stops + increasing*](https://shapr.github.io/posts/2023-07-30-goldilocks-property-tests.html) (2023) + (2023) and [trynocular](https://github.com/shapr/trynocular) library. + - This only uses coverage as a stopping condition, not to actually drive the generation... diff --git a/README.md b/README.md index aec485d..6d8eb7b 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ thinking](https://lcamtuf.coredump.cx/afl/historical_notes.txt) about how fuzzing can be combined with [evolutionary algorithms](https://en.wikipedia.org/wiki/Evolutionary_algorithm). -the idea being that +XXX: The idea being that... - AFL (2013), @@ -75,13 +75,15 @@ also arrays of ints, etc. - The idea of combining coverage-guidance and PBT -- Now let's have a look at what has happend since Dan wrote his post. +- Now let's have a look at what has happend since Dan's post. - First off, at some point he added an update to his post where he explicitly mentiones: - Go-fuzz? + - + - Hypothesis - Has notion of coverage: @@ -91,8 +93,8 @@ also arrays of ints, etc. - When you search for coverage guided property-based testing -- [FuzzChick](https://dl.acm.org/doi/10.1145/3360607)? Not released, - lives in an [unmaintained +- [FuzzChick](https://dl.acm.org/doi/10.1145/3360607) (2019). Not + released, lives in an [unmaintained branch](https://github.com/QuickChick/QuickChick/compare/master...FuzzChick) that [doesn't compile](https://github.com/QuickChick/QuickChick/issues/277)? @@ -100,6 +102,18 @@ also arrays of ints, etc. - coverage info is [same as in AFL](https://youtu.be/RR6c_fiMfJQ?t=2226) +- FuzzChick, related work mentions: + +- [JQF + Zest: Coverage-guided semantic fuzzing for + Java](https://github.com/rohanpadhye/jqf)? + +- [Crowbar](https://github.com/stedolan/crowbar) + + - [extended abstract from OCaml + workshop](https://github.com/ocaml/ocaml.org-media/blob/086fc25105cbccb188c28ec74126d72962921ff8/meetings/ocaml/2017/extended-abstract__2017__stephen-dolan_mindy-preston__testing-with-crowbar.pdf) + (2017) + - Uses fuzzing indirectly to generate the data? + - [libfuzzer](https://llvm.org/docs/LibFuzzer.html) and it's successor [FuzzTest](https://github.com/google/fuzztest) ("It is a first-of-its-kind tool that bridges the gap between fuzzing and @@ -107,21 +121,22 @@ also arrays of ints, etc. - [honggfuzz](https://github.com/google/honggfuzz) -- [Crowbar](https://github.com/stedolan/crowbar) + - open PR to add it to cargo fuzz: + + +- [Structure-aware fuzzing using libfuzzer-sys in + Rust](https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-fuzzing.html) - [MUTAGEN: Reliable Coverage-Guided, Property-Based Testing using Exhaustive Mutations](https://www.mista.me/assets/pdf/icst23-preprint.pdf) (2023) -- Shae "shapr" Erisson's post [*Run property tests until coverage stops - increasing*](https://shapr.github.io/posts/2023-07-30-goldilocks-property-tests.html) (2023) - and [trynocular](https://github.com/shapr/trynocular) library. - -- [JQF + Zest: Coverage-guided semantic fuzzing for - Java](https://github.com/rohanpadhye/jqf)? - -- [Structure-aware fuzzing using libfuzzer-sys in - Rust](https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-fuzzing.html) + - + - Uses GHC + [plugin](https://github.com/OctopiChalmers/mutagen/blob/main/src/Test/Mutagen/Tracer/Plugin.hs) + to annotate source code with coverage information of: function + clauses, case statements, multi-way ifs, and each branch of + if-then-else expressions ## Examples and the main idea of coverage-guidance @@ -163,6 +178,8 @@ $O(2^8 + 2^8 + 2^8 + 2^8) = O(2^8 \cdot 4) = O(2^{10}) = 1024$ tries. With other words coverage-guidence turns an exponential problem into a polynomial problem! +## Prototype implementation + Great, but where do we get this coverage information from? AFL and `go-fuzz` both get it from the compiler. @@ -208,12 +225,9 @@ So the question is: can we implement coverage-guided property-based testing using the internal notion of coverage that property-based testing already has? -## Prototype implementation - - QuickCheck as defined in the appendix of the original [paper](https://dl.acm.org/doi/10.1145/351240.351266) (ICFP, 2000) - - Extended with shrinking - Extended monadic properties - Edsko de Vries' @@ -382,6 +396,11 @@ The full source code is available "whitepaper"](https://lcamtuf.coredump.cx/afl/technical_details.txt) - [AFL mutation heuristics](https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-works.html) +- Shae "shapr" Erisson's post [*Run property tests until coverage stops + increasing*](https://shapr.github.io/posts/2023-07-30-goldilocks-property-tests.html) (2023) (2023) + and [trynocular](https://github.com/shapr/trynocular) library. + - This only uses coverage as a stopping condition, not to actually + drive the generation... [^1]: Here's Dan's example in full: