Skip to content

Commit

Permalink
More on structure.
Browse files Browse the repository at this point in the history
  • Loading branch information
@stevana
stevana committed Sep 24, 2024
1 parent ec5430c commit e4ebd70
Show file tree
Hide file tree
Showing 2 changed files with 63 additions and 37 deletions.
47 changes: 27 additions & 20 deletions README-unprocessed.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,7 @@ thinking](https://lcamtuf.coredump.cx/afl/historical_notes.txt) about how
fuzzing can be combined with [evolutionary
algorithms](https://en.wikipedia.org/wiki/Evolutionary_algorithm).

the idea
being that
XXX: The idea being that...

* AFL (2013),

Expand Down Expand Up @@ -77,45 +76,49 @@ etc.

* The idea of combining coverage-guidance and PBT

* Now let's have a look at what has happend since Dan wrote his post.
* Now let's have a look at what has happend since Dan's post.

* First off, at some point he added an update to his post where he explicitly mentiones:

+ Go-fuzz?
- https://adalogics.com/blog/structure-aware-go-fuzzing-complex-types

+ Hypothesis
- Has notion of coverage: https://hypothesis.readthedocs.io/en/latest/details.html#hypothesis.event)
- But coverage-guided testing was [removed](https://github.com/HypothesisWorks/hypothesis/pull/1564/commits/dcbea9148be3446392bc3af8892d49f3cc74fbe3)

* When you search for coverage guided property-based testing

* [FuzzChick](https://dl.acm.org/doi/10.1145/3360607)? Not released, lives in
* [FuzzChick](https://dl.acm.org/doi/10.1145/3360607) (2019). Not released, lives in
an [unmaintained
branch](https://github.com/QuickChick/QuickChick/compare/master...FuzzChick)
that [doesn't compile](https://github.com/QuickChick/QuickChick/issues/277)?
- coverage info is [same as in AFL](https://youtu.be/RR6c_fiMfJQ?t=2226)

* FuzzChick, related work mentions:

* [JQF + Zest: Coverage-guided semantic fuzzing for
Java](https://github.com/rohanpadhye/jqf)?

* [Crowbar](https://github.com/stedolan/crowbar)
- [extended abstract from OCaml workshop](https://github.com/ocaml/ocaml.org-media/blob/086fc25105cbccb188c28ec74126d72962921ff8/meetings/ocaml/2017/extended-abstract__2017__stephen-dolan_mindy-preston__testing-with-crowbar.pdf) (2017)
- Uses fuzzing indirectly to generate the data?
* [libfuzzer](https://llvm.org/docs/LibFuzzer.html) and it's successor
[FuzzTest](https://github.com/google/fuzztest) ("It is a first-of-its-kind
tool that bridges the gap between fuzzing and property-based testing") (2022?)

* [honggfuzz](https://github.com/google/honggfuzz)

* [Crowbar](https://github.com/stedolan/crowbar)
- open PR to add it to cargo fuzz: https://github.com/rust-fuzz/book/pull/14
* [Structure-aware fuzzing using libfuzzer-sys in
Rust](https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-fuzzing.html)

* [MUTAGEN: Reliable Coverage-Guided, Property-Based Testing using Exhaustive
Mutations](https://www.mista.me/assets/pdf/icst23-preprint.pdf) (2023)


* Shae "shapr" Erisson's post [*Run property tests until coverage stops
increasing*](https://shapr.github.io/posts/2023-07-30-goldilocks-property-tests.html)
(2023) and [trynocular](https://github.com/shapr/trynocular) library.

* [JQF + Zest: Coverage-guided semantic fuzzing for
Java](https://github.com/rohanpadhye/jqf)?

* [Structure-aware fuzzing using libfuzzer-sys in
Rust](https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-fuzzing.html)
- https://github.com/OctopiChalmers/mutagen/
- Uses GHC
[plugin](https://github.com/OctopiChalmers/mutagen/blob/main/src/Test/Mutagen/Tracer/Plugin.hs)
to annotate source code with coverage information of: function clauses,
case statements, multi-way ifs, and each branch of if-then-else
expressions

## Examples and the main idea of coverage-guidance

Expand Down Expand Up @@ -156,6 +159,8 @@ reduce the problem to only need $O(2^8 + 2^8 + 2^8 + 2^8) = O(2^8 \cdot 4) =
O(2^{10}) = 1024$ tries. With other words coverage-guidence turns an
exponential problem into a polynomial problem!

## Prototype implementation

Great, but where do we get this coverage information from?

AFL and `go-fuzz` both get it from the compiler.
Expand Down Expand Up @@ -196,11 +201,9 @@ very first version[^2]!
So the question is: can we implement coverage-guided property-based testing
using the internal notion of coverage that property-based testing already has?

## Prototype implementation

* QuickCheck as defined in the appendix of the original
[paper](https://dl.acm.org/doi/10.1145/351240.351266) (ICFP, 2000)
- Extended with shrinking
- Extended monadic properties

* Edsko de Vries'
Expand Down Expand Up @@ -272,6 +275,10 @@ The full source code is available
* [AFL "whitepaper"](https://lcamtuf.coredump.cx/afl/technical_details.txt)
* [AFL mutation
heuristics](https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-works.html)
* Shae "shapr" Erisson's post [*Run property tests until coverage stops
increasing*](https://shapr.github.io/posts/2023-07-30-goldilocks-property-tests.html) (2023)
(2023) and [trynocular](https://github.com/shapr/trynocular) library.
- This only uses coverage as a stopping condition, not to actually drive the generation...



Expand Down
53 changes: 36 additions & 17 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ thinking](https://lcamtuf.coredump.cx/afl/historical_notes.txt) about
how fuzzing can be combined with [evolutionary
algorithms](https://en.wikipedia.org/wiki/Evolutionary_algorithm).

the idea being that
XXX: The idea being that...

- AFL (2013),

Expand All @@ -75,13 +75,15 @@ also arrays of ints, etc.

- The idea of combining coverage-guidance and PBT

- Now let's have a look at what has happend since Dan wrote his post.
- Now let's have a look at what has happend since Dan's post.

- First off, at some point he added an update to his post where he
explicitly mentiones:

- Go-fuzz?

- <https://adalogics.com/blog/structure-aware-go-fuzzing-complex-types>

- Hypothesis

- Has notion of coverage:
Expand All @@ -91,37 +93,50 @@ also arrays of ints, etc.

- When you search for coverage guided property-based testing

- [FuzzChick](https://dl.acm.org/doi/10.1145/3360607)? Not released,
lives in an [unmaintained
- [FuzzChick](https://dl.acm.org/doi/10.1145/3360607) (2019). Not
released, lives in an [unmaintained
branch](https://github.com/QuickChick/QuickChick/compare/master...FuzzChick)
that [doesn't
compile](https://github.com/QuickChick/QuickChick/issues/277)?

- coverage info is [same as in
AFL](https://youtu.be/RR6c_fiMfJQ?t=2226)

- FuzzChick, related work mentions:

- [JQF + Zest: Coverage-guided semantic fuzzing for
Java](https://github.com/rohanpadhye/jqf)?

- [Crowbar](https://github.com/stedolan/crowbar)

- [extended abstract from OCaml
workshop](https://github.com/ocaml/ocaml.org-media/blob/086fc25105cbccb188c28ec74126d72962921ff8/meetings/ocaml/2017/extended-abstract__2017__stephen-dolan_mindy-preston__testing-with-crowbar.pdf)
(2017)
- Uses fuzzing indirectly to generate the data?

- [libfuzzer](https://llvm.org/docs/LibFuzzer.html) and it's successor
[FuzzTest](https://github.com/google/fuzztest) ("It is a
first-of-its-kind tool that bridges the gap between fuzzing and
property-based testing") (2022?)

- [honggfuzz](https://github.com/google/honggfuzz)

- [Crowbar](https://github.com/stedolan/crowbar)
- open PR to add it to cargo fuzz:
<https://github.com/rust-fuzz/book/pull/14>

- [Structure-aware fuzzing using libfuzzer-sys in
Rust](https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-fuzzing.html)

- [MUTAGEN: Reliable Coverage-Guided, Property-Based Testing using
Exhaustive
Mutations](https://www.mista.me/assets/pdf/icst23-preprint.pdf) (2023)

- Shae "shapr" Erisson's post [*Run property tests until coverage stops
increasing*](https://shapr.github.io/posts/2023-07-30-goldilocks-property-tests.html) (2023)
and [trynocular](https://github.com/shapr/trynocular) library.

- [JQF + Zest: Coverage-guided semantic fuzzing for
Java](https://github.com/rohanpadhye/jqf)?

- [Structure-aware fuzzing using libfuzzer-sys in
Rust](https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-fuzzing.html)
- <https://github.com/OctopiChalmers/mutagen/>
- Uses GHC
[plugin](https://github.com/OctopiChalmers/mutagen/blob/main/src/Test/Mutagen/Tracer/Plugin.hs)
to annotate source code with coverage information of: function
clauses, case statements, multi-way ifs, and each branch of
if-then-else expressions

## Examples and the main idea of coverage-guidance

Expand Down Expand Up @@ -163,6 +178,8 @@ $O(2^8 + 2^8 + 2^8 + 2^8) = O(2^8 \cdot 4) =
O(2^{10}) = 1024$ tries. With other words coverage-guidence turns an
exponential problem into a polynomial problem!

## Prototype implementation

Great, but where do we get this coverage information from?

AFL and `go-fuzz` both get it from the compiler.
Expand Down Expand Up @@ -208,12 +225,9 @@ So the question is: can we implement coverage-guided property-based
testing using the internal notion of coverage that property-based
testing already has?

## Prototype implementation

- QuickCheck as defined in the appendix of the original
[paper](https://dl.acm.org/doi/10.1145/351240.351266) (ICFP, 2000)

- Extended with shrinking
- Extended monadic properties

- Edsko de Vries'
Expand Down Expand Up @@ -382,6 +396,11 @@ The full source code is available
"whitepaper"](https://lcamtuf.coredump.cx/afl/technical_details.txt)
- [AFL mutation
heuristics](https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-works.html)
- Shae "shapr" Erisson's post [*Run property tests until coverage stops
increasing*](https://shapr.github.io/posts/2023-07-30-goldilocks-property-tests.html) (2023) (2023)
and [trynocular](https://github.com/shapr/trynocular) library.
- This only uses coverage as a stopping condition, not to actually
drive the generation...

[^1]: Here's Dan's example in full:

Expand Down

0 comments on commit e4ebd70

Please sign in to comment.