Skip to content

Commit

Permalink
More structure and code snippets.
Browse files Browse the repository at this point in the history
  • Loading branch information
stevana committed Sep 28, 2024
1 parent 6d741e3 commit 74e8aca
Show file tree
Hide file tree
Showing 3 changed files with 205 additions and 67 deletions.
69 changes: 48 additions & 21 deletions README-unprocessed.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ testing wasn't a thing.
In this post I'll survey the coverage-guided landscape, looking at what was
there before Dan's post and what has happened since.

The short version is: imperative languages seem to be in the forefront of
The short version is: today imperative languages seem to be in the forefront of
combining coverage-guidance and property-based testing.

In an effort to try to help functional programming languages catch up, I'll
Expand Down Expand Up @@ -210,24 +210,37 @@ note](https://github.com/HypothesisWorks/hypothesis/pull/1564/commits/dcbea9148b
As far as I can tell, it hasn't been reintroduced since.

However it's possible to hook Hypothesis up to [use external
fuzzers](https://hypothesis.readthedocs.io/en/latest/details.html#use-with-external-fuzzers).

XXX: how does this work? like in Crowbar? What are the disadvantages? Why isn't
this the default?

What else has happenend since Dan's post?

* > "Note: AFL hasn't been updated for a couple of years; while it should still
> work fine, a more complex fork with a variety of improvements and additional
> features, known as AFL++, is available from other members of the community
> and is worth checking out." -- https://lcamtuf.coredump.cx/afl/
One of the first things I noticed is that AFL is no longer
[maintained](https://lcamtuf.coredump.cx/afl/):

> "Note: AFL hasn't been updated for a couple of years; while it should still
> work fine, a more complex fork with a variety of improvements and additional
> features, known as AFL++, is available from other members of the community
> and is worth checking out."
+ [AFL++](https://www.usenix.org/system/files/woot20-paper-fioraldi.pdf)
(2020) incorporates all of
[AFLFast](https://mboehme.github.io/paper/CCS16.pdf)'s [power
schedules](https://aflplus.plus/docs/power_schedules/) and adds some news
ones
+ https://github.com/mboehme/aflfast
[AFL++](https://www.usenix.org/system/files/woot20-paper-fioraldi.pdf) (2020)
- incorporates all of
[AFLFast](https://mboehme.github.io/paper/CCS16.pdf)'s [power
schedules](https://aflplus.plus/docs/power_schedules/) and adds some new
ones
- explain what power schedules are?
- https://github.com/mboehme/aflfast

* When you search for "coverage-guided property-based testing" in the academic literature

* [FuzzChick](https://dl.acm.org/doi/10.1145/3360607) (2019). Not released, lives in
an [unmaintained
* [*Coverage guided, property based
testing*](https://dl.acm.org/doi/10.1145/3360607) by Leonidas Lampropoulos,
Michael Hicks, Benjamin C. Pierce (2019)
* FuzzChick Coq/Rocq library
* Not released, lives in an [unmaintained
branch](https://github.com/QuickChick/QuickChick/compare/master...FuzzChick)
that [doesn't compile](https://github.com/QuickChick/QuickChick/issues/277)?
- coverage info is [same as in AFL](https://youtu.be/RR6c_fiMfJQ?t=2226)
Expand Down Expand Up @@ -270,6 +283,7 @@ version, based on the original property-based testing implementation.
One key question we need to answer in order to be able to implement anything
that's coverage-guided is: where do we get the coverage information from?

### Getting the coverage information

AFL and `go-fuzz` both get it from the compiler.

Expand Down Expand Up @@ -314,23 +328,26 @@ very first version[^2]!
So the question is: can we implement coverage-guided property-based testing
using the internal notion of coverage that property-based testing already has?

### The first version of QuickCheck

* QuickCheck as defined in the appendix of the original
[paper](https://dl.acm.org/doi/10.1145/351240.351266) (ICFP, 2000)
- Extended monadic properties

* Edsko de Vries'
[Mini-QuickCheck](https://www.well-typed.com/blog/2019/05/integrated-shrinking/)

``` {.haskell include=src/QuickCheckV1.hs snippet=Gen}
```

Footnote: We'll not talk about the coarbitrary, which is used to generate
functions.

``` {.haskell include=src/QuickCheckV1.hs snippet=rand}
```

``` {.haskell include=src/QuickCheckV1.hs snippet=sized}
```

``` {.haskell include=src/QuickCheckV1.hs snippet=Arbitrary}
```

``` {.haskell include=src/QuickCheckV1.hs snippet=Property}
```

Expand All @@ -346,17 +363,27 @@ using the internal notion of coverage that property-based testing already has?
``` {.haskell include=src/QuickCheckV1.hs snippet=evaluate}
```

``` {.haskell include=src/QuickCheckV1.hs snippet=coverCheck}
``` {.haskell include=src/QuickCheckV1.hs snippet=classify}
```

``` {.haskell include=src/QuickCheckV1.hs snippet=testsC}
``` {.haskell include=src/QuickCheckV1.hs snippet=Config}
```

``` {.haskell include=src/QuickCheckV1.hs snippet=testsCPrime}
``` {.haskell include=src/QuickCheckV1.hs snippet=quickCheck}
```

``` {.haskell include=src/QuickCheckV1.hs snippet=classify}
### The extension to add coverage-guidance


``` {.haskell include=src/QuickCheckV1.hs snippet=coverCheck}
```

``` {.haskell include=src/QuickCheckV1.hs snippet=testsC1}
```

``` {.haskell include=src/QuickCheckV1.hs snippet=testsC2}
```

The full source code is available
[here](https://github.com/stevana/coverage-guided-pbt).

Expand Down
163 changes: 127 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ property-based testing wasn't a thing.
In this post I'll survey the coverage-guided landscape, looking at what
was there before Dan's post and what has happened since.

The short version is: imperative languages seem to be in the forefront
of combining coverage-guidance and property-based testing.
The short version is: today imperative languages seem to be in the
forefront of combining coverage-guidance and property-based testing.

In an effort to try to help functional programming languages catch up,
I'll show how coverage-guidence can be added to the first version of the
Expand Down Expand Up @@ -218,26 +218,44 @@ note](https://github.com/HypothesisWorks/hypothesis/pull/1564/commits/dcbea9148b
As far as I can tell, it hasn't been reintroduced since.

However it's possible to hook Hypothesis up to [use external
fuzzers](https://hypothesis.readthedocs.io/en/latest/details.html#use-with-external-fuzzers).

XXX: how does this work? like in Crowbar? What are the disadvantages?
Why isn't this the default?

What else has happenend since Dan's post?

- "Note: AFL hasn't been updated for a couple of years; while it should
> still work fine, a more complex fork with a variety of improvements
> and additional features, known as AFL++, is available from other
> members of the community and is worth checking out." --
> <https://lcamtuf.coredump.cx/afl/>
One of the first things I noticed is that AFL is no longer
[maintained](https://lcamtuf.coredump.cx/afl/):

> "Note: AFL hasn't been updated for a couple of years; while it should
> still work fine, a more complex fork with a variety of improvements
> and additional features, known as AFL++, is available from other
> members of the community and is worth checking out."
[AFL++](https://www.usenix.org/system/files/woot20-paper-fioraldi.pdf)
(2020)

- incorporates all of
[AFLFast](https://mboehme.github.io/paper/CCS16.pdf)'s [power
schedules](https://aflplus.plus/docs/power_schedules/) and adds some
new ones
- explain what power schedules are?
- <https://github.com/mboehme/aflfast>

- [AFL++](https://www.usenix.org/system/files/woot20-paper-fioraldi.pdf) (2020)
incorporates all of
[AFLFast](https://mboehme.github.io/paper/CCS16.pdf)'s [power
schedules](https://aflplus.plus/docs/power_schedules/) and adds some
news ones
- <https://github.com/mboehme/aflfast>
<!-- -->

- When you search for "coverage-guided property-based testing" in the
academic literature

- [FuzzChick](https://dl.acm.org/doi/10.1145/3360607) (2019). Not
released, lives in an [unmaintained
- [*Coverage guided, property based
testing*](https://dl.acm.org/doi/10.1145/3360607) by Leonidas
Lampropoulos, Michael Hicks, Benjamin C. Pierce (2019)

- FuzzChick Coq/Rocq library

- Not released, lives in an [unmaintained
branch](https://github.com/QuickChick/QuickChick/compare/master...FuzzChick)
that [doesn't
compile](https://github.com/QuickChick/QuickChick/issues/277)?
Expand Down Expand Up @@ -296,6 +314,8 @@ One key question we need to answer in order to be able to implement
anything that's coverage-guided is: where do we get the coverage
information from?

### Getting the coverage information

AFL and `go-fuzz` both get it from the compiler.

AFL injects code into every [basic
Expand Down Expand Up @@ -346,14 +366,11 @@ So the question is: can we implement coverage-guided property-based
testing using the internal notion of coverage that property-based
testing already has?

### The first version of QuickCheck

- QuickCheck as defined in the appendix of the original
[paper](https://dl.acm.org/doi/10.1145/351240.351266) (ICFP, 2000)

- Extended monadic properties

- Edsko de Vries'
[Mini-QuickCheck](https://www.well-typed.com/blog/2019/05/integrated-shrinking/)

``` haskell
newtype Gen a = Gen (Int -> StdGen -> a)

Expand All @@ -363,6 +380,9 @@ generate n rnd (Gen m) = m size rnd'
(size, rnd') = randomR (0, n) rnd
```

Footnote: We'll not talk about the coarbitrary, which is used to
generate functions.

``` haskell
rand :: Gen StdGen
rand = Gen (\_n r -> r)
Expand All @@ -373,6 +393,25 @@ sized :: (Int -> Gen a) -> Gen a
sized fgen = Gen (\n r -> let Gen m = fgen n in m n r)
```

``` haskell
class Arbitrary a where
arbitrary :: Gen a

instance Arbitrary Bool where
arbitrary = elements [True, False]

instance Arbitrary Char where
-- Avoids generating control characters.
arbitrary = choose (32,126) >>= \n -> return (chr n)

instance Arbitrary Int where
arbitrary = sized $ \n -> choose (-n,n)

instance Arbitrary a => Arbitrary [a] where
arbitrary = sized (\n -> choose (0,n) >>= vector)

```

``` haskell
newtype Property
= Prop (Gen Result)
Expand Down Expand Up @@ -424,6 +463,74 @@ evaluate :: Testable a => a -> Gen Result
evaluate a = gen where Prop gen = property a
```

``` haskell
label :: Testable a => String -> a -> Property
label s a = Prop (add `fmap` evaluate a)
where
add res = res{ stamp = s : stamp res }

classify :: Testable a => Bool -> String -> a -> Property
classify True name = label name
classify False _ = property
```

``` haskell
data Config = Config
{ maxTest :: Int
, maxFail :: Int
, size :: Int -> Int
, every :: Int -> [String] -> String
}

quick :: Config
quick = Config
{ maxTest = 100
, maxFail = 1000
, size = (+ 3) . (`div` 2)
, every = \n args -> let s = show n in s ++ [ '\b' | _ <- s ]
}

verbose :: Config
verbose = quick
{ every = \n args -> show n ++ ":\n" ++ unlines args
}
```

``` haskell
test, quickCheck, verboseCheck :: Testable a => a -> IO ()
test = check quick
quickCheck = check quick
verboseCheck = check verbose

check :: Testable a => Config -> a -> IO ()
check config a =
do rnd <- newStdGen
tests config (evaluate a) rnd 0 0 []

tests :: Config -> Gen Result -> StdGen -> Int -> Int -> [[String]] -> IO ()
tests config gen rnd0 ntest nfail stamps
| ntest == maxTest config = do done "OK, passed" ntest stamps
| nfail == maxFail config = do done "Arguments exhausted after" ntest stamps
| otherwise =
do putStr (every config ntest (arguments result))
case ok result of
Nothing ->
tests config gen rnd1 ntest (nfail+1) stamps
Just True ->
tests config gen rnd1 (ntest+1) nfail (stamp result:stamps)
Just False ->
putStr ( "Falsifiable, after "
++ show ntest
++ " tests:\n"
++ unlines (arguments result)
)
where
result = generate (size config ntest) rnd2 gen
(rnd1,rnd2) = split rnd0
```

### The extension to add coverage-guidance

``` haskell
coverCheck :: (Arbitrary a, Show a) => Config -> ([a] -> Property) -> IO ()
coverCheck config prop = do
Expand All @@ -432,11 +539,6 @@ coverCheck config prop = do
```

``` haskell
testsC' :: Show a => Config -> Gen a -> ([a] -> Property) -> StdGen -> Int -> Int -> [[String]] -> IO ()
testsC' config gen prop = tests config genResult
where
Prop genResult = forAll (genList gen) prop
genList gen = sized $ \len -> replicateM len gen
testsC :: Show a => Config -> Gen a -> ([a] -> Property) -> [a] -> Int
-> StdGen -> Int -> Int -> [[String]] -> IO ()
testsC config gen prop xs cov rnd0 ntest nfail stamps
Expand Down Expand Up @@ -477,17 +579,6 @@ testsC' config gen prop = tests config genResult
genList gen = sized $ \len -> replicateM len gen
```

``` haskell
label :: Testable a => String -> a -> Property
label s a = Prop (add `fmap` evaluate a)
where
add res = res{ stamp = s : stamp res }

classify :: Testable a => Bool -> String -> a -> Property
classify True name = label name
classify False _ = property
```

The full source code is available
[here](https://github.com/stevana/coverage-guided-pbt).

Expand Down
Loading

0 comments on commit 74e8aca

Please sign in to comment.