Debug 2025

[DonggeLiu]

by cherry-picking commits from #2025 and #2038

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-28-empty-seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

[DonggeLiu]

Experiment 2024-08-28-empty-seeds-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[DonggeLiu]

Seeing errors from trials:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/src/fuzzers/libafl_empty_seeds/fuzzer.py", line 71, in fuzz
    prepare_empty_corpus(input_corpus)
  File "/src/fuzzers/libafl_empty_seeds/fuzzer.py", line 42, in prepare_empty_corpus
    shutil.rmtree(input_corpus)
NameError: name 'shutil' is not defined

Did you have a chance to test this locally?

[tokatoka]

yeah sorry i forgot to add import for that part. will fix soon

[tokatoka]

5a717af
b8eecbe

Can you cherry pick these two?

[DonggeLiu]

5a717af b8eecbe

Can you cherry pick these two?

I don't think I can. They are from your forked repo, not FuzzBench.
You might need to create a PR based on this one to push those commits.

[tokatoka]

These two are pushed to the branch here #2025
(the last two commits)

[DonggeLiu]

These two are pushed to the branch here #2025 (the last two commits)

As I said, they are from your forked repo, not FuzzBench. Pushing them to a PR to FuzzBench does not make them FB.
Hence I cannot cherry-pick:

fatal: bad revision '5a717af'

It's better if you could create a PR based on this one and add them.

[tokatoka]

ok i opened here
#2041

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-30-empty-seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

[tokatoka]

i think we triggered it
we have coverage.json here but they are all empty https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-08-30-empty-seeds-1/coverage/data/
and the report is not generated either
did you see something in the log?

[tokatoka]

@DonggeLiu can you check the log?

[DonggeLiu]

OK, I just had some time to look into this.

Conclusion

The error is likely caused by missing .profraw files during experiments:

This error is from merge_profdata_files(). The error message indicates there is no src_files provided in the command, hence unable to merge them into the destination profdata file (/work/measurement-folders/sqlite3_ossfuzz-libfuzzer_empty_seeds/merged.profdata)

As a result, FuzzBench failed on generate_json_summary() and generate_coverage_report() while there are no profdata_file:

BTW, this is not the missing fuzz target binary error we have seen before, as this message was not printed and all binaries exist under /out:

---

Next

I reckon the main question is "Is this caused by FuzzBench or the new fuzzers"?
I can do three things to help confirm this:

1. Add more debug logs to confirm that src_files is not in vanilla libfuzzer but empty for its new variation.
2. Disable merging previous experiment result data.
3. Run another exp with libfuzzer and its new variation.

I will do 1&2 now, and if you have more logs to add, please do so too.
Let me know when it is ready, I will do 3.

[tokatoka]

I added more logs here
#2042

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-03-empty-seeds-1 --fuzzers libfuzzer_empty_seeds afl_empty_seeds libfuzzer

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-03-empty-seeds-1 --fuzzers libfuzzer_empty_seeds afl_empty_seeds libfuzzer

[tokatoka]

https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-09-03-empty-seeds-1/
this is empty. so nothing was run (?)