sys/syz-sysgen: serialize descriptions as gob and embed #5698
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dvyukov
requested review from
blackgnezdo,
mptre,
markjdb,
tuexen,
mvanotti,
glpesk and
eepeep
as code owners
Instead of generating Go files with descriptions serialize them as gob and compress with flate. This significantly reduces build time, go vet time, and solves scalability problems with some static analysis tools. Reference times (all after rm -rf ~/.cache/go-build) before: TIME="%e %P %M" time go install ./syz-manager 48.29 577% 4824820 TIME="%e %P %M" time go test -c ./prog 56.28 380% 6973292 After: TIME="%e %P %M" time go install ./syz-manager 22.81 865% 859788 TIME="%e %P %M" time go test -c ./prog 12.74 565% 267760 syz-manager size before/after: 194712597 -> 83418407 -57% even provided we now embed all descriptions instead of just a single arch. Deflate/decoding time for a single Linux arch is ~330ms. Fixes google#5542
dvyukov
force-pushed
the
dyvukov-serialize-descriptions
branch
from
e217255 to
c5d31d5
Compare
tarasmadan
approved these changes
Jan 23, 2025
a-nogikh
approved these changes
Jan 23, 2025
|
It's especially cool to see the speedup of |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Instead of generating Go files with descriptions
serialize them as gob and compress with flate.
This significantly reduces build time, go vet time,
and solves scalability problems with some static analysis tools.
Reference times (all after rm -rf ~/.cache/go-build) before:
TIME="%e %P %M" time go install ./syz-manager
48.29 577% 4824820
TIME="%e %P %M" time go test -c ./prog
56.28 380% 6973292
After:
TIME="%e %P %M" time go install ./syz-manager
22.81 865% 859788
TIME="%e %P %M" time go test -c ./prog
12.74 565% 267760
syz-manager size before/after: 194712597 -> 83418407
-57% even provided we now embed all descriptions
instead of just a single arch.
Deflate/decoding time for a single Linux arch is ~330ms.
Fixes #5542