Generated C code consistency debugging, part 1

[nikodemus]

I want both bootstrap* and target** compilers (transpilers) to produce identical C code, since it will mean I don't need a multi-stage bootstrap to get consistent results. This is not the case yet!

For now I've been focusing on generated_declarations.h. After recent changes, including the as-of-yet unmerged #771 the differences have gone down to having six extra closure ids from bootstrap compiler:

$ diff -u bootstrap-c/generated_declarations.h target-c/generated_declarations.h
--- bootstrap-c/generated_declarations.h        2021-07-18 16:29:41.822310400 +0300
+++ target-c/generated_declarations.h   2021-07-18 16:49:23.376063900 +0300
@@ -4132,9 +4132,3 @@
 struct Foo fooclosure_id1297(struct FooContext* ctx);
 struct Foo fooclosure_id1298(struct FooContext* ctx);
 struct Foo fooclosure_id1299(struct FooContext* ctx);
-struct Foo fooclosure_id1300(struct FooContext* ctx);
-struct Foo fooclosure_id1301(struct FooContext* ctx);
-struct Foo fooclosure_id1302(struct FooContext* ctx);
-struct Foo fooclosure_id1303(struct FooContext* ctx);
-struct Foo fooclosure_id1304(struct FooContext* ctx);
-struct Foo fooclosure_id1305(struct FooContext* ctx);

Cursory examinations showed that the stuff in those missing closure ids clearly exists on the target side as well: Closure ids don't have identical content on either side! Oops.

To figure out how bad that is, I instrumented the build to pretty-print the AST of every generated closure into standard output along with its id.

Debug println: "CLOSURE {index}:".
Debug println: "args: {closure argumentVariables collect: #name}".
Debug print: "body:\n    ".
AstPrinter
    print: closure body
    withIndent: 4
    to: Debug.
    Debug println: "\n--".

which prints out things like this:

CLOSURE 8:
args: ["a", "b"]
body:
    (a / b)
--

Examining this output from both bootstrap and target showed that closures have matching ids and ASTs upto id 400-something, after which first difference appears. (Which appeared to be a difference in ordering: the ones appearing on both sides were found on either side, just in different positions and ids.)

To find out how much content differences I have (never mind the ordering just now) I did my first bit of using Foolang for scripting: https://gist.github.com/nikodemus/2ececebfd6c786a6d7bdcbd575fa9dcf

That split the build logs containing closure ids and ASTs into individual files.

Then I did a bit of terminal work to index closure ids under SHA1s of their ASTs:

for file in bootstrap-closures/*
do
    echo $file; sha1=$(sha1sum $file | cut -d' ' -f1)
    echo $file >> bootstrap-shas/$sha1
done

(and the equivalent to generate target-shas/)

Now I could check what is actually missing:

for file in bootstrap-shas/*
do
    sha=$(basename $file)
    if [ ! -e target-shas/$sha ]
    then
        echo missing: $sha
        cat $file
        cat $(cat $file)
    fi
done

which, drumroll 🥁🥁🥁

shows I have the content of just three closures missing:

missing: 21a6a12acc875d31fb818f5526e18519c3ba540a
bootstrap-closures/CLOSURE_406
args: []
body:
    ((1 + to) - from)
missing: bd589a47a2bd21fc5666352bf310bed50c4f1fbc
bootstrap-closures/CLOSURE_424
args: ["i"]
body:
    (new
         put: (block
                   value: (self
                               at: i)
                   value: (other
                               at: i)))
missing: ff8210a656886308b20da3614c356e2fb58911e1
bootstrap-closures/CLOSURE_407
args: []
body:
    (((1 + to) - from) truncate / by)

...all of which on the face of it look like array / iterator / interval code.

Haven't yet had a chance to look for the sources of these, or the cause of this, or why I'm missing six ids but only three ASTs... that's going to be in part 2!

(*) bootstrap = compiling the compiler with itself while running under the bootstrap interpreter.
(**) target = compiling the compiler with the compiler produced by bootstrap stage.