Confusing behaviour with random initialisation and small model changes

[neworderofjamie]

Not really sure what the solution is but, in a model where you have set a fixed seed and use random initialisation, making small (and seemingly unrelated) changes to the model such as toggling TRAIN below:

model._model.set_seed(1337)
...
pn_kc = model.add_synapse_population("pn_kc", "SPARSE_GLOBALG", genn_wrapper.NO_DELAY,
    pn, kc,
    "StaticPulse", {}, {"g": PN_KC_WEIGHT}, {}, {},
    "ExpCurr", {"tau": PN_KC_TAU_SYN}, {},
    genn_model.init_connectivity("FixedNumberPreWithReplacement", {"colLength": PN_KC_COL_LENGTH}))

if TRAIN:
    kc_mbon = model.add_synapse_population("kc_mbon", "DENSE_INDIVIDUALG", genn_wrapper.NO_DELAY,
                                           kc, mbon,
                                           symmetric_stdp, KC_MBON_PARAMS, {"g": KC_MBON_WEIGHT}, {}, {},
                                           "ExpCurr", {"tau": KC_MBON_TAU_SYN}, {})
else:
    kc_mbon = model.add_synapse_population("kc_mbon", "DENSE_INDIVIDUALG", genn_wrapper.NO_DELAY,
                                           kc, mbon,
                                           "StaticPulse", {}, {"g": np.load("kc_mbon_g.npy").flatten()}, {}, {},
                                           "ExpCurr", {"tau": KC_MBON_TAU_SYN}, {})

can change how the synapses in pn_kc are initialised. This is because, if TRAIN == True, addition threads are included in the initialisation kernel before the connectivity initialisation threads in order to set all the weights to KC_MBON_WEIGHT, meaning that different RNG sub-streams will end up being used to initialise the connectivity. This would also effects the single-threaded CPU backend as the order of the substreams is reflected in the order that the serial code samples from the Mersenne Twister RNG (which is, if anything, harder to solve).