Document the system processes

[seanshahkarami]

There should be some high level documentation of what's happening in the system and what you should expect to see at every stage. This would be useful, not only to be able to hand off to some one else, but also as a general resource to help someone understand how the system should behave and respond under different circumstances.

Even for now, something simple thing like the following could be helpful:

Bootloader starts:

1. Powers NC on failsafe media if early NC power flag enabled.
2. Start setup process.

Setup process:

1. Get and clear boot flags.
2. Enable reset the watchdog.
3. Setup persistant records.
4. Setup hardware / sensors.
5. Update boot time and counts.
6. Setup device states.
7. Start main process.

Main process:

1. Start next device, if start timer expired.
2. Update device processes.
3. Process commands.
4. Send logs every 60 seconds.
5. Reset system, if reset requested.
6. Reset watchdog.

Device process (normal operation):
...will add more...

Device process (failsafe operation / 30+ failures):
Every 4 hours, the device is killed and then started on its other boot media.

Heartbeat process:
A background heartbeat check is run on a timer in the background. It checks if the last read state of a heartbeat pin has changed, and if so, it increments that pins changed counter. The devices later check if this change count is above a certain threshold and if so, trigger a heartbeat event.