Syncing Chores

[thormme]

Hi! I've been using the mod quite a bit, and have really appreciated it existing, so thanks!

I've also been looking into getting chores properly synced. I started using the FindNextChore patch as a base.
I modified it to only assign the next chore when the previous one had finished (if that's what happened on the host). I also added a patch to WorkChores that synchronizes them a little better by waiting until the host finishes before finishing on the client.
I did run into a couple of other issues, one is that often a chore would be picked up and sent to the client before it existed on the client. My solution was to create a queue that used coroutines to wait until the chore did exist.

I'm also a little worried that no amount of messing with chore priorities will be enough, since if the order of events is off at all between clients different chore will end up being available and quickly leads to unfixable desync. My only thought for getting around this is syncing the StateMachine Instances themselves, but they are both tricky to patch and have a lot of entry points. I have managed to patch GoTo, but haven't attempted to actually use the messages to sync the state yet, since I haven't sorted out where/how to interrupt the state transition.

My working branch is here if you're curious: https://github.com/thormme/oni_multiplayer/tree/thormme_chore_sync
It certainly isn't cleaned up or working much better than what was already there, I have mostly been throwing things at a wall and seeing what sticks, so I wouldn't expect it to get merged, but I thought I'd share.

What is your plan to sync them? Let me know if you'd like my help/input, or if you already have it sorted.

Thanks for a great mod!

[zuev93]

Hi @thormme
Thank you very much for this contribution and welcome to the community :)

Let me share our research and plans so far and we can think after how we can fit your work there.

Me and @polycone did 2 POCs for the similar thing: we made client creatures follow chores picked by the host. (one POC is in the main branch but with disabled harmony annotations, second one is here https://github.com/onimp/oni_multiplayer/tree/research/dupes-sync-via-preconditions ).

Both of them works but facing the same issues:

• Sometimes there is no corresponding chore on the client which leads to doing nothing
• Sometimes found chores are not exactly the same (either due to its dynamic nature or poor search functionality)
• There is problem with different execution speed

So all in all I feel like either of those 3 POCs are affected by mentioned issues in different ways.
I analyzed all chores and see that they can be splitted to 3 buckets:

1. Fully dependant on input (constructor parameters + BeginChore context), examples: AttackChore, DieChore, EmoteChore, etc)
2. Dynamic, but all changes are happens via *Parameter fields inside of State classes upon state transitions. Examples: AggresiveChore, RecoverBreathChore, etc
3. Dynamics but changes does not go through parameter fields (VomitChore, BansheeChore, etc)

So I feel like the entire epic could be broke down to following steps:

1. Create chore selection mechanism (we already have 3 POCs here)
2. Sync creation of chores from group # 1. (I'm working on it and I plan to push WIP branch in few days)
3. Chores from group # 2 requires additional sync of parameter changes upon transition + sync from previous step.
4. For chores from group # 3 we either need more complex syncs or we can patch them to follow group # 2 :)
5. Sync execution speed if needed (lerp chores progress / slow them down)
6. Proffit?

One more thing that I like about approach above is that it will allow us to enable chore syncs by small controlled groups. Instead of releasing the entire epic with quite drastic patches of the game logic.

Feel free to ask more question and join our discord server https://discord.gg/3TQ97w8Qwq
Best regards, zuev93

[thormme]

That's great context, thanks for the summary!
I don't know how I missed the link to the discord, I kept wishing that one existed, I joined up :)

For Step #4, are you suggesting rewriting the chores to rely entirely on parameter changes during transitions?
That's an interesting approach, I could see some complications relating to sim differences even doing that though. Could probably be done on a case by case basis though.

Step #5, I think this will be necessary for any chore that could possibly affect the execution of another chore, in my testing tiny differences in completion times can dramatically change the outcome. For example I've been doing testing with dig chores. Say one dupe is standing on a block that is being dug by another concurrently, if the block is dug out from under the dupe before finishing and they fall out of reach, they will be unable to complete the dig. Basically I'm pretty confident global chore completion (or worst case chore state transition) order will need to be preserved.

Some transitions, like the diggable task, are waiting on an event from the sim to make a transition. This complicates delaying the transition in order to synchronize state machines, as even if you prevent it, the physical modification of the environment has already happened. Simply waiting to do the work until the host has finished has mitigated this particular issue, but I haven't looked into every chore to see if this solution will be sufficient in general, since not everything is a WorkChore that can be delayed as simply.

[zuev93]

Hi,
Sorry for long delay.

But here some updates:
I've tried to break all known problems to few buckets:

So far I've done sync of chores creation (only some of them are enabled). That means that you can get rid off of that unstable search mechanism based on chore cell / type / name and refer Chore by MultiplayerId in a same way as we refer to GameObject and components.

In general I'd be happy for your contribution. However I think we need to go smaller steps here and try to address individual problems from table separately and enable those syncs only for some chores (until we are fully sure in it).

So far AFAIK neither me or @polycone continue working on our POCs about chore selection. So we are still missing this piece.
If you are interested perhaps you can revisit existing approaches, taking in account entire discussion and suggest some approach here?

Btw I've also introduced few integration-like tests for Chores (they don't spin up real Unity but mocking most parts of it).