Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Desktop, Mobile: Harden failsafe logic to check for the presence of info.json, rather than just the item count #11750

Open
wants to merge 5 commits into
base: dev
Choose a base branch
from

Conversation

mrjo118
Copy link
Contributor

@mrjo118 mrjo118 commented Jan 29, 2025

In the past 3-4 months there have been a number of issues raised on the support forum where Joplin users who use OneDrive sync have claimed that their notes 'suddenly disappeared'. While it is not possible to determine if all cases were caused by the same issue, there is evidence through user feedback from @BeefStorm on this thread Restore Deleted Backups - #18 by BeefStorm that this issue can occur without any user interaction, via a re-authentication of OneDrive happening in the background, which may sometimes cause the sync target (special) folder on OneDrive to change and somehow bypass the failsafe mechanism when that occurs.

I proposed and have now implemented a method for how the failsafe could be hardened to be more protective from data loss on all sync targets. In summary:
During the delta step of the synchronization code, make it check whether the sync target (remote) contains an info.json file, following each invocation of the delta api which fetches the list of remote items, and also when synchronization is started (just before info.json is created). If it does not, then trigger a failsafe error specifying that the sync target is empty or damaged. To avoid hitting this error when doing the initial sync on a device, bypass this error if the local database contains no items in the sync_items table matching the current sync target (then when it starts syncing it will create the info.json so that subsequent syncs should work without issue).

I have tested the following scenarios on Joplin desktop, using a profile containing at least a couple of notes:
-Test setting up a new sync target with file system sync, when sync has never been set up before: The sync completes without issues, regardless of whether failsafe is enabled
-Test setting up a new sync target with onedrive sync, when sync has never been set up before: The sync completes without issues, regardless of whether failsafe is enabled
-Test setting up a new sync target with onedrive sync, when sync has never been set up before for onedrive, but has for file system sync: The sync completes without issues, regardless of whether failsafe is enabled
-Test setting up file system sync and syncing fully, then setting up onedrive sync and syncing fully, then switching back to file system sync on the same directly but with the .sync folder and info.json deleted: This should trigger the error if failsafe is enabled, or no error if it is not
-Test .sync folder and info.json being deleted for file system sync for an existing target, then trigger the sync: This should trigger the error if failsafe is enabled, or no error if it is not
-Test failsafe check during the delta step: Make 60 dummy notes, duplicate them and click sync. As the sync status starts notifying of items created, delete the .sync folder and info.json before it completes. This should trigger a failsafe error if failsafe is enabled, or complete the sync if it is not
-Test upgrading the sync target by deleting info.json but not .sync/version.txt: This should work correctly, prompting the user to upgrade via a banner, and upong restarting via this banner the sync should be restored and the info.json file gets recreated. Note that on the desktop, the upgrade banner is dismissed if switching note after it shows, and requires restart of Joplin for it to come back, after which the message is persistent. Note that this is existing behaviour
-Test 'Delete local data and re-download from sync target' option: It should work correctly without errors and sync continues to work afterwards (tested with OneDrive)
-Test 'Re-upload local data to sync target' option: It should work correctly without errors and sync continues to work afterwards (tested with OneDrive)
-Test 'Delete local data and re-download from sync target' option with the .sync folder and info.json deleted: It should work correctly without errors and sync continues to work afterwards (tested with file system sync). Also when deleting just info.json, it triggers the sync upgrade banner, and upon restarting via the banner the sync completes successfully
-Test 'Re-upload local data to sync target' option with the .sync folder and info.json deleted: It should work correctly without errors and sync continues to work afterwards (tested with file system sync). Also when deleting just info.json, it triggers the sync upgrade banner, and upon restarting via the banner the sync completes successfully

Copy link
Collaborator

@personalizedrefrigerator personalizedrefrigerator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this!

I've left a few comments based on a quick look through the changes.

packages/lib/services/synchronizer/syncInfoUtils.ts Outdated Show resolved Hide resolved
packages/lib/services/synchronizer/syncInfoUtils.ts Outdated Show resolved Hide resolved
packages/lib/services/synchronizer/syncInfoUtils.ts Outdated Show resolved Hide resolved
@mrjo118
Copy link
Contributor Author

mrjo118 commented Jan 31, 2025

@personalizedrefrigerator all comments now addressed

Copy link
Collaborator

@personalizedrefrigerator personalizedrefrigerator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've briefly looked through the updated changes. All comments I've left have been marked "optional".

So far, it looks good to me!

Comment on lines 362 to 367
let errorMsg = null;
try {
await checkSyncTargetIsValid(fileApi());
} catch (error) {
errorMsg = error.message;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: Consider using Jest's .toThrow or .rejects.toThrow.

errorMsg = error.message;
}

expect(errorMsg).toBe('Fail-safe: Sync was interrupted to prevent data loss, because the sync target is empty or damaged. To override this behaviour disable the fail-safe in the sync settings.');
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: Rather than checking the full fail-safe message, consider checking just part of it:

Suggested change
expect(errorMsg).toBe('Fail-safe: Sync was interrupted to prevent data loss, because the sync target is empty or damaged. To override this behaviour disable the fail-safe in the sync settings.');
expect(errorMsg).toContain('Fail-safe: ');

Benefit: Such a change could simplify changing the fail-safe message in the future.

@personalizedrefrigerator personalizedrefrigerator added the sync sync related issue label Feb 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sync sync related issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants