[Data sync] Make data sync less error prone and easier to maintain

[gino-m]

The current data sync workflow is complicated, making it hard to understand and prone to errors (#2726, #2377, #2751, #2684). This is arguably most critical part of the system, so improving this should be P0+ priority.

The main components in question are LocalMutationSyncWorker, MediaUploadWorker, and SyncStatusViewModel and related fragment and repositories.

Suggestions:

• Expose a repository method to return mutations grouped by a new UploadQueueEntry model object, grouping create LOI+submission mutations together, sorted in reverse chronological order (oldest first, FIFO).
• Update LocalMutationSyncWorker as follows:
  • Use the above repo method to look at the entire queue on each run rather than just for one LOI.
  • Look at all incomplete mutations, not just FAILED. This will capture stuck operations. Since there will now only be at most one LocalMutationSyncWorker running at a time, this should be safe.
  • Treat each Operation as an atomic batch, let mutations in the operation fail or succeed together.
  • If any operation failed, retry() the worker.
• In addition to triggering LocalMutationSyncWorker on "submit", also trigger periodically (15 min?) to catch any "stuck" mutations, ie in case the worker crashed before it could retry() for some reason.
• Stop using "delete+insert" instead of "update" throughout and enable cascading on all foreign keys to guarantee consistency of local db and prevent accidental deletion of data.
• Ensure there's adequate unit test coverage for main use cases as well as all edge cases.
• Update the sync status screen to use the new UploadQueueEntry s returned by the repo.
• Simplify Mutation.SyncStatus and split off UploadQueueEntry.Status

See also #2235

@shobhitagarwal1612 @scolsen @sufyanAbbasi @jabramowitz5 @lecrabe @jo-spek @kenstershiro I would suggest prioritizing this over all other feature requests. Let's discuss.

[gino-m]

@shobhitagarwal1612 Let's clean split this work up and make this airtight.

[shobhitagarwal1612]

Should we also add a manual button to trigger data sync?

[gino-m]

Should we also add a manual button to trigger data sync?

The current behavior is to automatically retry whenever a connection is available, and I plan to add logic to periodically check the queue in case the last sync crashed. In that case, what purpose would the button serve? My concern is that by adding a button users will think they need to click it to have data be uploaded.

[gino-m]

A few action items remaining after #2867:

Essential cleanup:

• [Code health] Use UploadQueueEntry from MutationRepository.getUploadQueueFlow() to drive "Upload status" UI #2894

Nice-to-have:

• [PhotoSync] Track upload/retry status of individual photos #2120, potentially adding entries to a new table in the local db only once LocalMutationSyncWorker is successful. Once we do this, we can then only retry failed photos rather than all photos in a submission. This will be useful in jobs with more than one photo task.

[gino-m]

A few action items remaining after #2867:

Essential cleanup:

• [Code health] Use UploadQueueEntry from MutationRepository.getUploadQueueFlow() to drive "Upload status" UI #2894

Nice-to-have:

• [PhotoSync] Track upload/retry status of individual photos #2120, potentially adding entries to a new table in the local db only once LocalMutationSyncWorker is successful. Once we do this, we can then only retry failed photos rather than all photos in a submission. This will be useful in jobs with more than one photo task.

Closing this issue in favor of the above cleanup tickets. W00t!