Upload data async to HAPI

[ageryck]

Currently, the sync API operates on the principle of all or none this means if we have client-side resource changes that require sync up the API will attempt to batch all in a bundle that can entirely either succeed if all checks are passed or fail. This hinders data sync to the server side from this client completely until the issue is resolved from the client side, this issue is to try to figure out ways of changing the sync mechanism to eliminate stopping sync completely because of a few malformed/failing resources.

Options to discuss are;

1. Can we handle sync in batches and treat each batch as autonomous to the other such that if one fails it is marked as so and sync proceeds to the second batch
2. There's an option of syncing individual resources and treating individual fails separately from individual successes
3. Bundling related resources together for autonomous syncs

[pld]

(1) certainly makes sense, the other options sounds a bit more complex, (2) could make sense, but we'd expect lots of failure since bundles should reflect dependencies, but maybe those failures are OK, we don't lose anything with the failure (other than time), (3) would be a way to do (2) more effectively, if there's an existing libraries that creates a dependency graph of resources, and converts that graph into bundles, that'd be ideal.

@ellykits can you add some insight on how sync is done now, link to where that is in the code, and your thoughts on low effort ways to let sync complete for anything that does not fail (eg not fail all on single fail)

[ageryck]

Or if there can be a way to activate the dep graph only after the bundle fail, so in my example above sync batches of 500, and on the fourth bundle fails try organize it in dep graphs and in case of any success proceed to the next records in batches again to improve efficiency but maybe this is complicating it further

[pld]

@ageryck in practice, in the sync failures you've seen, about how many total resources remain unsynced on device?

[pld]

looking through the code, it is clear the change I suggested would be non-trivial, since we would have to implement our own sync worker, it looks like we could use the UploadRequestGeneratorFactory in UrlRequest to upload individual files, but still non-trivial. See if Google is interested in putting a depndency graph based sync approach on their roadmap. The plan I'd see for that, to consolidate would be:

1. create a dependency graph of all unsynced resources
2. iterate through the graph and for each tree create a bundle
3. sync each bundle independently in a transaction, using the existing transaction generation logic

[ageryck]

@pld This varies across devices we have devices that have over 800 tasks resources (this is an absolute count of create and update payloads), 250 careplans, 135 conditions etc so this can be as few as two tasks to as many as possible depending on when sync stopped working and how much more data has been collected afterward. Here a sample resources after sync failure

[ellykits]

See if Google is interested in putting a depndency graph based sync approach on their roadmap. The plan I'd see for that, to consolidate would be:

1. create a dependency graph of all unsynced resources
2. iterate through the graph and for each tree create a bundle
3. sync each bundle independently in a transaction, using the existing transaction generation logic

With the unsynced resource dependency graph available we can use the UrlRequest to send single requests for each bundle generated in step 2.

[ageryck]

Exploring options of uploading all data to HAPI then running validations server-side as documented here