Design: Offline API Calls

Author: Julian Kung

Updated:

• 5/4/2021
• 2/22/2021
• 1/15/2021

Related Links

• General Offline Design: TODO

Overview & Context

The app will be used in rural Myanmar and so must behave well in offline environments. The user should be able to simulate POST requests such as creating users or logging meter readings / payments while offline. These actions should be immediately reflected on the client side, and their corresponding API calls to update the backend/Airtable should be propagated when the user re-establishes connection to the internet.

Service workers will be used to run the app while offline. We will be using the workbox library for managing offline-related tasks. This document will focus on offline API calls rather than general offline functionality. The document related to general offline functionality is located in the "Related Links" section.

Goals

• Users should be able to create a customer, meter reading, or payment when offline
• Customers, meter readings, and payments made offline should be immediately reflected on the client side app
• Users should be able to create meter readings and payments for customers that were created offline.
• POST/PUT API requests made when offline should be made ASAP once connection to internet is re-established
• Summary statistics, such as financial reports, etc... should update accurately when customers, meter readings, or payments are made offline.
• Implementation should work on most major browsers (and noted on the ones that do not work).
  • This is relevant for features such as background-sync which may only work on Google Chrome.

Non-goals

• Fixing consistency issues that may occur between 2 users managing the same site.
  • We will naively employ a "last-POST-wins" based on the timestamp when the request was initially made.
• Preserving offline functionality when authentication is required (login)

Constraints

• Users are expected to have internet connection at least once a day to send and receive data

Data Storage

• Data pulled from Airtable will be kept in a Redux store.
• Data in the redux store will be refreshed every 15 minutes,
• Data will also be refreshed whenever the user goes from an "offline" state to an "online" state while in the app.
• Client-side and Airtable data may become inconsistent in some offline cases if multiple users manage the same site.
  • Consistency will be restored when Users return back online and refresh their data. We will use the Airtable as ground truth.
  • In cases where 2 users create a meter reading for the same customer while offline, we will persist the meter reading with the reading amount.
  • NOTE: Consistency issues are tricky (especially with POST requests) and we should avoid over-engineering "smart" solutions and instead advocate for good communication between users managing the same site.

Workbox Background Sync

Workbox background sync is a method using the workbox library that queues network requests to be made when a user re-establishes connection to the internet.

Using Background Sync

• Incorporates exponential drop off for failed requests (error safe)
• Requests waiting to be sent are stored in a queue in indexedb.
• Implementation uses a sync event fired by the browser to begin sending requests.
  • Delivery can happen even if the user has left the application
  • Backup methods available for browsers without sync event

Offline Cases and Solutions

Offline functionality is complicated by the existence of dependent objects in the database schema, such as meter readings, payments, customer updates, and incident report updates. These objects are linked to independent objects such as customers, incident reports, etc.

You can only "link" an object to another in Airtable by passing the recordID of an existing object (Airtable's foreign key for that object) as a value when creating the new object. This complicates cases for offline functionality because an object only receives a recordID once it is written to Airtable.

There are 3 distinct object creation cases that need to be accounted for:

Offline Case	Solution
An independent object (ie: customer, incident report, etc) gets created while offline	We will store the object in Redux so it is shown immediately to the client. We queue an API request to the backend that will be sent when connection is re-established to create the object in Airtable.
A dependent object (ie: meter reading for a customer) gets created while offline from an independent object with a record ID.	Same solution as above, since a record ID is available to link the two objects together, no special action needs to be taken for the queued API request to create the dependent object.
A dependent object gets created offline from an independent object without a record ID.	This case happens when an independent object is created while offline, and an object dependent on the independent object is also created. This solution is out of scope for the MVP.

Using Background Sync out of the box will allow the developers to abstract away the first two cases. This is because if the initial network request fails due to connection problems, the API request will be placed in the Background Sync queue and re-sent at the next available opportunity.

Concerns

• The API requests stored in the indexedb queue may be fired off by web workers when connectivity is re-established. If that's the case, we need to account for race conditions between modifying an API request and that API request being sent off.

Alternatives Solutions Considered

Since Background Sync inherently covers both the 1st and 2nd cases, we only considered alternative solutions to the 3rd case

• Chaining API Requests
  • For case 3, create a callback to be invoked by the independent object's API request that will invoke the dependent object's API request, passing the recordID returned when creating the independent object.
  • Concern: We couldn't find a reliable method to allow for chaining API requests in Workbox, and developing an in-house solution would be more advanced work.