The Anatomy of a Connector

A Connector has only two high-level duties:

1. Sync a local copy of all data down from an external source.
2. Provide bare bones access to the current state and historical archive of that data via events and a RESTful API.

There are additional requirements (such as handling authentication with the external service), but those serve to support one of the high-level duties.

The Bits

A Connector has several parts (using Foursquare as an example):

• foursquare.connector - This is the manifest file for the foursquare connector. Information about manifest files can be found in the App Manifest wiki page.
• init.js - This is a configuration file for the common client code, and is what actually launches the connector.
• sync-api.js - The data access API. Provides very basic RESTful API access to all stored data. All endpoints should be defined in this file, and all events that are emitted from the syncing logic are defined here.
• sync.js - Contains all logic for robustly syncing data. Handles all aspects of this including error handling, calming, scheduling, etc.

The Flow

These modules are added dynamically and only as needed. The startup process of the Connector is as follows (again, using Foursquare as an example, however, any connector utilizing the shared components will have a similar flow):

1. Core receives a request for one of the Connector's endpoints, buffers it, and spawns a process with node init.js.
2. Init.js will call upon the common client code that listens for data from core.
3. Core writes basic process info (port, workingDirectory, lockerUrl) to the process's stdin (as JSON).
4. Init.js then creates an empty webserver, loads either a custom auth library (in the case of Twitter, for example) or loads the shared oauth2 library, attaches the sync-api.js endpoints, moves into the proper working directory, and finally connects to mongo. Once it has connected to mongo, it will write process info back out to stdout to indicate that it's ready.
5. The Connector calls the authAndRun function of the auth module.
   • If the Connector has already been auth'd, this function calls back synchronously.
   • If the Connector has not been auth'd, it adds the necessary web service endpoints for authentication. When complete, it calls back.
6. When the auth lib calls back, client.js calls sync-api.authComplete, which adds the sync API endpoints.
7. Core passes on the original buffered request and any other requests that have been buffered during the startup process to the Connector.

State of the common code bits

There's still a bit too much copy and paste code floating around, and creating a new connector is still a bit more work than it needs to be. The entire sync-api file should be able to abstracted down to a few config file lines that list out the endpoints that should exist. The connector utils library should be expanded to include more repeated logic. It still takes a bit more code than should be necessary to walk through pages of an endpoint, or to drill down from an API list request (one that returns a list of IDs) to pulling down the information about each of those IDs. It shouldn't be unreasonable to be able to add a fully tested connector in less than a day, but right now there's still too much code to write.

Future Direction

Currently, connectors solely pipe data from the provider to the local data store. In time, these will evolve into bi-directional connections that will be able to funnel data back from applications to the provider.