Skip to content

Create a new connector

Jump to bottom Edit New page
smurthas edited this page Jul 19, 2011 · 26 revisions

The easiest way to get going is to copy and paste the code in Locker/Connectors/skeleton into a new directory for your connector. Let's pretend there isn't a yet a foursquare connector (I'll use "FoursquareExample" as the name so nothing bas happens):

cd Connectors
mkdir FoursquareExample
cd FoursquareExample
cp -r ../skeleton/* .

The skeleton provides the framework for a OAuth2-based connector to a REST API, but can easily be modified to handle other scenarios.

The first thing we'll want to do is change the name of the .connector manifest file:

mv skeleton.connector FoursquareExample.connector

Now let's edit that file. The only things that MUST be changed are the handle, mongoCollections, and provides fields

{
    "title":"Foursquare Example",
    "action":"Connect to a Foursquare account",
    "desc":"Collect and sync my data from my Foursquare account using the Foursquare API and an Foursquare 'app' that I create just for myself.  To enable this to show up in the available list of services, you'll need to set displayUnstable to true in your config.json",
    ...
    "handle":"foursquareexample",
    "mongoCollections": ["friends"],
    "provides":["contact/foursquareexample"]
}

More info can be found about the provides field on the service types page.

Next let's take a look at the init.js file

require('connector/client').init({...});

This loads the common connector bootstrap file and initializes it with some startup info - in this case it passes OAuth 2 settings. If your service uses OAuth 1.0, checkout the twitter connector for a well documented template. For username and password, see the Username and Password Auth section below.

For more details about the init.js file and client.init function, see the Custom Files section below.

Let's put foursquare info into the init.js file:

require('connector/client').init({"oauth2" :
    {"provider" : "Foursquare",
     "appIDName" : "Client ID",
     "appSecretName" : "Client Secret",
     "authEndpoint" : "authenticate",
     "accessTokenResponse" : "json",
     "endPoint" : "https://foursquare.com/oauth2/",
     "linkToCreate" : "https://foursquare.com/oauth/register",
     "grantType" : "authorization_code"}});

Next, let's edit the sync.js file. Add some code to the syncItems function:

exports.syncItems = function(callback) {
    //hit the friends API endpoint
    request.get({uri:'https://api.foursquare.com/v2/users/self/friends?oauth_token=' + auth.accessToken}, 
                        function(err, resp, body) {
        var json = JSON.parse(body);
        //probably should check for errors here
        addItems(json.response.friends.items, function() {
            //repeat every hour
            callback(err, 3600, "Updated " + json.response.friends.count + " friends");
        });
    });
}

And then add the addItems function:

function addItems(friends, callback) {
    if(!friends || !friends.length) {
        process.nextTick(callback);
        return;
    }
    var friend = friends.shift();
    dataStore.addObject('friends', friend, function(err) {
        var eventObj = {source:'friends', type:'new', data:friend};
        exports.eventEmitter.emit('contact/foursquareexample', eventObj);
        addItems(friends, callback);
    });
}

Finally, let's tailor the sync-api.js file to represent what we are doing. Let's change the name of the /items endpoint to /updateFriends:

app.get('/items', items);

becomes:

app.get('/updateFriends', updateFriends);

and then we change the items function to updateFriends:

// this is the basic structure of an endpoint for something you'd be parsing.
function updateFriends(req, res) {
    ...
    locker.at('/updateFriends', repeatAfter);
    ...
}

and finally edit the html in the index function to point to /updateFriends instead of /items:

res.end("<html>found a token, load <a href='updateFriends'>friends</a>");

Now we're ready to run.

cd ../..
node lockerd.js

Then open up the dashboard at http://localhost:8042 and navigate to the Services section. Click the "[show unstable]" link in the top right hand corner. "Foursquare Example" should now be listed as an installable connector. Clicking the Install Connector button should now install the connector and take you to the auth page. Follow the instructions from there.

You've made your first connector!

Recommended reading before proceeding: https://github.com/LockerProject/Locker/wiki/The-Anatomy-of-a-Connector

We've worked towards abstracting out as much of the connector logic as possible so that there is a bare minimum of repeated code. All of this base connector logic is provided via the Common/node/connector set of files. Those will be described first, and then this document will walk through the process of utilizing those to create a new connector.

The easiest way to get going is to copy and paste the code in Connectors/skeleton into a new directory for your connector:

# from the base Locker directory
cd Connectors
mkdir MyNewConnector
cd MyNewConnector
cp -r ../skeleton/* .

Common Files

  • api.js - Common set of data access endpoints.
  • client.js - This launches the connector, as well as establishing a connection to mongo, instantiating an express-based web server, and loading in the remainder of the files required for a connector to run.
  • dataStore.js - This library handles all the data storage for a connector. Currently, it's setup to write data to mongo and to dump data in a JSON-like format to disk.
  • oauth2.js - This will handle oauth2 connections. Right now, any other type of authentication will need to be custom written. There's the beginnings of an oauth1 provider baked into the Twitter connector that will need to be abstracted and made available to other connectors. The IMAP connector is an example of a connector that just uses username/password authentication. When storing username/passwords locally, please use the provided lcrypto functions to ensure we aren't storing plain text username and passwords on a person's machine. While this provides little to no security in the case of someone having access to the filesystem, it does at least ensure that a simple grep of a user's machine doesn't expose passwords.

Custom Files (using foursquare as the example)

  • foursquare.connector - The manifest file for a service. Must use the file extension of connector, as that will be what differentiates this service from applications and collections.

  • init.js - This is mostly a configuration file. These options are passed to the init function of the connector/client module which needs to be included from here. Options currently supported are:

    • 'enableCookies' : boolean (default false) - Setting this to true will cause the client to add in all the session logic from express.

    • 'id' : string (default 'id') - This is the ID key used for mongo. Usually this will be id, but sometimes (for example, twitter uses id_str instead since id is a very large integer that doesn't play nicely with javascript) the provider will use a different string as the id.

    • 'oauth2' : object - This object contains a set of options used for defining the way oauth2 is integrated into this connector. The defaults for the oauth2 object are as follows:

        options = {provider :            'Some oauth2 consumer',       // This is the name of the provider
             endPoint :            'http://consumer.com/oauth/', // This is the base endpoint at the provider
             linkToCreate :        'http://change.me/',          // This is the URL where someone would add an application
             appIDName :           'App ID',                     // String shown to the user when asking for app ID
             appSecretName :       'App Secret',                 // String shown to the user when asking for app secret
             authEndpoint :        'authorize',                  // End point used for authorization at the provider
             promptForUsername :   false,                        // Whether or not the user should be prompted for their username.
             accessTokenResponse : 'text',                       // Either text or JSON.  If the token is passed in the body of the response, leave this as the default.
             grantType :           '',                           // If a grant type needs to be specified to the provider, it should be defined here.
             extraParams :         '',                           // Any extra params that need to be included when establishing a connection.
             width :               980,                          // This defines the width of the popup window used for authenticating to the provider.
             height :              750};                         // This defines the height of the popup window used for authenticating to the provider.

  • sync-api.js - This file will contain all of the endpoints used to synchronize data from the provider. It must contain a function called 'authComplete' that accepts 2 parameters, the first being the authentication object, and the second being the mongo object. This function should establish all the sync API endpoints, and should handle emitting events from the sync library to the locker itself. This file should contain very little logic, it should simply establish endpoints that are used to call functions in the sync.js file that handle all the work of querying the provider.

  • sync.js - This is where the majority of custom code will reside. All the logic for pulling down data from the provider will reside in this file. All that data is subsequently loaded into the dataStore via the functions provided by the dataStore.js file. This file should export an eventEmitter for all of the supported service types to pass along events up the chain on create, update, and delete of objects.

Testing

Testing should be done via the fakeweb library to provide fixtures to the sync library. connector-local-foursquare-test.js is a fully featured example test. This covers the syncing functions, the events that are generated, as well as ensuring that the data is properly accessible via the common APIs.

Add a custom footer
Add a custom sidebar
Clone this wiki locally