synclets.json

A quick guide to the magical bits of how synclets are configured!

Here's an example twitter one, breakouts below:

{
    "provides":["contact/twitter", "mentions/twitter", "related/twitter", "timeline/twitter", "tweets/twitter"],
    "mongoId" : {
        "timeline":"id_str",
        "tweets":"id_str",
        "mentions":"id_str"
    },
    "types" : {
        "timeline":"tweet",
        "tweets":"tweet",
        "self":"contact",
        "mentions":"tweet"
    },
    "profileIds" : ["id_str", "screen_name"],
    "strip" : {
        "contact":["status"]
    },
    "synclets":[
        {"name": "self", "frequency": 7200},
        {"name": "friends", "frequency": 3600},
        {"name": "tweets", "frequency": 600},
        {"name": "timeline", "frequency": 120},
        {"name": "mentions", "frequency": 600},
        {"name": "related", "frequency": 3600},
        {"name": "tweet"}]
}

provides

    "provides":["contact/twitter", "mentions/twitter", "related/twitter", "timeline/twitter", "tweets/twitter"],

This is actually deprecated, it used to be a way to ask for a list of sources dynamically with these strings but for any new connectors/synclets it can be ignored as everything is a /Me/twitter/getCurrent/contacts or similar well-known pattern.

mongoId

    "mongoId" : {
        "timeline":"id_str",
        "tweets":"id_str",
        "mentions":"id_str"
    },

Each synclet returns one or more namespaces of data (though often they're 1:1). The objects within a space must each have a unique id, but since ideally they are the raw source JSON or as unaltered as possible, we need to have a way to say which field is the unique id. By default it's "id" so only ones that are different need to be here. While it's named mongoId, it has nothing to do with mongo and is just a legacy naming. Be aware that if the ID is more than one level deep in the graph, you'll have to extract it from within the synclet, as the configuration doesn't understand dot notation.

types

    "types" : {
        "timeline":"tweet",
        "tweets":"tweet",
        "self":"contact",
        "mentions":"tweet"
    },

While there are different namespaces of data, sometimes they are all the same JSON structure, so this provides a way to map from the space to a shared name for the type. The IDR is formed from this, tweet://twitter/timeline#12523634216.

profileIds

    "profileIds" : ["id_str", "screen_name"],

Every connector that has a social profile for the owner can return an auth.profile object, but it also needs to identify which fields in it represent the unique identifiers for the owner. Some services (like twitter) have multiple. This may get refactored out into a profiles collection at some point :)

strip

    "strip" : {
        "contact":["status"]
    },

Sometimes the source JSON has transient stuff in it we don't want to track, this is a simple way to strip it out. It's also OK to just do this work inside the synclet code itself.

synclets

    "synclets":[
        {"name": "self", "frequency": 7200},
        {"name": "friends", "frequency": 3600},
        {"name": "tweets", "frequency": 600},
        {"name": "timeline", "frequency": 120, "threshold", 5},
        {"name": "mentions", "frequency": 600},
        {"name": "related", "frequency": 3600},
        {"name": "tweet"}]

This is the array of all the worker tasks. By default the synclet manager looks for a (name).js, but custom scripts can be used by setting a "run" field (config from stdin, data back via stdout). They will be run approx every frequency, but if they return small amounts of new data they may be run less often, up to the threshold multiplier (default 10) of their frequency. All of the data returned is compared to past versions of the data (if any) based on it's id, and may trigger new/updated events.

They are all run in order the first time (after a successful auth) so that any config can be passed between them, and no two are allowed to be run at the same time.

If no frequency is given, it will never be run. This is used experimentally for "post" synclets, where a POST to /Me/twitter/tweet of a JSON body will be queued/fed to the tweet synclet.