Configuring qbank

Introduction

To run qbank, you need to configure the dlkit stack and decide if you are using the filesystem or MongoDB to persist data. The filesystem option is for light-weight, offline deployments on older CLIx PC's (running Windows XP), and that is how the master branch is configured.

Config files

The dlkit configs are in the dlkit_configs directory. You can see three files:

• configs.py
• handcar_configs.py
• registry.py

configs.py

This file defines the dlkit stack: which layers are activated and how they behave. Each object defines a layer (i.e. FILESYSTEM_ADAPTER_1) and the various parameters used by that layer. I'll describe one layer here as an example, before describing the structure of this file.

Dissecting an Adapter Config

To better understand what each configuration means, we'll take a closer look at one, the FILESYSTEM_ADAPTER_1 definition. This adapter intercepts all REPOSITORY calls and when a data blob is set on an assetContent, saves that data blob to the local filesystem. It also intercepts all get_url calls and generates a streamable URL back to the consumer (instead of just the location on file space).

FILESYSTEM_ADAPTER_1 = {
    'id': 'filesystem_adapter_configuration_1',
    'displayName': 'Filesystem Adapter Configuration',
    'description': 'Configuration for Filesystem Adapter',
    'parameters': {
        'implKey': impl_key_dict('filesystem_adapter'),
        'repositoryProviderImpl': {
            'syntax': 'STRING',
            'displayName': 'Repository Provider Implementation',
            'description': 'Implementation for repository service provider',
            'values': [
                {'value': 'JSON_1', 'priority': 1}
            ]
        },
        'dataStorePath': {
            'syntax': 'STRING',
            'displayName': 'Path to local filesystem datastore',
            'description': 'Filesystem path for setting the MongoClient host.',
            'values': [
                {'value': DATA_STORE_PATH, 'priority': 1}
            ]
        },
        'secondaryDataStorePath': {
            'syntax': 'STRING',
            'displayName': 'Path to local filesystem datastore',
            'description': 'Filesystem path for setting the MongoClient host.',
            'values': [
                {'value': STUDENT_RESPONSE_DATA_STORE_PATH, 'priority': 1}
            ]
        },
        'dataStoreFullPath': {
            'syntax': 'STRING',
            'displayName': 'Full path to local filesystem datastore',
            'description': 'Filesystem path for setting the JSONClient host.',
            'values': [
                {'value': ABS_PATH, 'priority': 1}
            ]
        },
        'urlHostname': {
            'syntax': 'STRING',
            'displayName': 'Hostname config for serving files over the network',
            'description': 'Hostname config for serving files.',
            'values': [
                {'value': 'https://qbank-clix-dev.mit.edu/api/v1', 'priority': 1}
            ]
        },
    }
}

Note that the object is called FILESYSTEM_ADAPTER_1. This object can then be referenced in other objects, to say "in this instance, use the FILESYSTEM_ADAPTER_1 code".

Let's break this object down further.

implKey (required)

In this case we have an implKey value of filesystem_adapter. In registry.py, you'll see near the bottom, a filesystem_adapter key -- so that tells dlkit which REPOSITORY manager to use, when the adapter is invoked. Every layer in configs.py needs to have an implKey value that is defined in registry.py.

repositoryProviderImpl (optional)

This points to another object in configs.py called JSON_1. What this means is that when using the REPOSITORY service, JSON_1 sits "below" FILESYSTEM_ADAPTER_1 in the stack. So the FILESYSTEM_ADAPTER_1 can modify stuff going down before it hits JSON_1, and it can modify stuff going up after JSON_1 returns the data.

While *ProviderImpl is an optional field for all configs, you'll see it appear in all of the "adapter" configs. This is because if you plan to use the X service (i.e. REPOSITORY) below that layer, you need to specify which provider / service sits below the adapter. In this way you can stack adapters on top of each other to suit your project needs. You can also do this on a service-by-service basis, like:

the various dataStorePath values (required & optional)

The FILESYSTEM_ADAPTER_1 uses several parameters to indicate where files are saved to on the filesystem. These parameters give the relevant paths. dataStorePath and dataStoreFullPath are required, but secondaryDataStorePath is optional, if you want response files duplicated somewhere. Note that these should be full paths, not relative.

urlHostname (required)

When get_url() is called on an assetContent, the adapter will generate a "streamable" url based off of this value. It will append /repositories/repository/:repositoryId/assets/:assetId/contents/:assetContentId/stream to whatever value appears here. This way you can refer to a specific domain or just provide a relative path.

General configs.py description

Back to the configs.py description. It is divided into two sections, one for production and one for testing. I will describe the production set, but the testing set follows the same logic.

BOOTSTRAP and SERVICE are required for the dlkit runtimes to work properly. SERVICE is the top-most layer in the dlkit stack, and the one that all consumer calls will interact with. After that, it depends on how you want dlkit to behave. Each *ProviderImpl must point to an adapter or persistence layer defined in configs.py.

handcar_configs.py

For this version of qbank, because it does not talk to the MC3 Handcar service, this file is not used.

registry.py

This file maps different configurations to the appropriate dlkit managers. You should not have to edit this file, unless you want to add in / access new dlkit packages that come out in future versions. In general, if you want to use a certain implementation and service, that combination has to be defined in registry.py. For example, if you want to use the REPOSITORY service in json, then those managers have to be defined in the json object, REPOSITORY values.

Filesystem or MongoDB

dlkit's json implementation can persist the data either in MongoDB or in json files stored on filespace. Either configuration can be used in conjunction with the FILESYSTEM_ADAPTER_1, since that only manages the storage of media asset files (not the dlkit data associated with it).

To configure dlkit to talk to either the filesystem (json files stored on filespace) or MongoDB, toggle the following flag in JSON_1:

        'useFilesystem': {
            'syntax': 'BOOLEAN',
            'displayName': 'Use the filesystem instead of MongoDB',
            'description': 'Use the filesystem instead of MongoDB',
            'values': [
                {'value': True, 'priority': 1}
            ]
        },

Note that even if the value is set to False, you can still use the FILESYSTEM_ADAPTER_1, because the adapter only handles where media files are stored, not where the data persists.

By default, if this parameter is not present, dlkit will use MongoDB.

Caching

To improve dlkit performance when heavy authorization or hierarchies are present, you can use memcached and cache some data. This is set with the following flag, in JSON_1:

        'useCachingForQualifierIds': {
            'syntax': 'BOOLEAN',
            'displayName': 'Flag to use memcached for authz qualifier_ids or not',
            'description': 'Flag to use memcached for authz qualifier_ids or not',
            'values': [
                {'value': False, 'priority': 1}
            ]
        },

For deployments where memcached is not available, this should be set to False, otherwise dlkit will throw an exception.