The Folk RNN web app is in two parts: a composer for users to generate tunes using folkrnn, and an archiver to host and tweak them (and any other machine co-authored tunes).
The web app is written in Python 3 using the Django framework. It uses the Channels architecture to handle websocket connections and a task queue. It uses the django-hosts middleware to route the composer and archiver apps on separate domains.
- Managed via vagrant
- OS: Ubuntu 16.04 LTS
- This ships with Python 3.5
- To run on your laptop you need to install
- vagrant
- virtualbox
- ...and nothing else, it's all contained and automatically set up within the VM.
- To deploy on a remote server, it's as above but we swap Linode's VPS service for VirtualBox.
- vagrant
- vagrant linode plugin
Create a directory on your host machine. Clone into that the following three repositories. Note the soundfont repository is optional, MIDI functionality will simply be hobbled without it.
- this folk-rnn-webapp, ie.
git clone https://github.com/tobyspark/folk-rnn-webapp.git - the folk-rnn library, ie.
git clone https://github.com/tobyspark/folk-rnn.git - the midi-js-soundfonts resources, ie.
git clone https://github.com/gleitz/midi-js-soundfonts.git
You should have three folders like so
some-directory
├── folk-rnn
├── folk-rnn-webapp
└── midi-js-soundfonts
Note the folk-rnn repository used here is a fork of the original folk-rnn project that provides an API to the generate functionality and has an orders-of-magnitude faster implementation of the generate code.
In a shell, navigate to the folk-rnn-webapp directory, and issue the following commands
vagrant up
vagrant ssh
/folk_rnn_webapp/runserver dev
On first run, vagrant up will download Ubuntu 16.04, create a VM and install the OS, and then run the configuration tasks, all as prescribed by the Vagrantfile.
Thereafter, vagrant up will boot the machine, and vagrant halt will shut it down.
To get in, vagrant ssh and you should see your shell prompt change to ubuntu@ubuntu. Type logout or <ctrl-d> to close the ssh connection.
Once you have issued the runserver command, navigate to http://127.0.0.1:8000 in your browser as it tells you.
Note: Using the localhost IP address as above will route to the default app (currently the composer). To mimick real-world use and be able to access both apps typing e.g. http://themachinefolksession.org.local:8000, add the following to your /etc/hosts file.
127.0.0.1 folkrnn.org.local
127.0.0.1 themachinefolksession.org.local
Note: if the specification for the VM changes, e.g. what's in the Vagrantfile, those changes will need to be retrospectively applied to any existing machines. Either vagrant reload --provision the existing machine, or vagrant destroy; vagrant up to create a fresh machine to this spec.
A production install should automatically backup the database and key files to a cloud service.
A django management command is included to apply this backup to a local machine, i.e. follow the above procedure with this.
vagrant up
vagrant ssh
python3.6 /folk_rnn_webapp/folk_rnn_site/manage.py applybackup
vagrant up
vagrant ssh
cd /folk_rnn_webapp/folk_rnn_site/
pytest
Before starting you will need
- Linode API key
- ssh key configured
You will also want a sibling folk_rnn_webapp repository to keep dev and production separate (also a requirement of vagrant, which can’t handle a virtualbox VM and a remote Linode server simultaneously). So clone a fresh repo in addition to the above, adding a production suffix. You should have four folders like so
some-directory
├── folk-rnn
├── folk-rnn-webapp
├── folk-rnn-webapp-production
└── midi-js-soundfonts
In a shell, navigate to the folk-rnn-webapp-production directory.
Create an environment file to configure the app for production. Create an .env file using template.env.
Issue the following commands
export LINODE_API_KEY=<your key here>
export FOLKRNN_PRODUCTION=foo
vagrant up --provider=linode
After some time, the server should be deployed and the webapp running (see note below).
As before, vagrant ssh to log in to the server, and vagrant provision to update.
Note: there is a bug in vagrant up for the first time, the provisioning scripts that should run unprivileged instead run as root. However, rather than vagrant destroy; vagrant up to rebuild a machine, do this: vagrant halt, pause, vagrant rebuild, pause, vagrant provision. This will keep the IP address and provision correctly. So for a new server, do the rebuild dance and all should be good (alternatively, chown vagrant:vagrant the directory /var/opt/folk_rnn_task/tune and the contents of /folk_rnn_static).
Browse to <composer url>/dataset to download a CSV of all the tunes generated by the composer app.
Browse to <archiver url>/dataset to download a CSV of all the tunes and settings contributed to the archiver app.
On a host machine, run the following django management command python3.6 /folk_rnn_webapp/folk_rnn_site/manage.py stats to view i. a tune-centric view of activity, ii. a composer-session-centric view of activity, and iii. descriptive statistics suitable for academic write-up.
All datasets are anonymous.
The RNN models used by the composer app are re-packaged versions of the pickle files used by folk-rnn. They contain meta-data such as the display name for the model, and are in Python 3 format.
The packaging process happens automatically on vagrant provision, and the webapp should dynamically load all relevant settings from the pickles on app intialisation. For example, mode and meter keys are extracted from the model’s tokens.
To include a new model, first update your copy of the folk-rnn library to include the model, i.e. if you are training the model elsewhere, find the folk-rnn directory alongside folk-rnn-webapp, and place the model file pkl in the metadata folder. With this done, the model will be copied onto the host machine on provisioning.
With the new model in place, edit the webapp provisioning script to import the model into the webapp, and provision:
tools/create_model_from_config_meta.py
The script starts with a list of models to include, as tuples structured as (folk-rnn-metadata-pickle, save-as-name, display-name, default_meter, default_mode, default_tempo) e.g.
(
'/folk_rnn/metadata/config5-wrepeats-20160112-222521.pkl',
'thesession_with_repeats',
'thesession.org (w/ :| |:)’,
'4/4',
'Cmaj',
120,
)
For models yet to be developed, it is recommended to normalise to L:1/8 and include info-fields in header and body as separate tokens. The code here has been adapted to handle a model without these, i.e. a corpus with L, M, K headers, and those headers may be [L:x/y] rather than L:x/y. Models such as these (e.g. Swedish) need a tool run to generate further metadata for the folkrnn.org model. See tools/analyse_l_for_m.py, and how the resulting information is applied in the create_model_from_config_meta script.