OSF SciNet -- Crowd-Sourcing the Scientific Citation Network

What is SciNet?

SciNet is a open source, collaborative project being developed at the Center for Open Science. The primary purpose of Scholarly is to provide the public with a free, open, and comprehensive dataset containing meta-data for academic citations as well as corresponding references. This dataset will allow the public to access, analyze, and distribute academic citation meta-data without restriction.

Relevant Wiki Pages:

• Contributing -- Working with Git branches

• API Call Examples

• Resources

Getting started with SciNet

Note: This set of packages is not production ready, but it should be robust enough for some early development. Additionally, these instructions were prepared using a Digial Ocean server running Ubuntu 12.04 but they can easily be adapted for running locally or on another cloud service.

1. Create your droplet: http://digitalocean.com

2. Check your email and make note of: IP Address of your droplet Username created for your droplet Password for your user

3. SSH into your droplet:

4. Open up a terminal and type:

    ssh <username>@<ip address>
   

   enter the password located in the email

5. Set up a local (non-root) account:

6. Create the user account:

    adduser <username>
   

7. Give it sudo access:

    visudo
    # Below "root ALL=(ALL:ALL) All" enter
    <username> ALL=(ALL:ALL) ALL
    # Save the file and exit
   

8. Logout of your current account and SSH into the server with the new one:

9. From your terminal enter:

    exit
    ssh <username>@<ip address>
   

10. Create a directory to hold our data:

11. From your terminal enter:

     sudo mkdir /vol
     sudo chown <username>:<username> /vol
     cd /vol
    

12. Install necessary packages (Note: This can take a minute):

13. From your terminal enter:

     sudo apt-get update
     sudo apt-get install git python-flask python-pip build-essential python-dev python-requests
     sudo apt-get install python-pyquery uwsgi-plugin-python uwsgi nginx
     sudo pip install pymongo reppy nameparser uwsgi
    

14. Clone and install SciNet:

15. Checkout the repo:

     # Unless you have a forked SciNet use: https://github.com/centerforopenscience/scinet.git
     git clone https://github.com/user_name/repo.git
    

16. Install required libraries:

     cd /vol/scinet
     sudo pip install -r requirements.txt
    

Note: If you are installing on Ubuntu, remove the lxml requirement from requirements.txt before running pip install.

1. Test it:

    cd /vol/scinet/scinet
    python main.py
   

You should get a pymongo "Connection Refused" error.

1. Install and setup mongodb:

2. Add the 10gen repo:

    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
    echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/10gen.list
    sudo apt-get update
    sudo apt-get install mongodb-10gen
   

3. Edit /etc/mongodb.conf and change dbpath=/var/lib/mongodb to dbpath=/vol/mongodb, and update permissions:

    sudo mkdir /vol/mongodb
    sudo chown -R mongodb /vol/mongodb
   

4. Restart mongo:

    sudo service mongodb restart
   

At this point, you should be able to (optionally) restart the instance and run:

    python /vol/scinet/scinet/main.py

without errors.

1. Get Citelet:

2. Checkout the repo:

    cd /vol
    # Unless you have forked scinet-citelet use: https://github.com/centerforopenscience/scinet-citelet.git
    git clone https://github.com/user_name/repo.git
   

3. Install required libraries::

    cd /vol/citelet
    sudo pip install -r requirements.txt
   

4. Build Citelet:

5. Verify config file is pointing to your desired urls:

    vi /vol/citelet/cfg.py
   

For more information visit the Citelet docs.

1. Run the fabric build process:

    fab rsudo deploy
   

2. Test it:

    cd /vol/citelet/app
    python main.py
   

3. Setup boto config so our keys will be loaded automatically:

4. sudo vi /etc/boto.cfg:

    [Credentials]
    aws_access_key_id = <your access key>
    aws_secret_access_key = <your secret access key>
   

5. Install and setup NGINX and uWSGI:

6. Configure NGINX, for example, replace /etc/nginx/sites-available/default with:

    server {
        listen   80;
              server_name scinet;
          # scinet endpoint
            location /scinet {
                uwsgi_pass unix:///tmp/scinet.sock;
            include uwsgi_params;
            # strip path before handing it to app
            uwsgi_param SCRIPT_NAME /scinet;
            uwsgi_modifier1 30;
        }
        # citelet endpoint
            location /citelet {
                uwsgi_pass unix:///tmp/citelet.sock;
            include uwsgi_params;
            # strip path before handing it to app
            uwsgi_param SCRIPT_NAME /citelet;
            uwsgi_modifier1 30;
        }
    }
   

7. Finally symlink it to its sites-enabled folder:

    sudo ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/default
   

8. Configure UWSGI sockets, for example, create /etc/uwsgi/apps-available/scinet.ini and populate it with:

    [uwsgi]
    chdir = /vol/scinet/scinet
    uid = www-data
    gid = www-data
    chmod-socket = 666
    socket = /tmp/scinet.sock
    module = app
    callable = app
   

9. And, create /etc/uwsgi/apps-available/citelet.ini and populate it with:

    [uwsgi]
    chdir = /vol/citelet/app
    uid = www-data
    gid = www-data
    chmod-socket = 666
    socket = /tmp/citelet.sock
    module = main
    callable = app
   

10. Finally symlink them to their respective enabled folders:

     sudo ln -s /etc/uwsgi/apps-available/scinet.ini /etc/uwsgi/apps-enabled/scinet.ini
     sudo ln -s /etc/uwsgi/apps-available/citelet.ini /etc/uwsgi/apps-enabled/citelet.ini
    

11. Enable the app and restart:

     sudo service nginx restart
     sudo service uwsgi restart
    

The site should now be up and running. You can, for instance, install lynx and visit Test the site by visiting, for example, http:///scinet -or- http:///citelet.

1. Still on the to-do list:

• Pushing the site via git, with automatic server restarts, etc., implemented as git post commit hooks.
• Automating the server setup process via the tools that Jeff and Lindsy were talking about.