Writing & Managing Tests for Autolab

Overview

Autolab uses RSpec and capybara for its front-end tests. RSpec is the general framework for testing Rails applications, and capybara is especially useful for acceptance tests by simulating user's browser actions (such as clicking a link). Also, Travis CI has been set up for the front-end repository, such that the specs are verified for any new commits and pull requests.

The test-related files are located under spec/, as listed below:

• spec_helper.rb, rails_helper.rb: These two files are loaded every time rspec is run. What they do is to include testing libraries and set up global configurations.
• support/*: Files in this directory contains helper methods that can be called throughout the testing routine, such as login_user and get_course_id_by_uid.
• controllers/*: Specs in this directory are unit tests of controller methods -- it's very useful for testing if a page renders successfully with different user groups.
• features/*: Specs in this directory are mainly acceptance tests, where larger tasks (such as Autograding roundtrip) are tested as a whole.
• factories/*: Files in this directory use FactoryGirl to define new Rails objects, so that test developers do not need to instantiate objects in specs.

Adding New Tests

Whenever a developer writes a new feature, the corresponding tests need to be added. Currently, tests should cover any methods that appear in the routes with GET method, for example courses#index. Each controller has its own specs file, named as spec/#{CONTROLLER_NAME}_spec.rb; if you created a new controller, you will need to add the corresponding spec file, and write the stub as follows:

require "rails_helper"

RSpec.describe XXXController, type: :controller do
  render_views
end

Within each controller spec, there are two levels of scopes, i.e. describe and context. describe is often named as #method_name, which contains tests of a controller method in different situations. Within the describe block, each context is named with the description of the situation, for example "when user is Autolab admin". Eventually, use it to describe the desired outcome in the situation, such as it "renders successfully". An example is as follows (note that Lines 5-7 are methods in Rack::test):

describe "#show" do
  context "when user is Autolab admin" do
    login_admin
    it "renders successfully" do
      get :show
      expect(response).to be_success
      expect(response.body).to match(/Admin Autolab/m)
    end
  end
end

Maintaining Existing Tests

Anyone that submits a pull request needs to make sure that all specs pass. Before submitting a pull request and letting Travis CI do all the work, developers should run the specs locally by using rspec and make sure all tests pass.

If changes in the code cause some specs to fail, the developer is responsible for fixing the corresponding test to reflect the change. Since Autolab uses Travis CI for continuous integration & deployment, any pull request or commit would trigger a new Travis build, and the pull request reviewer can see its status next to the "Merge PR" button.

If the build fails, one can follow the link on the pull request page to see which test(s) have failed. Questions about the "build failure" message or why a test failed? Simply ping the team.

Resources

To learn more about RSpec, check out the RSpec Guide by Relish.

To learn more about Capybara, check out its Github Page and Official API Page.

To learn more about Travis CI, check out its documentation page.