RSpec::Fortify

RSpec::Fortify is a hard fork of rspec-retry

RSpec::Fortify adds a :retry option for intermittently failing rspec examples. If an example has the :retry option, rspec will run the example the specified number of times until the example succeeds.

Installation

Add this line to your application's Gemfile:

gem 'rspec-fortify', group: :test # Unlike rspec, this doesn't need to be included in development group

And then execute:

$ bundle

Or install it yourself as:

$ gem install rspec-fortify

require in spec_helper.rb

# spec/spec_helper.rb
require 'rspec/fortify'

RSpec.configure do |config|
  # run retry only on features
  config.around :each, :js do |ex|
    ex.run_with_retry retry: 3
  end

  # callback to be run between retries
  config.retry_callback = proc do |ex|
    # run some additional clean up task - can be filtered by example metadata
    if ex.metadata[:js]
      Capybara.reset!
    end
  end
end

Usage

it 'should randomly succeed', :retry => 3 do
  expect(rand(2)).to eq(1)
end

it 'should succeed after a while', :retry => 3, :retry_wait => 10 do
  expect(command('service myservice status')).to eq('started')
end
# RSpec::Fortify: 2nd try ./spec/lib/random_spec.rb:49
# RSpec::Fortify: 3rd try ./spec/lib/random_spec.rb:49

Calling run_with_retry programmatically

You can call ex.run_with_retry(opts) on an individual example.

Configuration

• :verbose_retry(default: false) Print retry status
• :display_try_failure_messages (default: false) If verbose retry is enabled, print what reason forced the retry
• :default_retry_count(default: 1) If retry count is not set in an example, this value is used by default. Note that currently this is a 'try' count. If increased from the default of 1, all examples will be retried. We plan to fix this as a breaking change in version 1.0.
• :default_sleep_interval(default: 0) Seconds to wait between retries
• :clear_lets_on_failure(default: true) Clear memoized values for lets before retrying
• :exceptions_to_hard_fail(default: []) List of exceptions that will trigger an immediate test failure without retry. Takes precedence over :exceptions_to_retry
• :exceptions_to_retry(default: []) List of exceptions that will trigger a retry (when empty, all exceptions will)
• :retry_callback(default: nil) Callback function to be called between retries
• :retry_on_failure(default: main? || pr?) Retry examples on failure. This is useful for flaky tests that are not marked with :retry metadata.
• :retry_on_failure_count(default: 2) Run examples on failure this many times.
• :retry_on_success(default: pr? && changed_specs.size < 30) Retry examples on success. This is useful in order to prove that new tests are not flaky.
• :retry_on_success_count(default: 10) Run examples on success this many times.

Environment Variables

• RSPEC_FORTIFY_RETRY_COUNT can override the retry counts even if a retry count is set in an example or default_retry_count is set in a configuration.
• CHANGED_SPECS can be set to a comma-separated list of spec files that have changed. This is used to determine if an example should be retried on success.
• CI is used to determine if the current environment is a CI environment. This is used to determine if examples should be retried on success.
• CIRCLE_PULL_REQUEST is used to determine if the current CI build is a pull request or a default branch build. This is used to determine if examples should be retried on success.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a pull request