Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run scrapers against live data sources #1059

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Run scrapers against live data sources #1059

wants to merge 7 commits into from

Commits on Nov 17, 2024

  1. Optionally disable all HTTP request mocks

    @tillprochaska
    tillprochaska committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    73e0ffe View commit details
    Browse the repository at this point in the history

  2. Run scraper tests against live data sources every Tuesday night

    @tillprochaska
    tillprochaska committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    52120e1 View commit details
    Browse the repository at this point in the history

  3. Update procedure scraper URLs 

    This isn’t a breaking change in practice as the old URLs still work and redirect to the new URLs.
    @tillprochaska
    tillprochaska committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    3e72aa8 View commit details
    Browse the repository at this point in the history

  4. Fix extraction of geographic areas after OEIL markup change

    @tillprochaska
    tillprochaska committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    b10164e View commit details
    Browse the repository at this point in the history

  5. Fix group memberships test so that it can run against live data source 

    This test was using an outdated fixture. As the 9th plenary term is over, there are no ongoing group memberships anymore and all group memberships now have an end date. I've updated the test to use Markus Weber's group memberships from the current term. We will have to update the test when something changes (or for the 11th term the latest), but that should be manageable.
    @tillprochaska
    tillprochaska committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    1d91dcf View commit details
    Browse the repository at this point in the history

  6. Allow some tests to always mock HTTP requests, even when request mock… 

    …s are disable globally
    @tillprochaska
    tillprochaska committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    1adbaac View commit details
    Browse the repository at this point in the history

  7. Update members scraper so that it can run against live data sources 

    The test fixture used in this case was different from the live source. I've replaced it with a copy of the original source.
    @tillprochaska
    tillprochaska committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    a47a129 View commit details
    Browse the repository at this point in the history