Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: GTFS feed import #1013

Merged
merged 36 commits into from
Oct 18, 2024
Merged

feat: GTFS feed import #1013

merged 36 commits into from
Oct 18, 2024

Commits on Sep 5, 2024

  1. feat: Create tables for GTFS-static archive import (#1006) 

    * feat: Create tables for GTFS-static archive import

* Clean up comments
    @jzimbel-mbta
    jzimbel-mbta authored Sep 5, 2024
    Configuration menu
    Copy the full SHA
    b023be3 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Fixes and adjustments to the gtfs_* migrations

    @jzimbel-mbta
    jzimbel-mbta committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    799e4e9 View commit details
    Browse the repository at this point in the history

  2. Add Ecto schemas for all GTFS-imported tables

    @jzimbel-mbta
    jzimbel-mbta committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    5cda8b4 View commit details
    Browse the repository at this point in the history

  3. Add logic to read from GTFS archive saved at local path and import da… 

    …ta into gtfs_* tables
    @jzimbel-mbta
    jzimbel-mbta committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    da71fb2 View commit details
    Browse the repository at this point in the history

  4. Make GTFS import function more communicative

    @jzimbel-mbta
    jzimbel-mbta committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    e7832e7 View commit details
    Browse the repository at this point in the history

  5. Add /import-GTFS endpoint

    @jzimbel-mbta
    jzimbel-mbta committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    4972ace View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. tweak: Use Repo.insert_all, defer constraint checking

    @jzimbel-mbta
    jzimbel-mbta committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    8a9dfb4 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Use Postgres COPY for larger tables; Undo all superfluous deferrable … 

    …constraints; define and use Importable behaviour
    @jzimbel-mbta
    jzimbel-mbta committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    a0aad60 View commit details
    Browse the repository at this point in the history

  2. Remove now-unused create_deferrable migration helper

    @jzimbel-mbta
    jzimbel-mbta committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    f3eb075 View commit details
    Browse the repository at this point in the history

  3. Appease credo, except for one TODO that still needs DOing

    @jzimbel-mbta
    jzimbel-mbta committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    87bae04 View commit details
    Browse the repository at this point in the history

  4. fix: increase length to 100MB for multipart uploads in Plug.Parsers (#… 

    …1014)

* fix: increase length to 100MB for multipart uploads in Plug.Parsers

* ci: upgrade Github Action upload-artifact to fix deprecation error
    @meagharty
    meagharty authored Sep 26, 2024
    Configuration menu
    Copy the full SHA
    6fb7312 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Tweak service/calendar related tables and schemas

    @jzimbel-mbta
    jzimbel-mbta committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    e614628 View commit details
    Browse the repository at this point in the history

  2. Remove TODO comment that is done

    @jzimbel-mbta
    jzimbel-mbta committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    2de6323 View commit details
    Browse the repository at this point in the history

  3. Update test for service / calendar / calendar_dates tables

    @jzimbel-mbta
    jzimbel-mbta committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    7348be9 View commit details
    Browse the repository at this point in the history

  4. Appease credo

    @jzimbel-mbta
    jzimbel-mbta committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    1da6406 View commit details
    Browse the repository at this point in the history

  5. Keep GTFS timestamps as strings throughout; provide utility fns to co… 

    …nvert timestamps to other representations
    @jzimbel-mbta
    jzimbel-mbta committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9d355d2 View commit details
    Browse the repository at this point in the history

  6. Use Ecto.Enum for GTFS enum fields; convert corresponding CSV fields … 

    …to int before casting
    @jzimbel-mbta
    jzimbel-mbta committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    119887a View commit details
    Browse the repository at this point in the history

  7. Appease dialyzer

    @jzimbel-mbta
    jzimbel-mbta committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9dd52f1 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Change how int-code and string enum values are defined in the DB

    @jzimbel-mbta
    jzimbel-mbta committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    c74b271 View commit details
    Browse the repository at this point in the history

  2. Oops, forgot to commit auto-generated structure.sql in prev commit

    @jzimbel-mbta
    jzimbel-mbta committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    ed237f6 View commit details
    Browse the repository at this point in the history

  3. structure.sql was out of date somehow, even though migrations were al… 

    …ready up??? Fixed now
    @jzimbel-mbta
    jzimbel-mbta committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    2a83ffd View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into jz-import-gtfs-endpoint--ecto-schemas

    @jzimbel-mbta
    jzimbel-mbta committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    34c7f2d View commit details
    Browse the repository at this point in the history

  5. Update mix.lock

    @jzimbel-mbta
    jzimbel-mbta committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    4c205b5 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Simplify some execute/2 calls in GTFS migrations

    @jzimbel-mbta
    jzimbel-mbta committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    fd60b41 View commit details
    Browse the repository at this point in the history

  2. Install Oban

    @jzimbel-mbta
    jzimbel-mbta committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    ad44969 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Add GTFS import Oban workers; Add logic for job-based GTFS import

    @jzimbel-mbta
    jzimbel-mbta committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    6607e6b View commit details
    Browse the repository at this point in the history

  2. Update/add API endpoints for async GTFS import

    @jzimbel-mbta
    jzimbel-mbta committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    7b0660a View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Remove Expires header from S3 put_object requests, it doesn't do wh… 

    …at I thought it did
    @jzimbel-mbta
    jzimbel-mbta committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    b793df0 View commit details
    Browse the repository at this point in the history

  2. Add more comprehensive logging for GTFS import process

    @jzimbel-mbta
    jzimbel-mbta committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    f847d4a View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into jz-import-gtfs-endpoint--ecto-schemas

    @jzimbel-mbta
    jzimbel-mbta committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    d8d4606 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Do not defer optionally-deferrable constraints during GTFS import/val… 

    …idation
    @jzimbel-mbta
    jzimbel-mbta committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    10f7267 View commit details
    Browse the repository at this point in the history

  2. Set queue_target and queue_interval config for DB connection

    @jzimbel-mbta
    jzimbel-mbta committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    40de6b6 View commit details
    Browse the repository at this point in the history

  3. Fix duplicate import/validation job handling

    @jzimbel-mbta
    jzimbel-mbta committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    08dac72 View commit details
    Browse the repository at this point in the history

  4. Add /api/gtfs/check_jobs endpoint

    @jzimbel-mbta
    jzimbel-mbta committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    1b0ddd1 View commit details
    Browse the repository at this point in the history

  5. fix: Use correct module for checking validation jobs

    @jzimbel-mbta
    jzimbel-mbta committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    511b278 View commit details
    Browse the repository at this point in the history

  6. Fix route

    @jzimbel-mbta
    jzimbel-mbta committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    06a6d6f View commit details
    Browse the repository at this point in the history