Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Jore3-Importer export logic for conflicting validity periods between lines and route #137

Merged
merged 2 commits into from
Jan 25, 2024

Conversation

jarkkoka
Copy link
Contributor

@jarkkoka jarkkoka commented Jan 15, 2024

Fix jore3-importer logic related to database constraint violations caused by validity period inconsistencies between transformed lines and routes.

In the Jore3 data model, the line itself (the network_lines table in the internal database of jore3-importer) has no validity period. Instead, the line-specific headers (network_line_headers table) have validity periods that do not overlap. Originally, it was thought that a separate line instance would be created for each line header into the Jore4 database. However, it appears that Jore3 route directions (network_route_directions table) may overlap with multiple line headers. This causes a transformation problem, because in Jore4 the validity period of a route MUST NOT extend further into the past or future than the line to which the route belongs.

Basically, there are two options to solve the problem:

  1. For each line in Jore3 (and all related line headers), create only single line in Jore4.
  2. Group and join line headers together by validity periods in such a way that no route overlaps multiple line instances transformed to Jore4.

In this PR, we have chosen the option (2) to import as much (version) data as possible into Jore4. In the line export SQL, we group the line headers into temporally continuous clusters. The last line header of each cluster is given a Jore4 ID that can be referenced by routes.

Resolves HSLdevcom/jore4#1659


This change is Reviewable

@jarkkoka jarkkoka requested review from Leitsi and culka and removed request for Leitsi January 15, 2024 14:33
@jarkkoka jarkkoka changed the title Fix Jore3-Importer export logic for conflicting validity periods between lines and routes in order to import more routes into Jore4 Fix Jore3-Importer export logic for conflicting validity periods between lines and route Jan 15, 2024
Copy link
Contributor

@Leitsi Leitsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 38 of 39 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @culka)

Copy link
Contributor

@culka culka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 38 of 39 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @jarkkoka)

@jarkkoka jarkkoka force-pushed the fix-line-export branch 4 times, most recently from 59336dc to 5a09bc5 Compare January 25, 2024 16:28
…etween

transformed lines and routes.

In the Jore3 data model, the line itself (the `network_lines` table in the
internal database of jore3-importer) has no validity period. Instead, the line-
specific headers (`network_line_headers` table) have validity periods that do
not overlap (within one line). Originally, it was thought that a separate line
instance would be created for each line header into the Jore4 database.
However, it appears that Jore3 route directions (`network_route_directions`
table) may overlap with multiple line headers. This causes a transformation
problem, because in Jore4 the validity period of a route MUST NOT extend
further into the past or future than the line to which the route belongs.

Basically, there are two options to solve the problem:

(1) For each line in Jore3 (and all related line headers), create only single
line in Jore4.
(2) Group and join line headers together by validity periods in such a way that
no (transformed) route overlaps multiple line instances transformed to Jore4.

ATM, we have chosen the option (2) to import as much (version) data as possible
into Jore4. In the SQL, we group the line headers into temporally continuous
clusters. The last line header of each cluster is given a Jore4 ID that can be
referenced by routes.

In addition, it has been found that in some cases the validity period of the
latest line header (for a single line) does not extend as far into the future
as an associated route direction, causing database constraint violations during
the route export stage if no date corrections are made. Therefore, for each
line, the latest validity date of the route directions associated with it is
determined, and if necessary, it is used as the end date of the last header of
the line, if it is later than what is originally in the header.

Schema changes:
(i) Add Jore4 ID to the `network_line_headers` (and the corresponding history)
table.
(ii) Remove Jore4 ID from the `network_lines` (and corresponding history)
table.

Resolves HSLdevcom/jore4#1659
@jarkkoka jarkkoka merged commit fc55a30 into main Jan 25, 2024
7 of 8 checks passed
@jarkkoka jarkkoka deleted the fix-line-export branch January 25, 2024 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix Jore3-Importer export logic for conflicting validity periods between lines and routes
3 participants