Skip to content

Commit

Permalink
Fix jore3-importer logic related to validity period inconsistencies b…
Browse files Browse the repository at this point in the history
…etween

transformed lines and routes.

In the Jore3 data model, the line itself (the `network_lines` table in the
internal database of jore3-importer) has no validity period. Instead, the line-
specific headers (`network_line_headers` table) have validity periods that do
not overlap (within one line). Originally, it was thought that a separate line
instance would be created for each line header into the Jore4 database.
However, it appears that Jore3 route directions (`network_route_directions`
table) may overlap with multiple line headers. This causes a transformation
problem, because in Jore4 the validity period of a route MUST NOT extend
further into the past or future than the line to which the route belongs.

Basically, there are two options to solve the problem:

(1) For each line in Jore3 (and all related line headers), create only single
line in Jore4.
(2) Group and join line headers together by validity periods in such a way that
no (transformed) route overlaps multiple line instances transformed to Jore4.

ATM, we have chosen the option (2) to import as much (version) data as possible
into Jore4. In the SQL, we group the line headers into temporally continuous
clusters. The last line header of each cluster is given a Jore4 ID that can be
referenced by routes.

In addition, it has been found that in some cases the validity period of the
latest line header (for a single line) does not extend as far into the future
as an associated route direction, causing database constraint violations during
the route export stage if no date corrections are made. Therefore, for each
line, the latest validity date of the route directions associated with it is
determined, and if necessary, it is used as the end date of the last header of
the line, if it is later than what is originally in the header.

Schema changes:
(i) Add Jore4 ID to the `network_line_headers` (and the corresponding history)
table.
(ii) Remove Jore4 ID from the `network_lines` (and corresponding history)
table.

Resolves HSLdevcom/jore4#1659
  • Loading branch information
jarkkoka committed Jan 25, 2024
1 parent 100dcfb commit 5a09bc5
Show file tree
Hide file tree
Showing 39 changed files with 1,035 additions and 756 deletions.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 5a09bc5

Please sign in to comment.