Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updating time on csv output #818

Merged
merged 1 commit into from
Aug 6, 2024

Conversation

AminTorabi-NOAA
Copy link
Contributor

Update on csv output results. column t0 and time added to just show current_time
The output is like below
image

Additions

Removals

Changes

Testing

Screenshots

Notes

Todos

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Visually tested in supported browsers and devices (see checklist below 👇)
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

  • Windows
  • Linux
  • Browser

Accessibility

  • Keyboard friendly
  • Screen reader friendly

Other

  • Is useable without CSS
  • Is useable without JS
  • Flexible from small to large screens
  • No linting errors or warnings
  • JavaScript tests are passing

@AminTorabi-NOAA AminTorabi-NOAA merged commit 5865910 into NOAA-OWP:master Aug 6, 2024
4 checks passed
@aaraney
Copy link
Member

aaraney commented Aug 7, 2024

Unfortunately, this is a breaking change for software that uses t-route output (e.g. calibration) as the csv "schema" has changed. That's not a huge deal and is an easy fix on our end, however if I didn't see this PR then I wouldn't have known calibration now has a bug. At this stage I fully expect that breaking changes will be introduced from time to time, but I would like to figure out a solution that works well for the t-route team and downstream software / users of t-route output (in all forms) to be better prepared for future changes.

I know some groups use message boards or email lists to communicate these kinds of changes, but in reality, sometimes (at least for me) these things slip during development and communication breaks down. Alternatively, it would be ideal if there were a way for downstream software to detect if there were a change and t-route tests run as github actions to detect changes. This is more or less what we do in the calibration code now to check if the csv header looks a certain way. Ideally though, we could check a version field that is always present to determine how to process the file. csv doesn't really have a great way for providing what we might think of as metadata where other formats like hdf5 or netcdf do. As for t-route tests, it could be as simple as after running t-route in a github action checking that the output headers are correct for associated version.

Of course there is a discussion to be had about how to best implement this sort of feature for different formats and how to document that in t-route's docs. My initial thought for the csv format is to add a version field and include the version as the value for each row of that field. That will increase the output file size by roughly 4 bytes per output row if you were to use a version format like 1.0. Its not ideal, but my intuition is that the csv format is not intended for deployment environments.

I might be getting ahead of myself, but I think the version only needs to contain a single decimal (e.g. 1.0). The ones place (aka major) would be increased when an existing field is removed or an existing field's format changes (e.g date field was 2020-01-01 00:00 and is now 2020-01-01T00:00). The tenths place (aka minor) would be increased when a field is added. For some formats depending on how they work all changes may need to be a ones place (major) version bump.

@shorvath-noaa
Copy link
Contributor

Apologies for the downstream impacts! I like the idea of adding github actions tests to verify that changes like this will not have unforeseen impacts. In general though, I think we can be more cognizant and communicative about changes to t-route's inputs or outputs since those are what would most likely impact users.

Changes to inputs/outputs shouldn't need to happen much going forward. The only changes would be in response to user feedback (as this one was). But I guess the output format can't be perfect for everyone, so some more care should be taken in updating these file formats and contents...

@aaraney
Copy link
Member

aaraney commented Aug 7, 2024

All good! Just part of writing software that people use!

Changes are okay and expected, just looking to minimize what could fall through the cracks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants