how to make sense of epochs?

[BurntSushi]

I was reviewing some of the code for this library to better understand some of the differences (in detail) between it and chrono/time. I stumbled across something that I couldn't figure out, and I wonder if you might be able to clear up my confusion.

OK, so I'm looking at this code:

hifitime/src/timescale.rs

Lines 60 to 64 in 822c0a0

	/// The UNIX reference epoch of 1970-01-01 in TAI duration, accounting only for IERS leap seconds.
	pub const UNIX_REF_EPOCH: Epoch = Epoch::from_tai_duration(Duration {
	centuries: 0,
	nanoseconds: 2_208_988_800_000_000_000,
	});

And this code:

hifitime/src/lib.rs

Lines 63 to 67 in 822c0a0

	/// The duration between J2000 and J1900: one century **minus** twelve hours. J1900 starts at _noon_ but J2000 is at midnight.
	pub const J2000_TO_J1900_DURATION: Duration = Duration {
	centuries: 0,
	nanoseconds: 3_155_716_800_000_000_000,
	};

To check my understanding, I wanted to try and write a datetime calculation that arrived at these numbers. So I wrote this program. First, Cargo.toml:

[package]
publish = false
name = "datetime"
version = "0.1.0"
edition = "2021"

[dependencies]
hifitime = "3.9.0"

[[bin]]
name = "datetime"
path = "main.rs"

And main.rs:

fn main() {
    let t1 = hifitime::Epoch::from_gregorian_utc(1900, 1, 1, 12, 0, 0, 0);
    let d = hifitime::Duration::from_parts(0, 2_208_988_800_000_000_000);
    println!("unix: {t1:?} + {d:?} = {:?}", t1 + d);

    let t1 = hifitime::Epoch::from_gregorian_utc(1900, 1, 1, 12, 0, 0, 0);
    let d = hifitime::Duration::from_parts(0, 3_155_716_800_000_000_000);
    println!("J2000: {t1:?} + {d:?} = {:?}", t1 + d);
}

The first block is meant to create a datetime corresponding to the J1900 epoch, add the number of seconds used above to get to the Unix epoch and see what comes out. I then did the same thing in the second block, but for getting from the J1900 epoch to J2000. The output of this program is:

$ cargo run -q
unix: 1900-01-01T12:00:00 UTC + Duration { centuries: 0, nanoseconds: 2208988800000000000 } = 1970-01-01T12:00:00 UTC
J2000: 1900-01-01T12:00:00 UTC + Duration { centuries: 0, nanoseconds: 3155716800000000000 } = 2000-01-02T00:00:00 UTC

My understanding here is that the output should have been this:

$ cargo run -q
unix: 1900-01-01T12:00:00 UTC + Duration { centuries: 0, nanoseconds: 2208988800000000000 } = 1970-01-01T00:00:00 UTC
J2000: 1900-01-01T12:00:00 UTC + Duration { centuries: 0, nanoseconds: 3155716800000000000 } = 2000-01-01T00:00:00 UTC

That is, the J2000 epoch should be 24 hours earlier and the Unix epoch should be 12 hours earlier.

I did a little digging, and it seems like the J2000 epoch is actually 2000-01-01T12:00:00 UTC. That is, at noon, not midnight (despite the code comment above). If that's true, then the expected output would be:

$ cargo run -q
unix: 1900-01-01T12:00:00 UTC + Duration { centuries: 0, nanoseconds: 2208988800000000000 } = 1970-01-01T00:00:00 UTC
J2000: 1900-01-01T12:00:00 UTC + Duration { centuries: 0, nanoseconds: 3155716800000000000 } = 2000-01-01T12:00:00 UTC

Which means the actual output is 12 hours ahead of both epochs (instead of being 12 hours ahead of one and 24 hours ahead of the other). What am I missing?

Thank you!

[ChristopherRabotin]

Thanks for looking into hifitime. I'm currently traveling without access to a laptop, but I'll get back to you when I return. The examples you show make me question whether something is wrong, so I want to run other tests to double check.

In general, Hifitime epochs are currently converted to TAI from any time system that they were initialized in. However, the duration operations will operate on the original time scale. For example, if an epoch was initialized in UTC near the boundary with a leap second, and ten seconds are added to that epoch, then a full ten seconds in UTC are added (ie the leap second is "uncounted").

The next version of hifitime is expected to change this behavior (although it shouldn't lead to any changes to the results of the external api). @gwbres put a lot of really good work into making the time scale change lazy. In that testing, we got all of the tests to pass apart from the TDB time scale initialization which was exactly twelve hours off, similar to what you've seen. That said, the test suite is exhaustive and I suspect that if there is an error, it ends up being masked away by another shift of twelve hours.

[BurntSushi]

Thanks for the response! I'd bet on me misunderstanding something, but your note about tests being 12 hours off sounds suspicious. :)

Your note about leap seconds makes sense I think. My understanding is that if this library didn't handle leap seconds and, say, 10 seconds were added to 11:59:55 (where 11:59:59 is a leap second boundary), then the result would be 00:00:05, which is actually 11 seconds, not 10.

[gwbres]

side note, negative leap seconds may exist as well, although I don't think it ever happened, due to the tendency of Earth's rotation to slow down compare to atomic resonance frequencies

[BurntSushi]

Right. Never happened. Not yet. :) https://hpiers.obspm.fr/iers/bul/bulc/ntp/leap-seconds.list

[ChristopherRabotin]

You're correct that there is an issue with this constant: #282. It's used in the conversion to/from astrodynamical time systems (specifically ET and TDB), and in practice, it hides the fact that there's a twelve hour error in the definition of the J1900 reference time. As you have noted, hifitime keeps track of epochs as an offset from the reference of the given time scale. This offset is then used to convert to/from the Gregorian representation of time on demand. (I would argue that is the only correct way to define time physically speaking.) This bug is likely related to #261 as well.

This shows that I need to pick up work on hifitime, ha!

A quick question for you, as a highly prolific contributor to the Rust ecosystem: in fixing this bug, I'll necessarily introduce breaking changes to those who expect the buggy behavior. Therefore, should this lead to a new major version?

Thanks

P.S.: I'm keeping this discussion open until I resolve the ticket.

[BurntSushi]

Thank you for looking into this!

A quick question for you, as a highly prolific contributor to the Rust ecosystem: in fixing this bug, I'll necessarily introduce breaking changes to those who expect the buggy behavior. Therefore, should this lead to a new major version?

I think it's really up to your judgment. Typically a bug fix, even if it is technically a breaking change in some way, is acceptable to put into a semver compatible release. But some judgment is required to make sure you don't fall into the "cure is worse than the disease" trap. If fixing the bug would lead to widespread breaking changes for users and the bug is otherwise not severely impacting users, then it might justifiably be the wrong thing to do.

[ChristopherRabotin]

Just a quick update on this discussion from a renewed understanding from working the bug fix (for version 4.0). The variable J2000_TO_J1900_DURATION suffered from two errors that "canceled" each other out. First, I incorrectly noted J1900 as being the "prime epoch" of NTP: the original hifitime referenced NTP, and I incorrectly thought that J1900 and NTP started at the same time, but they don't (J1900 starts at noon, NTP at zero hours). The second error is that the 20th century, from 1900 to 2000, only had 36524 days, instead of 36525 days, because 1900 was not a leap year. However, a century is defined as 36425 days (even though a century only has that many days every fourth century).

Hence, the variable J1900 to J2000 should be the equivalent of 36524 days ... but the hifitime reference is the zeroth hour of 01 Jan 1900, so the variable is incorrectly named but correctly sets the J2000 offset.

I'll be removing this error promptly.

For posterity, I checked this math in SPICE. However, SPICE initializes everything in UTC, and incorrectly assumes that there were nine leap seconds before the first leap second of 1972, so let's just ignore the seconds here.

In [17]: sp.str2et("1900-01-01")
Out[17]: -3155716758.8160305

In [18]: sp.str2et("2000-01-01")
Out[18]: -43135.816087188054

In [20]: Unit.Second*(-3155716758.8160305 - -43135.816087188054)
Out[20]: -36524 days 22 s 999 ms 943 μs 168 ns @ 0x7f429db88490

[BurntSushi]

Thank you! :)

[ChristopherRabotin]

The upcoming release of version 4 of hifitime tackles this bug in the constants. I've added regression tests in this PR: #307 . Version 4 will be released in stages (alpha, beta, and stable), so definitely feel free to reopen this discussion or an issue if you notice other inconsistent or unexpected behavior. Although hifitime is mostly tailored toward precision engineering and scientific applications, I think it includes most features provided by the time and chrono crates, with the exception of providing "date only" epochs. If it's missing important features of those crates though, I'd be happy to implement them.

Thanks again for the report and the in-depth the conversation!

[BurntSushi]

I think the biggest difference, at least with Chrono, is time zone handling and integration with tzdb. And all that comes with that, for example, dealing with DST and performing time zone aware date arithmetic. For Chrono, you need chrono-tz for tzdb integration, but chrono itself provides the building blocks for dealing with time zone aware operations.

I have opinions on how Chrono handles this (I think it could be a lot better), but I do think it is a key practical difference between it and hifitime. Without it, you can't really express concepts like "what is the precise instant of 2024-03-11T05:00:00 in the America/New_York time zone?" Indeed, in some cases, the question has an ambiguous answer! (As in the case of 2024-03-10T02:30:00.)

hifitime, from a glance, is about on par with time. That is, both crates support offsets, but I believe stop there.

[BurntSushi]

Note that the time zone features (and especially time zone aware datetime arithmetic) is a rather large scope increase. I think it's perfectly fine to keep hifitime focused more on scientific applications. I just wanted to point out what I perceive as a major difference.