Merge pull request #46 from bxparks/develop

merge v1.5 into master

.github/workflows/aunit_tests.yml

@@ -13,7 +13,7 @@ jobs:
     - name: Setup
       run: |
         cd ..
-        git clone https://github.com/bxparks/UnixHostDuino
+        git clone https://github.com/bxparks/EpoxyDuino
         git clone https://github.com/bxparks/AceButton
         git clone https://github.com/bxparks/AceRoutine
         git clone https://github.com/bxparks/AUnit

.github/workflows/validation.yml

@@ -26,7 +26,7 @@ jobs:
     - name: Checkout 3rd Party Repos
       run: |
         cd ..
-        git clone https://github.com/bxparks/UnixHostDuino
+        git clone https://github.com/bxparks/EpoxyDuino
         git clone https://github.com/bxparks/AUnit
         git clone https://github.com/bxparks/AceCommon
         git clone https://github.com/eggert/tz

CHANGELOG.md

@@ -1,6 +1,63 @@
 # Changelog
 
 * Unreleased
+* 1.5
+    * Use binary search for both `ZoneManager::createForZoneName()` and
+      `ZoneManager::createForZoneId()`.
+        * Previously, the `zone_registry.cpp` was sorted by zoneName, so only
+          the `createForZoneName()` could use the binary search. The new
+          solution sorts the `zone_registry.cpp` entries by `zoneId` instead of
+          `zoneName`. The `createForZoneId()` can use the binary search
+          algorith.
+        * The `createForZoneName()` can also use the binary search because
+          the `zoneName` is converted dynamically to its `zoneId` using the same
+          djb2 hash algorithm used by the `tzcompiler.py`. If there is a match,
+          a final verification against the exact `zoneName` is performed to make
+          sure that there was no hash collision.
+        * Updated `AutoBenchmark.ino` to determine that a binary search on the
+          266 zones in `zonedb/zone_registry.cpp` is 9-10X faster (on average)
+          than a linear search through the same list. (Linear search takes ~190
+          iterations; binary search takes ~9 iterations.)
+    * Upgrade Link entries to be "fat links".
+        * Links become essentially identical to Zone entries, with references to
+          the same underlying `ZoneEra` records.
+        * Add `kZoneAndLinkRegistry[]` array in `zone_registry.h` that contains
+          all Links as well as Zones.
+        * Add "Zones and Links" section in `USER_GUIDE.md`.
+    * Implement zoneName compression using `ace_common::KString`.
+        * Saves about 1500-2300 bytes for basic `zonedb` info files, and
+          2500-3400 bytes for extended `zonedbx` info files.
+    * **Potentially Breaking Change**: Remove `transitionBufSize` from
+      `ZoneInfo` struct, and migrate to `kZoneBufSize{xxx}` constants in the
+      `zone_infos.h` files.
+        * This was used only in validation tests under `tests/validation` and
+          only for `Extended{xxx}` tests. Saves 1 byte per Zone on 8-bit
+          processors, but none on 32-bit processors due to 4-byte alignment.
+        * This should have no impact on client code since this field was used
+          only for validation testing.
+    * **API Breaking Change**: Replace `BasicZone::name()` and `shortName()
+      with `printNameTo()` and `printShortNameTo()`. Same with
+      `ExtendedZone::name()` and `shortName()`, replaced with `printNameTo()`
+      and `printShortNameTo()`.
+        * After implementing zoneName compression, it was no longer possible to
+          return a simple pointer to the `name` and `shortName` without using
+          static memory buffers.
+        * I expect almost no one to be using the `BasicZone` and `ExtendedZone`
+          classes, since they are mostly useful for internal algorithms.
+        * Client code that needs the old functionality can use
+          `BasicZone::printNameTo(Print&)`,
+          `BasicZone::printShortNameTo(Print&)` (similarly for `ExtendedZone`)
+          to print to a `ace_common::PrintStr<>` object, then extract the
+          c-string using `PrintStr::getCstr()`.
+    * Update UnixHostDuino 0.4 to EpoxyDuino 0.5.
+    * Explicitly blacklist megaAVR boards, and SAMD21 boards using
+      `arduino:samd` Core >= 1.8.10.
+        * This allows a helpful message to be shown to the user, instead of the
+          pages and pages of compiler errors.
+    * Update TZ Database to 2021a.
+        * https://mm.icann.org/pipermail/tz-announce/2021-January/000065.html
+        * "South Sudan changes from +03 to +02 on 2021-02-01 at 00:00."
+    * Officially support STM32 and STM32duino after testing on STM32 Blue Pill.
 * 1.4.1 (2020-12-30, TZDB version 2020f for real)
     * Actually update `src/ace_time/zonedb` and `src/ace_time/zonedbx`
       zone info files to 2020f. Oops.

DEVELOPER.md

@@ -45,7 +45,7 @@ The `kTzDatabaseVersion` string constant identifies the version of the TZ
 Database that was used to generate these files, e.g. "2019a".
 
 The `kZoneContext` variable points to an instance of `common::ZoneContext`
-which identifies the `startYear` and `endYear` of the current database. These
+which identifies the `startYear` and `untilYear` of the current database. These
 will normally be 2000 and 2050 respectively, but a custom version of the
 zonedb files could be generated whose startYear and endYear could be smaller (to
 produce smaller ZoneDB files).
@@ -58,17 +58,40 @@ known inaccuracies for a particular zone.
 
 ### `zone_infos.cpp`
 
+The top of this file contains a summary of the sizes of various parts of
+the data structures in this file, and the 2 registries supplied by
+`zone_registry.cpp`:
+
+```
+// Zones: 386
+// Links: 207
+// kZoneRegistry sizes (bytes):
+//   Names: 3667 (originally 6100)
+//   Formats: 597
+//   Fragments: 122
+//   Memory (8-bit): 16848
+//   Memory (32-bit): 24464
+// kZoneAndLinkRegistry sizes (bytes):
+//   Names: 5620 (originally 9027)
+//   Formats: 597
+//   Fragments: 122
+//   Memory (8-bit): 21492
+//   Memory (32-bit): 31385
+```
+
 Each zone entry in the `zone_info.cpp` contains a comment section that
 describes some metadata about the given entry. For example, the entry for
-`zonedb::kZoneEraAmerica_Los_Angeles` contains the following, showing how
+`zonedb::kZoneAmerica_Los_Angeles` contains the following, showing how
 much memory it will consume on the 8-bit Arduino controllers and the 32-bit
 Arduino controllers:
 
-* `// Zone name: America/Los_Angeles`
-* `// Era count: 1`
-* `// Strings: 24`
-* `// Memory (8-bit): 43`
-* `// Memory (32-bit): 53`
+```
+// Zone name: America/Los_Angeles
+// Zone Eras: 1
+// Strings (bytes): 17 (originally 24)
+// Memory (8-bit): 39
+// Memory (32-bit): 53
+```
 
 ### `zone_policies.h`
 
@@ -80,14 +103,100 @@ Each policy entry starts with a comment secion that contains some metadata
 about the policy. For example:
 ```
 // Policy name: US
-// Rule count: 5
+// Rules: 5
 // Memory (8-bit): 51
-// Memory (32-bit): 55
+// Memory (32-bit): 72
 ```
 Just like `zone_infos.cpp`, the Memory section describes the amount of static
 RAM consumed by the particular `ZonePolicy` data structure (and associated
 `ZoneRule`.
 
+## Encoding Offset and Time Fields
+
+There are 5 offsets and moment-in-time quantities from the TZ zoneinfo files
+which are captured in the `zone_info.{h,cpp}` and `zone_policies.{h,cpp}` files:
+
+* `STDOFF` field in `Zone` entry (previously `OFFSET`), 1-minute resolution
+* `RULES` field in `Zone` entry when numeric (e.g. "1:00"), 15-minute resolution
+* `UNTIL` field in `Zone` entry, 1-minute resolution
+* `SAVE` field in `Rule` entry, 15-minute resolution
+* `AT` field in `Rule` entry, 1-minute resolution
+
+To reduce flash memory size, these fields are encoded in non-obvious ways which
+are difficult to remember. Here is my attempt to document the encoding. In the
+following diagram, a field labled `code` (e.g. `offsetCode`) has a unit of 15
+minutes. For example, an `offsetCode` of 2 means 30 minutes. To capture time
+offsets or moments with a 1-minute resolution, we store the remaining 15-minutes
+(0 to 14 inclusive), using 4-bits in the upper 4-bits or the lower 4-bits of one
+of the other fields. For the "extended" zoneinfo files (i.e. `zonedbx`), the
+`code` value is shifted by +4 (in other words, 1 hour), so that a 4-bit field is
+able to represent offsets from -1:00 to +2:45. This allows us to capture zones
+which have a negative -1:00 hour offset, and zones which undergo a "double DST"
+shift of +2:00 hours.
+
+```
+TZ DB Fields                            ZoneEra
+============                            =======
+
+STDOFF (1-min resolution)
+-> offset_seconds                       +--------------------+
+  -> offset_code ---------------------> |     offsetCode     |
+     offset_minute                      |     (8 bits)       |
+                  \                     +--------------------+
+                   \
+                    \                   +--------------------+
+                     \                  |     deltaCode      |
+                      \                 +--------------------+
+                       ---------------> | minute  | code     |
+                                        | (4-bits)| (4-bits) |
+                                        +--------------------+
+                                                      ^
+RULES (numeric, 15-min resolution)                   /
+-> rules_delta_seconds                              /
+    -> rules_delta_code -- (basic) ----------------/
+                        -- (extended) + 4 ---------
+
+
+UNTIL (1-min resolution)
+-> until_time_seconds                   +--------------------+
+   until_time_suffix (w, s, u)          |   untilTimeCode    |
+   -> until_time_code   --------------> |   (8-bits)         |
+      until_time_minute -------.        +--------------------+
+      until_time_suffix --.     \
+                           \     --------------------------------.
+                             \                                    \
+                              \         +--------------------+     |
+                               \        | untilTimeModifier  |     |
+                                \       +--------------------+    /
+                                 -----> | suffix  | minute   | <-'
+                                        | (4-bits)| (4-bits) |
+                                        +--------------------+
+
+
+                                        ZoneRule
+                                        ========
+AT (1-min resolution)
+-> at_time_seconds                      +--------------------+
+   at_time_suffix (w, s, u)             |   atTimeCode       |
+   -> at_time_code   -----------------> |   (8-bits)         |
+      at_time_minute -----------.       +--------------------+
+      at_time_suffix -----.      \
+                           \      -------------------------------.
+                             \                                    \
+                              \         +--------------------+     |
+                               \        |   atTimeModifier   |     |
+                                \       +--------------------+    /
+                                 -----> | suffix  | minute   | <-'
+                                        | (4-bits)| (4-bits) |
+                                        +--------------------+
+
+SAVE (15-min resolution)
+-> delta_seconds                        +--------------------+
+    -> delta_code -- (basic) ---------> |     deltaCode      |
+                  -- (extended) + 4 --> |     (8 bits)       |
+                                        +--------------------+
+```
+
 ## Upgrading ZoneInfo Files to a New TZDB Version
 
 About 2-4 times a year, a new TZDB version is released. Here are some notes
@@ -136,7 +245,7 @@ available.
         * `$ make`
     * `tools/zonedbpy`
         * `$ cd tools/zonedbpy`
-        * Edit the Makefile and update the `TZ_VERSION`.
+        * Edit the `Makefile` and update the `TZ_VERSION`.
         * `$ make`
 * Update the CHANGELOG.md.
 * Commit the changes to git
@@ -159,9 +268,6 @@ will fail until the underying timezone database of the OS is updated.
     * `docs/doxygen.cfg`
     * `library.properties`
     * `CHANGELOG.md`
-    * Bump the version numbers in the unit tests:
-        * `BasicZoneProcessorTest/BasicZoneProcessorTest.ino`
-        * `ExtendedZoneProcessorTest/ExtendedZoneProcessorTest.ino`
     * `$ git commit -m "..."`
 * Update and commit the Doxygen docs. This is done as a separate git commit
   because the Doxygen changes are often so large that they obscure all other