v0.36.0

[steven-sheehy]

We are happy to announce the availability of a publicly accessible, free-to-use, mainnet Mirror Node operated by the Hedera team. As part of this, we put a large amount of effort into fine tuning our Kubernetes deployment. We migrated to Flux 2, a GitOps-based deployment tool that allows us to declaratively specify the expected state of the Mirror Node in git and manage our rollouts. You can browse our deploy branch and see the exact config and versions rolled out to various clusters and environments. The Helm chart was updated to add PodDisruptionBudgets, adjust alert rules and other improvements to make it easier to automate the deployment.

This release is the first version of the Mirror Node with preliminary support for non-fungible tokens (NFTs). NFT support is being added to the Hedera nodes as outlined in HIP 17. We spent time designing how that NFT support will look like for the Mirror Node. Modifications to the schema were made to add new tables and fields and the Importer was updated to ingest NFT transactions. The existing REST APIs were updated to add NFT related fields to the response. This includes adding a type field to the token related APIs to indicate fungibility and anft_transfers to /api/v1/transactions/{id}:

{
  "transactions": [{
      "consensus_timestamp": "1234567890.000000001",
      "name": "CRYPTOTRANSFER",
      "nft_transfers": [
        {
          "receiver_account_id": "0.0.121",
          "sender_account_id": "0.0.122",
          "serial_number": 104,
          "token_id": "0.0.14873"
        }
      ]
  }]
}

One thing to note is that we did not add NFT transfers to the list transactions endpoint in an effort to reduce the size and improve the performance of that endpoint. In the next release we will add new NFT specific REST APIs.

Continuing upon the theme of the last release, we made additional changes to the Rosetta API to bring it up to par with the rest of the components. Rosetta now includes support for HTS via both is data and construction APIs.

The Importer saw a large focus on improving performance and resiliency. It is now highly available (HA) when run inside Kubernetes. This allows more than one instance to run at a time and to failover to the secondary instance when the primary becomes unhealthy. A special Kubernetes ConfigMap named leaders is used to atomically elect the leader.

We’re improving our ingestion time dramatically for entity creation. Previously those were database finds followed by updates. Since inserts are always faster than find and updates, we’ve optimized this to insert the updates into a temporary table and at the end upsert those to the final table. A record file with 6,000 new entities went from 21 seconds to 600 ms, making it 35x improvement. Balance file processing was optimized to greatly reduce memory by only keeping one file in memory at a time.

Breaking Changes

In honor of Juneteenth and as part of general industry-wide movement, we renamed our master branch to main. If you have a clone or fork of the Mirror Node Git repository, you will need to take the below steps to update it to use main:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

As part of our optimization to reduce memory usage, we now process some things earlier in the lifecycle. Due to this we had to rename some properties to reflect this change. We also changed the disk structure if you are using the keepFiles (now renamed to writeFiles) properties to write the stream files to disk after download. It is no longer archived into folders by day. Instead, the folder structure will exactly match the structure in the bucket. This opens the possibility for a mirror node to download and mirror the bucket itself using a S3 compatible API like MinIO. Below is a summary of the renamed properties:

• Renamed hedera.mirror.importer.downloader.balance.keepSignatures to hedera.mirror.importer.downloader.balance.writeSignatures
• Renamed hedera.mirror.importer.parser.balance.keepFiles to hedera.mirror.importer.downloader.balance.writeFiles
• Renamed hedera.mirror.importer.parser.balance.persistBytes to hedera.mirror.importer.downloader.balance.persistBytes
• Renamed hedera.mirror.importer.downloader.event.keepSignatures to hedera.mirror.importer.downloader.event.writeSignatures
• Renamed hedera.mirror.importer.parser.event.keepFiles to hedera.mirror.importer.downloader.event.writeFiles
• Renamed hedera.mirror.importer.parser.event.persistBytes to hedera.mirror.importer.downloader.event.persistBytes
• Renamed hedera.mirror.importer.downloader.record.keepSignatures to hedera.mirror.importer.downloader.record.writeSignatures
• Renamed hedera.mirror.importer.parser.record.keepFiles to hedera.mirror.importer.downloader.record.writeFiles
• Renamed hedera.mirror.importer.parser.record.persistBytes to hedera.mirror.importer.downloader.record.persistBytes

Enhancements

• Remove NFT transfers from transactions list endpoint (cherry-pick) #2183
• Rename references from master to main (cherry-pick) #2170
• Improve acceptance accountId and retry logic (cherry-pick) (Improve acceptance accountId and retry logic #2162) #2169
• Add a PodDisruptionBudget to gRPC and REST #2138
• Add NFT support to existing REST APIs #2129
• Adjust Traefik rate limit #2125
• Add high availability to importer #2124
• Nft importer #2109
• Automate deploy to integration #2102
• Add Rosetta HTS support #2093
• Add workflow for building on windows #2089
• Add a global hostname, pullPolicy and podAnnotations properties to chart #2073
• NFT Database Schema #2070
• Bump versions for v0.36.0-SNAPSHOT #2068
• NFT Design Doc V2 #2061
• NFT existing REST APIs #2058
• NFT design update #2057
• NFT database schema #2056
• Fix slow domain object ingestion #2006
• NFT importer support #1996
• Slow record parsing with high number of created entities #1926
• Rosetta HTS support #1920
• Enable importer leader election #1899
• Migrate to Flux 2 #1811
• Utilize cache for EntityRecordItemListener bulk insert/update operations on domain objects #1440

Bug Fixes

• Fix helm test failing during retry (0.36) #2203
• Deduplicate NFT transfers (cherry-pick) #2168
• Fix Grafana & adjust alerts (0.36) #2165
• Add multi batch support to upsert pgcopy flow to release 0.36 #2155
• Upsert pgcopy flow attempt to create already present temp table #2154
• Adjust alert thresholds & fix alert dashboard #2150
• Duplicate nft transfers returned by /transactions rest api #2139
• Increase verify timeout to 2s in RedisEntityListenerTest #2128
• RedisEntityListenerTest.onSlowPublish fails in github CI #2127
• Simplify automated release workflow #2126
• Optimize importer memory footprint #2103
• Fix transactions query duplicate timestamps issue #2097
• Dissociate accounts from tokens post run #2090
• OOMKilled in performance Kubernetes #2087
• Update HTS Acceptance Tests to Dissociate payer on complete #2069
• Duplicate token transfers returned by /transactions rest api #2066
• Transactions returned in single account rest query less than expected #2021
• RedisEntityListenerTest.onSlowPublish failed in github action #1651

Dependency Upgrades

• Bump hibernate-types-52 from 2.11.1 to 2.12.0 #2143
• Bump grpc.version from 1.38.0 to 1.38.1 #2142
• Bump commons-io from 2.9.0 to 2.10.0 #2121
• Bump software.amazon.awssdk:bom from 2.16.78 to 2.16.84 #2120
• Bump jib-maven-plugin from 3.0.0 to 3.1.1 #2119
• Bump protobuf-java from 3.17.2 to 3.17.3 #2118
• Bump jackson-dataformat-msgpack from 0.8.24 to 0.9.0 #2117
• Bump spring-cloud-gcp-dependencies from 2.0.2 to 2.0.3 #2116
• Bump spring-boot-starter-parent from 2.4.6 to 2.5.1 #2114
• Bump dependency-check-maven from 6.2.0 to 6.2.2 #2113
• Bump sdk from 2.0.5 to 2.0.6 #2112
• Bump aws-sdk from 2.922.0 to 2.927.0 in /hedera-mirror-rest #2111
• Bump jest-junit from 12.1.0 to 12.2.0 in /hedera-mirror-rest #2110
• Upgrade Spring Boot to 2.4.6 #2101
• Bump normalize-url from 4.5.0 to 4.5.1 in /hedera-mirror-rest #2100
• Bump normalize-url from 4.5.0 to 4.5.1 in /hedera-mirror-rest/monitoring/monitor_apis #2099
• Bump glob-parent from 5.1.1 to 5.1.2 in /hedera-mirror-rest #2092
• Bump glob-parent from 5.1.1 to 5.1.2 in /hedera-mirror-rest/monitoring/monitor_apis #2091
• Bump software.amazon.awssdk:bom from 2.16.74 to 2.16.78 #2086
• Bump mathjs from 9.4.1 to 9.4.2 in /hedera-mirror-rest #2085
• Bump jest from 27.0.3 to 27.0.4 in /hedera-mirror-rest/check-state-proof #2084
• Bump protobuf-java from 3.17.1 to 3.17.2 #2083
• Bump vertx-pg-client from 4.0.3 to 4.1.0 #2082
• Bump embedded.testcontainers.version from 2.0.8 to 2.0.9 #2081
• Bump aws-sdk from 2.918.0 to 2.922.0 in /hedera-mirror-rest #2080
• Bump @godaddy/terminus from 4.8.0 to 4.9.0 in /hedera-mirror-rest #2079
• Bump prettier from 2.3.0 to 2.3.1 in /hedera-mirror-rest #2078
• Bump jest-circus from 27.0.3 to 27.0.4 in /hedera-mirror-rest #2077
• Bump jest from 27.0.3 to 27.0.4 in /hedera-mirror-rest #2076
• Bump mathjs from 9.4.1 to 9.4.2 in /hedera-mirror-rest/monitoring/monitor_apis #2075
• Bump jest from 26.6.3 to 27.0.4 in /hedera-mirror-rest/monitoring/monitor_apis #2074
• Bump hedera-protobuf-java-api from 0.13.2 to 0.14.0 #2015

Contributors

We'd like to thank all the contributors who worked on this release!

• @si618
• @hedera-github-bot
• @xin-hedera
• @Nana-EC
• @dependabot[bot]
• @ijungmann
• @steven-sheehy

---

This discussion was created from the release v0.36.0.