Upgrades and Historical Queries

[aaronc]

When we do an upgrade with x/upgrade, historical queries have a high chance of breaking. Even though the node still has the historical state, the query handlers are expecting state in the new format which may have been migrated starting with the upgrade block.

So we need to find a way to gracefully support historical queries. The simplest method would probably be having nodes that are halted at upgrade heights. To make this user friendly we'd likely want a proxy server which routes requests to different nodes based on the height parameter. Possibly this is something that cosmovisor could do. I'm not sure if there would be an easy way for historical binaries to be spun up in query only mode against the same data dir. Anyway, maybe something worth thinking about just for RPC nodes.

Another idea could be somehow trying to migrate historical states for RPC nodes... but I'm not sure about the feasibility of that. Seems like a lot of work and proofs would break, but maybe...

Any thoughts?

[alexanderbez]

I think you summed it up pretty nicely @aaronc. As far as I see it, the proxy/service approach might be the easiest path forward. All an operator would have to do is state which height ranges map to which versions and what endpoints the service can reach each individual binary version at. This service itself should be fairly trivial to understand and implement. It just requires for operators to run all desired (old) binary versions they wish to support.

[alexanderbez]

Having this be part of comosvisor also probably makes sense since it already keeps track of previous binary versions too. I'm not too sure how well this will play with Rosetta though. Does rosetta even need support for historical data?

[tac0turtle]

How would a user route queries to the respective node? Is this some sort of overlay network which communicates who has the info?

[alexanderbez]

No, no network. This is a simple proxy service. From the client's perspective, it's no different from requesting from a regular application API on 1317.

$ curl <proxy>/some/path?height=3456

e.g. this proxy knows 3456 maps to some process and routes the request to that process and simply returns the response.

[tac0turtle]

ah so this service would need to be continuously fed new RPC endpoints?

[alexanderbez]

No, it's preconfigured.

example config could look like:

[[service]]
version = "v4.x.x"
endpoint = "localhost:1311"
start_height = 0
end_height = 1000000

[[service]]
version = "v5.x.x"
endpoint = "localhost:1312"
start_height = 1000001

Then the proxy uses the height query param or header to look up the appropriate service to proxy the request to. Every time an application launches a new version, then you'd add a new [[service]] to the proxy service.

[cmwaters]

Another idea could be somehow trying to migrate historical states for RPC nodes... but I'm not sure about the feasibility of that. Seems like a lot of work and proofs would break, but maybe...

I explored this option with the soft upgrades RFC. It is feasible but challenging and probably not worth the endeavour.

You might also find this issue interesting which looks at introducing a lean archive mode in Tendermint that runs a few bare services designed for data availability of previous versions of a chain

[robert-zaremba]

Another idea is to support a limited number of "old" query services (eg for the previous version of the SDK). This is how it can be implemented:

1. once we will have protobuf generated code versioned separately from the module, we can implement both query services
2. an old query service will be registered as a separate service in the module.

IMHO, however, supporting historical state should be delegated to indexers.

[iramiller]

This issue highlights an architectural shortfall of the current Cosmos SDK application design; that the app itself that tendermint is interacting with has no concept of internal versioning against the on chain configuration.

The Cosmovisor process offers a partial solution to this problem by trying to start and stop binary versions as requested by the chain. The proxied multiprocess query server(s) approach above is also offering partial solutions around this core design problem. Module version number maps are also a step along this path but as they do not go for enough as they only focus on upgrades instead of parallel support.

In the future it would make sense if a full design effort was implemented that allowed the abci applicaiton to understand and execute these different versions in parallel according to the on chain configuration. In this type of design a user could always run the "latest" version of the app as it would contain all the versioned protocols as internal components/process... upgrades for nodes would be much easier to deal with as the additional configuration could be loaded into the processes in advance, validated, and transitioned to when scheduled. These same libraries could be maintained for historical queries.

[ethanfrey]

Note, if you try this with CosmWasm contracts, it should just work.

If I migrate a contract at address cosmos1234 at some height, then the queries after that height will use the new data and the new code. Older queries will use the original data format and the original code.

I never thought of this effect, but nice to see how "code as data" (configuration stored in IAVL tree) has some nice side-effects.

[iramiller]

@ethanfrey does your logic apply to major upgrades that change the wasmvm version as well? Under the typical case it does seem like it would be be seamless... similar to Avaro having an embedded schema vs proto...

[ethanfrey]

This applies to most upgrades of wasmvm was well.

We guarantee that a given contract (wasm blob) that ran in v1.0.0-beta will run the same in wasmvm/cosmwasm-vm v1.9.5, thus the historical queries would work. The reasoning holds to all those upgrades. And the new codes that were added that need v1.7 support will not be present in historical queries before that upgrade.

If you upgrade from wasmvm v1.x to v2.x (not even in our roadmap or an idea of what that would contain), then this may be breaking/break historical queries. We will need to put lots of thinking on any sort of 2.x upgrade roadmap and I cannot imagine this before 2023 at the earliest. That would give time to discuss/address such concerns.

[alexanderbez]

I understand the frustration @iramiller, but to have the application be aware of previous versions and be able to serve data from those version(s) is very much non-trivial, so I would hesitate to call it a "architectural shortfall", but rather a problem that is not trivial to solve.

While it could be done and I can imagine a few ways to go about it, it would require significant engineering effort to support such functionality.

[iramiller]

Architecture decisions this deep are always trade offs and something is always left behind. I don't mean my comment as a direct criticism of anyone who pulled this together--it's the reality of any system that trade offs are made.

In this case focus on use of a unique binary at a given block height range means there are these hard transitions. Unfortunately that detail surfaces to clients in unexpected ways when performing queries as a request for data that returns an empty response might not indicate the same thing in all cases.

[baabeetaa]

Great idea, I found this old discussion when started working on subnode (https://github.com/notional-labs/subnode). It also fix the problem of archive node too big, too slow.

[popterlog]

Great idea!