-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Temperature check: support address_getAppearances #4054
Comments
Also, without a complete history of appearances, it's impossible to build a reconcilable accounting (in both the financial sense and the 'state change' sense). This is exactly why all of us are struggling to do automated accounting "off-chain" against an "on-chain" system that reconciles 240,000,000 addresses every 12 seconds. |
I'm not speaking for the original poster here, but, yes, this would require an additional index and associated hard drive space. That's why it should be optional. It's also why I proposed (and @perama-v wrote) this feature as well: ethereum/execution-apis#452.
If I'm not suggesting that |
Here are some notes after reviewing existing structures
|
i'm conservatively supportive. Between this, and otterscan index needs, we probably need better abstractions to accommodate feature/flag based indexes without being too messy I'm skeptical about the estimate... 80GB seems low, but if it's optional it's whatever
If i'm not mistaken, this implies having to do "something" during the execution stage as well |
The estimate is based on the UnchainedIndex, which has the same data (address-->appearances), but is represented in a custom format suited for distribution by 12 hour block ranges (total size ~80gb, no compression)
Yes, for each tx certain locations are checked for addresses. So this is additional work that must be performed. The specifics are in the spec in the linked execution apis PR. The work roughly amounts to a trace with calltracer, and checking fields within that response. Plus a check of addresses in withdrawals and block rewards. |
Added a task list for the foreseeable steps. Regarding the proposed implement One consideration is that it this re-executes the transactions, separately from the Execution stage. One benefit is that |
There's a third option here. Enhance the
|
I would encourage you to think very carefully about this. Adding what I call a "complete" index of appearances allows one to produce a "complete" transactional history for any account. This allows for perfect 18-decimal place accurate re-creation of an accounts activity off-chain. This allows perfect, 18-decimal place accurate automated accounting every 12 seconds. If you ever wondered why no-one can produce automated tax information from a system that comes to balance for 200,000,000 million accounts every 12 seconds, it's because there is no way to get a complete history. |
Also, if it was combined into the |
I'd love this thinking aloud here, does putting this in an down the road I would love to see additional advanced querying rpc methods (like so maybe something like |
🍺
Great and important question. I am open to this, let's discuss over where namespace is defined |
yes, am supportive. This sort of capability in the EVM node software seems to have been an oversight in the original EVM spec. Unclear what storage is required, but as long as it is optional, can be a node-specific index. I would gladly trade ~100 GB for such an index for my nodes. |
A contract read is a superset of a log.
Would definitely go with the first option, as the second path introduces dependency that increases complexity.
Would double that estimation as we need to save this inside db, but db size is not that relevant for discussion.
AccountHistory has index to the block but it seem for this rpc it is more preferred to go with tx index.
Although this endpoint would include just an Ethereum balance change, it would not have the address/storages of the ERC20 contract storage slots that are, I assume, needed for accounting. imo appearance is defined very broadly here. And we are talking here just for account appearance (storage is not mentioned). The request is to have a list of transaction numbers (not blocks number that we use in AccountHistory) this is not possible with current tx numbers as we have appearances at the end of the block that would fall in the middle of two transactions (last of the block and first of the next block). When we have tx level changesets we had an additional index transition index to accommodate this. On the naming (not relevant), I would see this name as |
I totally understand everything you're saying here, and yes, the index as original envisioned would index into
Concerning the issue of state change, you're exactly right. All the existing indexes that I'm aware of in the node ignore the changes to storage where the thing being stored is an address. This is exactly the problem. This is why it's impossible to re-create state changes (including accurate balance histories) off-chain. I would almost go so far as to say that implementing an index that does not capture these types of appearances (where an address is "used" by a smart contract) wouldn't be worth it. You can get all the other "appearances" already. This "adding all appearances" may be more complicated that you might thing though because a HUGE number of such appearances happen as left-side zero-padded 32-bytes. You have to use some heuristics to identify things that "appear" to be addresses. I've written extensively about this issue on pages 32-41 of this document: https://trueblocks.io/papers/2023/specification-for-the-unchained-index-v0.51.0-beta.pdf Concerning naming, I think I probably fall down on the side of |
Thanks
Yes, it is a broad definition to answer the question: "which transactions should I look further into for my address?". While broad in coverage, the Appearances spec (see link at top of issue) aims to have a precise inclusion definition. See also the implementation in #4162. The
The main idea is to be agnostic to the end use case. The method returns a list of relevant transaction ids that could be used by the user to start ERC20 accounting, ERC721 accounting, ether accounting, or any other purpose. They would fetch and inspect/re-execute transactions according to their needs, which may or may not involve looking at storage changes.
I have implemented a possible mechanism to achieve this, using a wrapper that adds bitflags to a |
I wanted to highlight this for two reasons: (1) the definition of appearance should be as broad as possible so that it includes any possible use case-let the end user decide which particulars they are interested in, and (2) almost every time I explain this to someone, I use the idea of accounting, but the "any other purpose" of this may be the most important. With a complete list of appearances, once could re-create any state change whatsoever. Not just changes in balances. Changes in any state. This also argues for as inclusive an index as possible. That's why I focus on what I call "Every appearance of any address anywhere on the chain." But, note that there has to be the notion of a |
Love the progress on all this - just to say it explicitly, we're happy to iterate on this in the branch, but ideally we can figure out ways to introduce APIs like this w/o necessarily introducing large main codebase maintenance overheads.. We already have utilities for extending the CLI, see here: https://github.com/paradigmxyz/reth/blob/main/examples/additional-rpc-namespace-in-cli/src/main.rs#L52. So it might be a matter of showing how to also run additional stages. |
This issue is stale because it has been open for 14 days with no activity. |
This issue was closed because it has been inactive for 7 days since being marked as stale. |
Happy to see this reopened, just 'cause I'm really looking forward to seeing Reth implement a solution to, in my view, the early failure to spec this into the EVM in 2015. |
Describe the feature
This issue is to introduce the concept of "address appearances" and assess appetite for inclusion in reth.
Specifications
See also the following specifications:
address_getAppearances
ethereum/execution-apis#453eth_getAddressesInBlock
method ethereum/execution-apis#452Motivation
A user (or local application) running reth can start with an address (e.g.,
0x30a4...1382
) important to them.They query reth and get the set of transactions that are relevant to that address.
This kicks off the journey, which can now proceed in many ways:
As addresses are the foundation of user and protocol activity, the feature is general purpose.
Why "in reth" and not separate
The presence of the
address_getAppearances
natively within reth would mean first class support for user-focused applications. This could lead to:address_getAppearances
to locate relevant transactions. The frontend does not require any node restart / trace filter / external application to function.As the method is general purpose it always supports new wallets/protocols that a user is suddenly interested in.
Prior work
The UnchainedIndex, accessible via trueblocks-core (https://github.com/TrueBlocks/trueblocks-core) is an example of the utility of the information that
address_getAppearances
would provide.The method is equivalent to the following command in
trueblocks-core
:Disk impact
The support for
address_getAppearances
creates additional disk burden. An upper ceiling estimate is 80GB.The estimate is based on the UnchainedIndex (80GB), which is specially designed for distribution/sharding on IPFS. Such requirements mean that the index adds immutable pieces of data every 12 hours. An address that appears every 12 hours is therefore duplicated.
There are likely means to significantly reduce the disk footprint of a native reth version of this data.
Additional context
What is an "appearance"
Appearances are defined in a way to capture instances when an address arises in a meaningful way in the chain.
A simple example is the subject of a
CALL
opcode to an EOA. The recipient "appears" in that transaction (they receive ether). Upon tracing, the meaning of the transaction can be determined.An "appearance" is defined in the following PR:
address_getAppearances
methodThe method is defined in the following PR:
address_getAppearances
ethereum/execution-apis#453This is a chicken an egg situation: The method is stronger if there is an indication of existence/support in a client and vice versa.
Temperature check
Seeking feedback on this concept. Do any of the following resonate with you? Please feel free to add thoughts of any kind
address_
namespace foraddress_getAppearances
be considered?some_application_idea
"<other>
Thanks for your time
Checklist
The text was updated successfully, but these errors were encountered: