CCIP-1691 Using more accurate and detailed query for fetching messages and execution state changes

[mateusz-sekara]

Motivation

Every time root is batched and scheduled for execution we snooze that for some time (usually 3-5 minutes). After that time, the root returns to the queue and we verify if it’s executed. Root is considered fully executed when all messages are executed and related ExecutionStateChanges events are finalized. For instance, if the time to finality is 30 minutes and the root snooze time is set to 3 minutes, the root will be checked 10 times before marking it as fully executed.

There is one caveat here, we fetch messages from OnRamp using range, we pick the oldest and youngest roots and fetch all messages where seq >= olderRoot.MinSeqNr and seq <= youngestRoot.MaxSeqNr

Suppose you have the following scenario

Instead of fetching messages 1-100 and 401 - 500, the current implementation will fetch 1 → 500. The slower the finality the bigger the problem with current implementation is.

Solution

Switch to the query that supports fetching only the sequence numbers we need. This can be achieved with FilteredLogs and querying by ranges (report1.min <= seqNr <= report2.max) OR ... OR (reportN.min <= seqNr <= reportN.max)

Offramp: Go bench results

Scenario 1: Using previous LogPoller query for fetching based on min/max

offRamp.GetExecutionStateChangesBetweenSeqNums(testutils.Context(b), 1024, 2047, 0)

Benchmark_FilteredLogsQuery
Benchmark_FilteredLogsQuery-12    	      38	  33244664 ns/op
Benchmark_FilteredLogsQuery-12    	      37	  31802605 ns/op
Benchmark_FilteredLogsQuery-12    	      37	  33733635 ns/op
Benchmark_FilteredLogsQuery-12    	      33	  33035030 ns/op
Benchmark_FilteredLogsQuery-12    	      38	  31528083 ns/op

Scenario 2: Using query built with FilteredLogs but using single range

offRamp.GetExecutionStateChangesForSeqNums(testutils.Context(b), []cciptypes.SequenceNumberRange{{Min: 1024, Max: 2047}}, 0)

Benchmark_FilteredLogsQuery
Benchmark_FilteredLogsQuery-12    	      34	  33821957 ns/op
Benchmark_FilteredLogsQuery-12    	      33	  33094408 ns/op
Benchmark_FilteredLogsQuery-12    	      33	  32995907 ns/op
Benchmark_FilteredLogsQuery-12    	      30	  39200622 ns/op
Benchmark_FilteredLogsQuery-12    	      30	  38383450 ns/op

Scenario 3: Using new query but with more sophisticated case and multiple ranges passed (on average ~10ms to the query time)

	logs, err1 := offRamp.GetExecutionStateChangesForSeqNums(testutils.Context(b), []cciptypes.SequenceNumberRange{
		{Min: 1000, Max: 1099},
		{Min: 1200, Max: 1299},
		{Min: 1800, Max: 1999},
		{Min: 2200, Max: 2499},
	}, 0)

Benchmark_FilteredLogsQuery
Benchmark_FilteredLogsQuery-12    	     259	  47371107 ns/op
Benchmark_FilteredLogsQuery-12    	     246	  47616777 ns/op
Benchmark_FilteredLogsQuery-12    	      76	  47426643 ns/op
Benchmark_FilteredLogsQuery-12    	      73	  46377761 ns/op
Benchmark_FilteredLogsQuery-12    	      80	  46935801 ns/op

Offramp: Filtering larger ranges with proper smaller ranges with query analyzer

The old query needs to scan the entire range from report[0].Min till report[-1].Max, whereas the new query selects proper smaller ranges which leads to fetching deterministic and smaller datasets from the database. Tests run on the prod-testnet environment.

explain analyze
select *
from evm.logs
where evm_chain_id = 11155111
  and address = '\x000b26f604eaadc3d874a4404bde6d64a97d95ca'
  and event_sig = '\xd4f851956a5d67c3997d1c9205045fef79bae2947fdee7e9e2641abc7391ef65'
  and topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c5a'
  and topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002ec0'
  and block_number <= (SELECT greatest(block_number , 0)
                       FROM evm.log_poller_blocks
                       WHERE evm_chain_id = 11155111
                       ORDER BY block_number DESC
                       LIMIT 1)
order by block_number asc, log_index asc;

Sort  (cost=80.77..80.77 rows=1 width=367) (actual time=3.469..3.507 rows=623 loops=1)
   Sort Key: logs.block_number, logs.log_index
   Sort Method: quicksort  Memory: 363kB
   InitPlan 1 (returns $0)
     ->  Limit  (cost=0.43..1.39 rows=1 width=16) (actual time=1.609..1.610 rows=1 loops=1)
           ->  Index Only Scan using idx_evm_log_poller_blocks_order_by_block on log_poller_blocks  (cost=0.43..101902.73 rows=106056 width=16) (actual time=1.608..1.609 rows=1 loops=1)
                 Index Cond: (evm_chain_id = '11155111'::numeric)
                 Heap Fetches: 1
   ->  Index Scan using idx_evm_logs_ordered_by_block_and_created_at on logs  (cost=0.56..79.37 rows=1 width=367) (actual time=1.664..3.291 rows=623 loops=1)
         Index Cond: ((evm_chain_id = '11155111'::numeric) AND (address = '\x000b26f604eaadc3d874a4404bde6d64a97d95ca'::bytea) AND (event_sig = '\xd4f851956a5d67c3997d1c9205045fef79bae2947fdee7e9e2641abc7391ef65'::bytea) AND (block_number <= $0))
         Filter: ((topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c5a'::bytea) AND (topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002ec0'::bytea))
         Rows Removed by Filter: 807
 Planning Time: 0.268 ms
 Execution Time: 3.574 ms

explain analyze
select *
from evm.logs
where evm_chain_id = 11155111
  and address = '\x000b26f604eaadc3d874a4404bde6d64a97d95ca'
  and event_sig = '\xd4f851956a5d67c3997d1c9205045fef79bae2947fdee7e9e2641abc7391ef65'
  and (
        (topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c5a' and
         topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c5b')
        or
        (topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c5d' and
         topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c5e')
        or
        (topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c61' and
         topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c63')
        or
        (topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c67' and
         topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c6a')
        or
        (topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c6f' and
         topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c78')
        or
        (topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c81' and
         topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c87')
        or
        (topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c8d' and
         topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c93')
        or
        (topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002eaf' and
         topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002ec0')
    )
    and block_number <= (SELECT greatest(block_number , 0)
                        FROM evm.log_poller_blocks
                        WHERE evm_chain_id = 11155111
                        ORDER BY block_number DESC
                        LIMIT 1)
order by block_number asc, log_index asc, tx_hash asc;

Sort  (cost=81.47..81.47 rows=1 width=367) (actual time=2.782..2.787 rows=51 loops=1)
   Sort Key: logs.block_number, logs.log_index, logs.tx_hash
   Sort Method: quicksort  Memory: 53kB
   InitPlan 1 (returns $0)
     ->  Limit  (cost=0.43..1.39 rows=1 width=16) (actual time=0.065..0.066 rows=1 loops=1)
           ->  Index Only Scan using idx_evm_log_poller_blocks_order_by_block on log_poller_blocks  (cost=0.43..101902.73 rows=106056 width=16) (actual time=0.064..0.065 rows=1 loops=1)
                 Index Cond: (evm_chain_id = '11155111'::numeric)
                 Heap Fetches: 1
   ->  Index Scan using idx_evm_logs_ordered_by_block_and_created_at on logs  (cost=0.56..80.07 rows=1 width=367) (actual time=0.116..2.763 rows=51 loops=1)
         Index Cond: ((evm_chain_id = '11155111'::numeric) AND (address = '\x000b26f604eaadc3d874a4404bde6d64a97d95ca'::bytea) AND (event_sig = '\xd4f851956a5d67c3997d1c9205045fef79bae2947fdee7e9e2641abc7391ef65'::bytea) AND (block_number <= $0))
         Filter: (((topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c5a'::bytea) AND (topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c5b'::bytea)) OR ((topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c5d'::bytea) AND (topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c5e'::bytea)) OR ((topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c61'::bytea) AND (topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c63'::bytea)) OR ((topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c67'::bytea) AND (topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c6a'::bytea)) OR ((topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c6f'::bytea) AND (topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c78'::bytea)) OR ((topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c81'::bytea) AND (topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c87'::bytea)) OR ((topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002c8d'::bytea) AND (topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002c93'::bytea)) OR ((topics[2] >= '\x0000000000000000000000000000000000000000000000000000000000002eaf'::bytea) AND (topics[2] <= '\x0000000000000000000000000000000000000000000000000000000000002ec0'::bytea)))
         Rows Removed by Filter: 1379
 Planning Time: 0.441 ms
 Execution Time: 2.838 ms
(14 rows)

PR in the chainlink-common enabling new function in readers smartcontractkit/chainlink-common#633

[ilija42]

Can this feature be added to this PR?

[mateusz-sekara]

Yeah, I can add it there ;)

[mateusz-sekara]

smartcontractkit/chainlink-common#654

[winder]

I wrote a similar function that might be helpful here: https://github.com/smartcontractkit/chainlink-ccip/blob/d88fcd5b95bf952c57eb33d41f6aab7afb850331/execute/plugin_functions.go#L83

[mateusz-sekara]

Nice, I guess I need to copy paste that code then :D

[cl-sonarqube-production]

Quality Gate failed

Failed conditions
11.9% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR was closed because it has been stalled for 10 days with no activity.