Merge branch 'dev' into sz-stablecontainer

ethereum · Jun 19, 2024 · 0adb7ae · 0adb7ae
2 parents 8dc34bc + 29d3a24
commit 0adb7ae
Show file tree

Hide file tree

Showing 28 changed files with 965 additions and 701 deletions.
diff --git a/.github/workflows/generate_vectors.yml b/.github/workflows/generate_vectors.yml
@@ -0,0 +1,78 @@
+name: Run test vector generation
+
+defaults:
+  run:
+    shell: zsh {0}
+
+on:
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: The branch, tag or SHA to checkout and build from
+        default: dev
+        type: string
+        required: true
+
+jobs:
+  generate-tests:
+    runs-on: [self-hosted-ghr-custom, size-chungus-x64, profile-consensusSpecs]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          repository: 'ethereum/consensus-specs'
+          path: 'consensus-specs'
+          ref: ${{ inputs.source_ref }}
+      - name: Checkout consensus-spec-tests repository
+        uses: actions/checkout@v4
+        with:
+          repository: 'ethereum/consensus-spec-tests'
+          path: 'consensus-spec-tests'
+          fetch-depth: 1
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+          cache: ''
+      - name: Clean up Spec Repository
+        run: |
+          cd consensus-specs
+          make clean
+      - name: Install dependencies and generate pyspec
+        run: |
+          cd consensus-specs
+          make install_test
+          make -B pyspec
+      - name: Generate tests
+        run: |
+          cd consensus-specs
+          make -j 16 generate_tests 2>&1 | tee ../consensustestgen.log
+          cp -r presets/ ../consensus-spec-tests/presets
+          cp -r configs/ ../consensus-spec-tests/configs
+          find . -type d -empty -delete
+      - name: Archive configurations
+        run: |
+          cd consensus-spec-tests
+          tar -czvf general.tar.gz tests/general
+          tar -czvf minimal.tar.gz tests/minimal
+          tar -czvf mainnet.tar.gz tests/mainnet
+      - name: Upload general.tar.gz
+        uses: actions/upload-artifact@v4
+        with:
+          name: General Test Configuration
+          path: consensus-spec-tests/general.tar.gz
+      - name: Upload minimal.tar.gz
+        uses: actions/upload-artifact@v4
+        with:
+          name: Minimal Test Configuration
+          path: consensus-spec-tests/minimal.tar.gz
+      - name: Upload mainnet.tar.gz
+        uses: actions/upload-artifact@v4
+        with:
+          name: Mainnet Test Configuration
+          path: consensus-spec-tests/mainnet.tar.gz
+      - name: Upload consensustestgen
+        uses: actions/upload-artifact@v4
+        with:
+          name: consensustestgen.log
+          path: consensustestgen.log
diff --git a/scripts/build_run_docker_tests.sh b/scripts/build_run_docker_tests.sh
@@ -1,4 +1,4 @@
-#! /bin/sh
+#! /bin/bash
 
 # Run 'consensus-specs' tests from a docker container instance.
 # *Be sure to launch Docker before running this script.*

diff --git a/setup.py b/setup.py
@@ -552,7 +552,7 @@ def run(self):
         "pycryptodome==3.15.0",
         "py_ecc==6.0.0",
         "milagro_bls_binding==1.9.0",
-        "remerkleable==0.1.27",
+        "remerkleable==0.1.28",
         "trie==2.0.2",
         RUAMEL_YAML_VERSION,
         "lru-dict==1.2.0",

diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md
@@ -17,6 +17,7 @@
   - [Custody setting](#custody-setting)
   - [Containers](#containers)
     - [`DataColumnSidecar`](#datacolumnsidecar)
+    - [`MatrixEntry`](#matrixentry)
   - [Helper functions](#helper-functions)
     - [`get_custody_columns`](#get_custody_columns)
     - [`compute_extended_matrix`](#compute_extended_matrix)
@@ -53,12 +54,10 @@ The following values are (non-configurable) constants used throughout the specif
 
 ## Custom types
 
-We define the following Python custom types for type hinting and readability:
-
 | Name | SSZ equivalent | Description |
 | - | - | - |
-| `DataColumn` | `List[Cell, MAX_BLOB_COMMITMENTS_PER_BLOCK]` | The data of each column in EIP-7594 |
-| `ExtendedMatrix` | `List[Cell, MAX_CELLS_IN_EXTENDED_MATRIX]` | The full data of one-dimensional erasure coding extended blobs (in row major format). |
+| `RowIndex` | `uint64` | Row identifier in the matrix of cells |
+| `ColumnIndex` | `uint64` | Column identifier in the matrix of cells |
 
 ## Configuration
 
@@ -79,7 +78,7 @@ We define the following Python custom types for type hinting and readability:
 
 | Name | Value | Description |
 | - | - | - |
-| `SAMPLES_PER_SLOT` | `8` | Number of `DataColumn` random samples a node queries per slot |
+| `SAMPLES_PER_SLOT` | `8` | Number of `DataColumnSidecar` random samples a node queries per slot |
 | `CUSTODY_REQUIREMENT` | `1` | Minimum number of subnets an honest node custodies and serves samples from |
 | `TARGET_NUMBER_OF_PEERS` | `70` | Suggested minimum peer count |
 
@@ -90,13 +89,23 @@ We define the following Python custom types for type hinting and readability:
 ```python
 class DataColumnSidecar(Container):
     index: ColumnIndex  # Index of column in extended matrix
-    column: DataColumn
+    column: List[Cell, MAX_BLOB_COMMITMENTS_PER_BLOCK]
     kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]
     kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK]
     signed_block_header: SignedBeaconBlockHeader
     kzg_commitments_inclusion_proof: Vector[Bytes32, KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH]
 ```
 
+#### `MatrixEntry`
+
+```python
+class MatrixEntry(Container):
+    cell: Cell
+    kzg_proof: KZGProof
+    column_index: ColumnIndex
+    row_index: RowIndex
+```
+
 ### Helper functions
 
 #### `get_custody_columns`
@@ -132,37 +141,52 @@ def get_custody_columns(node_id: NodeID, custody_subnet_count: uint64) -> Sequen
 #### `compute_extended_matrix`
 
 ```python
-def compute_extended_matrix(blobs: Sequence[Blob]) -> ExtendedMatrix:
+def compute_extended_matrix(blobs: Sequence[Blob]) -> List[MatrixEntry, MAX_CELLS_IN_EXTENDED_MATRIX]:
     """
     Return the full ``ExtendedMatrix``.
 
     This helper demonstrates the relationship between blobs and ``ExtendedMatrix``.
     The data structure for storing cells is implementation-dependent.
     """
     extended_matrix = []
-    for blob in blobs:
-        extended_matrix.extend(compute_cells(blob))
-    return ExtendedMatrix(extended_matrix)
+    for blob_index, blob in enumerate(blobs):
+        cells, proofs = compute_cells_and_kzg_proofs(blob)
+        for cell_index, (cell, proof) in enumerate(zip(cells, proofs)):
+            extended_matrix.append(MatrixEntry(
+                cell=cell,
+                kzg_proof=proof,
+                row_index=blob_index,
+                column_index=cell_index,
+            ))
+    return extended_matrix
 ```
 
 #### `recover_matrix`
 
 ```python
-def recover_matrix(cells_dict: Dict[Tuple[BlobIndex, CellID], Cell], blob_count: uint64) -> ExtendedMatrix:
+def recover_matrix(partial_matrix: Sequence[MatrixEntry],
+                   blob_count: uint64) -> List[MatrixEntry, MAX_CELLS_IN_EXTENDED_MATRIX]:
     """
-    Return the recovered ``ExtendedMatrix``.
+    Return the recovered extended matrix.
 
-    This helper demonstrates how to apply ``recover_all_cells``.
+    This helper demonstrates how to apply ``recover_cells_and_kzg_proofs``.
     The data structure for storing cells is implementation-dependent.
     """
-    extended_matrix: List[Cell] = []
+    extended_matrix = []
     for blob_index in range(blob_count):
-        cell_ids = [cell_id for b_index, cell_id in cells_dict.keys() if b_index == blob_index]
-        cells = [cells_dict[(BlobIndex(blob_index), cell_id)] for cell_id in cell_ids]
-
-        all_cells_for_row = recover_all_cells(cell_ids, cells)
-        extended_matrix.extend(all_cells_for_row)
-    return ExtendedMatrix(extended_matrix)
+        cell_indices = [e.column_index for e in partial_matrix if e.row_index == blob_index]
+        cells = [e.cell for e in partial_matrix if e.row_index == blob_index]
+        proofs = [e.kzg_proof for e in partial_matrix if e.row_index == blob_index]
+
+        recovered_cells, recovered_proofs = recover_cells_and_kzg_proofs(cell_indices, cells, proofs)
+        for cell_index, (cell, proof) in enumerate(zip(recovered_cells, recovered_proofs)):
+            extended_matrix.append(MatrixEntry(
+                cell=cell,
+                kzg_proof=proof,
+                row_index=blob_index,
+                column_index=cell_index,
+            ))
+    return extended_matrix
 ```
 
 #### `get_data_column_sidecars`
@@ -182,15 +206,15 @@ def get_data_column_sidecars(signed_block: SignedBeaconBlock,
     proofs = [cells_and_proofs[i][1] for i in range(blob_count)]
     sidecars = []
     for column_index in range(NUMBER_OF_COLUMNS):
-        column = DataColumn([cells[row_index][column_index]
-                             for row_index in range(blob_count)])
-        kzg_proof_of_column = [proofs[row_index][column_index]
-                               for row_index in range(blob_count)]
+        column_cells = [cells[row_index][column_index]
+                        for row_index in range(blob_count)]
+        column_proofs = [proofs[row_index][column_index]
+                         for row_index in range(blob_count)]
         sidecars.append(DataColumnSidecar(
             index=column_index,
-            column=column,
+            column=column_cells,
             kzg_commitments=block.body.blob_kzg_commitments,
-            kzg_proofs=kzg_proof_of_column,
+            kzg_proofs=column_proofs,
             signed_block_header=signed_block_header,
             kzg_commitments_inclusion_proof=kzg_commitments_inclusion_proof,
         ))
@@ -283,7 +307,7 @@ Such trailing techniques and their analysis will be valuable for any DAS constru
 
 ### Row (blob) custody
 
-In the one-dimension construction, a node samples the peers by requesting the whole `DataColumn`. In reconstruction, a node can reconstruct all the blobs by 50% of the columns. Note that nodes can still download the row via `blob_sidecar_{subnet_id}` subnets.
+In the one-dimension construction, a node samples the peers by requesting the whole `DataColumnSidecar`. In reconstruction, a node can reconstruct all the blobs by 50% of the columns. Note that nodes can still download the row via `blob_sidecar_{subnet_id}` subnets.
 
 The potential benefits of having row custody could include:
 

diff --git a/specs/_features/eip7594/p2p-interface.md b/specs/_features/eip7594/p2p-interface.md
@@ -28,7 +28,7 @@
       - [DataColumnSidecarsByRange v1](#datacolumnsidecarsbyrange-v1)
   - [The discovery domain: discv5](#the-discovery-domain-discv5)
     - [ENR structure](#enr-structure)
-      - [`custody_subnet_count`](#custody_subnet_count)
+      - [Custody subnet count](#custody-subnet-count)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 <!-- /TOC -->
@@ -284,10 +284,10 @@ After the initial data column sidecar, clients MAY stop in the process of respon
 
 #### ENR structure
 
-##### `custody_subnet_count`
+##### Custody subnet count
 
-A new field is added to the ENR under the key `custody_subnet_count` to facilitate custody data column discovery.
+A new field is added to the ENR under the key `csc` to facilitate custody data column discovery.
 
-| Key                    | Value        |
-|:-----------------------|:-------------|
-| `custody_subnet_count` | SSZ `uint64` |
+| Key   | Value                                    |
+|:------|:-----------------------------------------|
+| `csc` | Custody subnet count, big endian integer |