-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #318 from trilitech/staging
Update main from staging, Feb 13
- Loading branch information
Showing
22 changed files
with
518 additions
and
275 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| --- | ||
| title: The Data Availability Layer | ||
| authors: "Tim McMackin" | ||
| last_update: | ||
| date: 7 February 2024 | ||
| --- | ||
|
|
||
| :::note Experimental | ||
| The Data Availability Layer is an experimental feature that is not yet available on Tezos Mainnet. | ||
| The way the DAL works may change significantly before it is generally available. | ||
| ::: | ||
|
|
||
| The Data Availability Layer (DAL) is a companion peer-to-peer network for the Tezos blockchain, designed to provide additional data bandwidth to Smart Rollups. | ||
| It allows users to share large amounts of data in a way that is decentralized and permissionless, because anyone can join the network and post and read data on it. | ||
|
|
||
| For a tutorial on how to use the DAL, see [Implement a file archive with the DAL and a Smart Rollup](../tutorials/build-files-archive-with-dal). | ||
|
|
||
| ## How the DAL works | ||
|
|
||
| The DAL relies on a network of DAL nodes that distribute data via a peer-to-peer network. | ||
| Layer 1 bakers verify that the data is available. | ||
| After the bakers attest that the data is available, the DAL nodes provide the data to Smart Rollups. | ||
| Smart Rollups that need the data must use it or store it promptly, because it is available only temporarily on the DAL. | ||
|
|
||
| The DAL works like this: | ||
|
|
||
| 1. Users post data to a DAL node. | ||
| 1. The DAL node returns a certificate, which includes two parts: | ||
|
|
||
| - The _commitment_ is like a hash of the data but has the additional ability to identify individual shards of the data and reconstruct the original data from a certain percentage of the shards. | ||
| The number of shards needed depends on how the data is spread across shards, which is controlled by a parameter called the _redundancy factor_. | ||
| - The _proof_ certifies the length of the data to prevent malicious users from overloading the layer with data. | ||
|
|
||
| 1. Users post the certificate to Tezos layer 1 via the Octez client. | ||
| 1. When the certificate is confirmed in a block, the DAL splits the data into shards and shares it through the peer-to-peer network. | ||
| 1. Layer 1 assigns the shards to bakers. | ||
| 1. Bakers verify that they are able to download the shards that they are assigned to. | ||
| 1. Bakers attest that the data is available in their usual block attestations to layer 1. | ||
|
|
||
| Each Tezos network has a delay of a certain number of blocks known as the _attestation lag_. | ||
| This number of blocks determines when bakers attest that the data is available and when the data becomes available to Smart Rollups. | ||
| For example, if a certificate is included in level 100 and the attestation lag is 4, bakers must attest that the data is available in level 104, along with their usual attestations that build on level 103. | ||
|
|
||
| If enough shards are attested in that level, the data becomes available to Smart Rollups at the end of layer 104. | ||
| If not enough shards are attested in that level, the certificate is considered bogus and the related data is dropped. | ||
|
|
||
| 1. The Smart Rollup node monitors the blocks and when it sees attested DAL data, it connects to a DAL node to request the data. | ||
| Smart Rollups must store the data if they need it because it is available on the DAL for only a short time. | ||
|
|
||
| The overall workflow is summarized in the following figure: | ||
|
|
||
|  | ||
| <!-- https://lucid.app/lucidchart/cc422278-7319-4a2f-858a-a7b72e1ea3a6/edit --> | ||
|
|
||
| ## Data structure | ||
|
|
||
| Internally, the Data Availability Layer stores information about the available data in layer 1 blocks. | ||
| Each block has several byte-vectors called _slots_, each with a maximum size. | ||
| DAL users can add information about the available data as _pages_ in these slots, as shown in this figure: | ||
|
|
||
|  | ||
| <!-- https://lucid.app/lucidchart/46fa8412-8443-4491-82f6-305aafaf85f2/edit --> | ||
|
|
||
| The data in a slot is broken into pages to ensure that each piece of data can fit in a single Tezos operation. | ||
| This data must fit in a single operation to allow the Smart Rollup refutation game to work, in which every execution step of the Smart Rollup must be provable to layer 1. | ||
|
|
||
| When clients publish data, they must specify which slot to add it to. | ||
| Note that because the DAL is permissionless, clients may try to add data to the same slot in the same block. | ||
| In this case, the first operation in the block takes precedence, which leaves the baker that creates the block in control of which data makes it into the block. | ||
| Other operations that try to add data to the same slot fail. | ||
|
|
||
| The number and size of these slots can change. | ||
| Different networks can have different DAL parameters. | ||
| Future changes to the protocol may allow the DAL to resize dynamically based on usage. | ||
|
|
||
| ## Getting the DAL parameters | ||
|
|
||
| Clients can get information about the current DAL parameters from the RPC endpoint `GET /chains/main/blocks/head/context/constants` or the Smart Rollup kernel SDK function `reveal_dal_parameters`. | ||
| These parameters include: | ||
|
|
||
| - `number_of_slots`: The maximum number of slots in each block | ||
| - `slot_size`: The size of each slot in bytes | ||
| - `page_size`: The size of each page in bytes | ||
| - `attestation_lag`: The number of blocks after a certificate is published that bakers attest that the data is available; if enough attestations are available in this block, the data becomes available to Smart Rollups | ||
| - `redundancy_factor`: How much redundancy is used to split the data into shards; for example, a redundancy factor of 2 means that half of all shards are enough to reconstruct the original data and a redundancy factor of 4 means that 25% of all shards are required | ||
|
|
||
| ## Sending data to the DAL | ||
|
|
||
| Sending data to the DAL is a two-step process: | ||
|
|
||
| 1. Send the data to a DAL node by passing it to its `POST /slot` endpoint, as in this example: | ||
|
|
||
| ```bash | ||
| curl -X POST http://dal-node.example.com:10732/slot --data '"Hello, world!"' -H 'Content-Type: application/json' | ||
| ``` | ||
|
|
||
| The DAL node returns the commitment and proof of the data, as in this abbreviated example: | ||
|
|
||
| ```json | ||
| { | ||
| "commitment": "sh1u3tr3YKPDY", | ||
| "commitment_proof": "8229c63b8e858d9a9" | ||
| } | ||
| ``` | ||
|
|
||
| 1. Send an operation to include the commitment and proof in a block by running this Octez client command, using an RPC endpoint for `$ENDPOINT` and an account alias or address for `$MY_ACCOUNT`: | ||
|
|
||
| ```bash | ||
| commitment="sh1u3tr3YKPDY" | ||
| proof="8229c63b8e858d9a9" | ||
| octez-client --endpoint ${ENDPOINT} \ | ||
| publish dal commitment "${commitment}" from ${MY_ACCOUNT} for slot 10 \ | ||
| with proof "${proof}" | ||
| ``` | ||
|
|
||
| For an example of sending larger amounts of data, see [Implement a file archive with the DAL and a Smart Rollup](../tutorials/build-files-archive-with-dal). | ||
|
|
||
| ## Getting data from the DAL | ||
|
|
||
| Smart Rollups can use data from the DAL **only after it has been attested by the bakers**. | ||
| Therefore, they cannot access DAL data in the current level, because not enough blocks have elapsed to allow bakers to attest the data. | ||
|
|
||
| The latest level that Smart Rollups can access is the current level minus the attestation lag. | ||
| They can access the data in that level with the Smart Rollup kernel SDK function `reveal_dal_page`, which accepts the level, slot, and page to receive, as in this example: | ||
|
|
||
| ```rust | ||
| let param = host.reveal_dal_parameters(); | ||
|
|
||
| let sol = host.read_input()?.unwrap(); | ||
|
|
||
| let target_level = sol.level as usize - param.attestation_lag as usize; | ||
|
|
||
| let mut buffer = vec![0u8; param.page_size as usize]; | ||
|
|
||
| let bytes_read = host.reveal_dal_page(target_level as i32, slot_index, 0, &mut buffer)?; | ||
|
|
||
| if 0 < bytes_read { | ||
| debug_msg!( | ||
| host, | ||
| "Attested slot at index {} for level {}: {:?}\n", | ||
| slot_index, | ||
| target_level, | ||
| &buffer.as_slice()[0..10] | ||
| ); | ||
| } else { | ||
| debug_msg!( | ||
| host, | ||
| "No attested slot at index {} for level {}\n", | ||
| slot_index, | ||
| target_level | ||
| ); | ||
| } | ||
| ``` | ||
|
|
||
| ## Reference | ||
|
|
||
| For more information about the DAL, see [DAL overview](https://tezos.gitlab.io/shell/dal_overview.html) in the Octez documentation. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| --- | ||
| title: Implement a file archive with the DAL and a Smart Rollup | ||
| authors: 'Tezos Core Developers' | ||
| last_update: | ||
| date: 9 February 2024 | ||
| --- | ||
|
|
||
| :::note Experimental | ||
| The Data Availability Layer is an experimental feature that is not yet available on Tezos Mainnet. | ||
| The way the DAL works may change significantly before it is generally available. | ||
| ::: | ||
|
|
||
| The Data Availability Layer (DAL) is a companion peer-to-peer network for the Tezos blockchain, designed to provide additional data bandwidth to Smart Rollups. | ||
| It allows users to share large amounts of data in a way that is decentralized and permissionless, because anyone can join the network and post and read data on it. | ||
|
|
||
| In this tutorial, you will set up a file archive that stores and retrieves files with the DAL. | ||
| You will learn: | ||
|
|
||
| - How data is organized and shared with the DAL and the reveal data channel | ||
| - How to read data from the DAL in a Smart Rollup | ||
| - How to host a DAL node | ||
| - How to publish data and files with the DAL | ||
|
|
||
| Because the DAL is not yet available on Tezos Mainnet, this tutorial uses the [Weeklynet test network](https://teztnets.com/weeklynet-about), which runs on a newer version of the protocol that includes the DAL. | ||
| Weeklynet runs just like other Tezos networks like Mainnet and Ghostnet, with its own nodes, bakers, and accusers, so you don't need to run your own nodes and bakers. | ||
|
|
||
| See these links for more information about the DAL: | ||
|
|
||
| - For technical information about how the DAL works, see [Data Availability Layer](https://tezos.gitlab.io/shell/dal.html) in the Octez documentation. | ||
| - For more information about the approach for the DAL, see [The Rollup Booster: A Data-Availability Layer for Tezos](https://research-development.nomadic-labs.com/data-availability-layer-tezos.html). | ||
|
|
||
| ## Tutorial applications | ||
|
|
||
| In this tutorial, you set up these components: | ||
|
|
||
| - The Octez client, which you use to manage a local wallet, deploy a Smart Rollup, and send data to the DAL | ||
| - A Data Availability Layer node (not to be confused with a layer 1 node), which stores data temporarily and distributes it to Smart Rollups | ||
| - A Smart Rollup that listens for data published to the DAL, retrieves it from the DAL node, and stores it locally | ||
| - A Smart Rollup node that runs your Smart Rollup | ||
|
|
||
| For simplicity, you do not set up a layer 1 node or a baker, which are responsible for verifying that the data is available before Smart Rollups can access it. | ||
| Instead, you use the existing nodes and bakers that are running on Weeklynet. | ||
|
|
||
| ## Tutorial diagram | ||
|
|
||
| Here is a diagram that shows the components that you set up in this tutorial in a light blue background: | ||
|
|
||
|  | ||
|
|
||
| <!-- https://lucid.app/lucidchart/58f5577e-91b5-4237-89c4-a8cdf81c71ad/edit --> | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| This article assumes some familiarity with Smart Rollups. | ||
| If you are new to Smart Rollups, see the tutorial [Deploy a Smart Rollup](./smart-rollup). | ||
|
|
||
| ## Why the DAL? | ||
|
|
||
| The DAL has earned the nickname "Rollup Booster" from its ability to address | ||
| the last bottleneck Smart Rollups developers could not overcome without | ||
| sacrificing decentralization: block space. Smart Rollups offload | ||
| *computation* from layer 1, but the transactions that they process still need to | ||
| originate from somewhere. | ||
|
|
||
| By default, that "somewhere" is the layer 1 blocks, yet the size of a Tezos | ||
| block is limited to around 500KBytes. In this model, while Smart Rollups do not | ||
| compete for layer 1 gas anymore, they still compete for block space. | ||
|
|
||
| The DAL allows third parties to publish data and have bakers attest that the data is available. | ||
| When enough bakers have attested that the data is available, Smart Rollups can retrieve the data without the need for additional trusted third-parties. | ||
|
|
||
| ## How the DAL works | ||
|
|
||
| In this tutorial, you create a file archive application that allows clients to upload data to the DAL. | ||
| You also create a Smart Rollup that listens to the DAL and responds to that data. | ||
|
|
||
| The DAL works like this: | ||
|
|
||
| 1. Users post data to a DAL node. | ||
| 1. The DAL node returns a certificate, which includes two parts: | ||
|
|
||
| - The _commitment_ is like a hash of the data but has the additional ability to identify individual shards of the data and reconstruct the original data from a certain percentage of the shards. | ||
| The number of shards needed depends on how the data is spread across shards, which is controlled by a parameter called the _redundancy factor_. | ||
| - The _proof_ certifies the length of the data to prevent malicious users from overloading the layer with data. | ||
|
|
||
| 1. Users post the certificate to Tezos layer 1 via the Octez client. | ||
| 1. When the certificate is confirmed in a block, the DAL splits the data into shards and shares it through the peer-to-peer network. | ||
| 1. Layer 1 assigns the shards to bakers. | ||
| 1. Bakers verify that they are able to download the shards that they are assigned to. | ||
| 1. Bakers attest that the data is available in their usual block attestations to layer 1. | ||
|
|
||
| Each Tezos network has a delay of a certain number of blocks known as the _attestation lag_. | ||
| This number of blocks determines when bakers attest that the data is available and when the data becomes available to Smart Rollups. | ||
| For example, if a certificate is included in level 100 and the attestation lag is 4, bakers must attest that the data is available in level 104, along with their usual attestations that build on level 103. | ||
|
|
||
| If enough shards are attested in that level, the data becomes available to Smart Rollups at the end of layer 104. | ||
| If not enough shards are attested in that level, the certificate is considered bogus and the related data is dropped. | ||
|
|
||
| 1. The Smart Rollup node monitors the blocks and when it sees attested DAL data, it connects to a DAL node to request the data. | ||
| Smart Rollups must store the data if they need it because it is available on the DAL for only a short time. | ||
|
|
||
| The overall workflow is summarized in the following figure: | ||
|
|
||
|  | ||
| <!-- https://lucid.app/lucidchart/cc422278-7319-4a2f-858a-a7b72e1ea3a6/edit --> | ||
|
|
||
| There are many steps in the DAL process, but the most complicated parts (storing and sharing data) are handled automatically by the various daemons in the Octez suite. | ||
|
|
||
| :::note | ||
| When you install a Smart Rollup, you provide only the installer kernel on layer 1 and the full kernel via the reveal data channel. | ||
| Currently, you cannot send the full kernel data over the Data Availability Layer, so this tutorial relies on the reveal data channel to install the kernel as usual. | ||
| ::: | ||
|
|
||
| When you are ready, get started by going to [Part 1: Setting up an environment](./build-files-archive-with-dal/set-up-environment). |
Oops, something went wrong.