Update main from staging, Feb 13

docs/architecture/data-availability-layer.md

@@ -0,0 +1,157 @@
+---
+title: The Data Availability Layer
+authors: "Tim McMackin"
+last_update:
+  date: 7 February 2024
+---
+
+:::note Experimental
+The Data Availability Layer is an experimental feature that is not yet available on Tezos Mainnet.
+The way the DAL works may change significantly before it is generally available.
+:::
+
+The Data Availability Layer (DAL) is a companion peer-to-peer network for the Tezos blockchain, designed to provide additional data bandwidth to Smart Rollups.
+It allows users to share large amounts of data in a way that is decentralized and permissionless, because anyone can join the network and post and read data on it.
+
+For a tutorial on how to use the DAL, see [Implement a file archive with the DAL and a Smart Rollup](../tutorials/build-files-archive-with-dal).
+
+## How the DAL works
+
+The DAL relies on a network of DAL nodes that distribute data via a peer-to-peer network.
+Layer 1 bakers verify that the data is available.
+After the bakers attest that the data is available, the DAL nodes provide the data to Smart Rollups.
+Smart Rollups that need the data must use it or store it promptly, because it is available only temporarily on the DAL.
+
+The DAL works like this:
+
+1. Users post data to a DAL node.
+1. The DAL node returns a certificate, which includes two parts:
+
+   - The _commitment_ is like a hash of the data but has the additional ability to identify individual shards of the data and reconstruct the original data from a certain percentage of the shards.
+   The number of shards needed depends on how the data is spread across shards, which is controlled by a parameter called the _redundancy factor_.
+   - The _proof_ certifies the length of the data to prevent malicious users from overloading the layer with data.
+
+1. Users post the certificate to Tezos layer 1 via the Octez client.
+1. When the certificate is confirmed in a block, the DAL splits the data into shards and shares it through the peer-to-peer network.
+1. Layer 1 assigns the shards to bakers.
+1. Bakers verify that they are able to download the shards that they are assigned to.
+1. Bakers attest that the data is available in their usual block attestations to layer 1.
+
+   Each Tezos network has a delay of a certain number of blocks known as the _attestation lag_.
+   This number of blocks determines when bakers attest that the data is available and when the data becomes available to Smart Rollups.
+   For example, if a certificate is included in level 100 and the attestation lag is 4, bakers must attest that the data is available in level 104, along with their usual attestations that build on level 103.
+
+   If enough shards are attested in that level, the data becomes available to Smart Rollups at the end of layer 104.
+   If not enough shards are attested in that level, the certificate is considered bogus and the related data is dropped.
+
+1. The Smart Rollup node monitors the blocks and when it sees attested DAL data, it connects to a DAL node to request the data.
+Smart Rollups must store the data if they need it because it is available on the DAL for only a short time.
+
+The overall workflow is summarized in the following figure:
+
+![Overall diagram of the workflow of the Data Availability Layer](/img/architecture/dal-workflow.png)
+<!-- https://lucid.app/lucidchart/cc422278-7319-4a2f-858a-a7b72e1ea3a6/edit -->
+
+## Data structure
+
+Internally, the Data Availability Layer stores information about the available data in layer 1 blocks.
+Each block has several byte-vectors called _slots_, each with a maximum size.
+DAL users can add information about the available data as _pages_ in these slots, as shown in this figure:
+
+![Two example blocks with different DAL slots in use in each](/img/architecture/dal-slots-in-blocks.png)
+<!-- https://lucid.app/lucidchart/46fa8412-8443-4491-82f6-305aafaf85f2/edit -->
+
+The data in a slot is broken into pages to ensure that each piece of data can fit in a single Tezos operation.
+This data must fit in a single operation to allow the Smart Rollup refutation game to work, in which every execution step of the Smart Rollup must be provable to layer 1.
+
+When clients publish data, they must specify which slot to add it to.
+Note that because the DAL is permissionless, clients may try to add data to the same slot in the same block.
+In this case, the first operation in the block takes precedence, which leaves the baker that creates the block in control of which data makes it into the block.
+Other operations that try to add data to the same slot fail.
+
+The number and size of these slots can change.
+Different networks can have different DAL parameters.
+Future changes to the protocol may allow the DAL to resize dynamically based on usage.
+
+## Getting the DAL parameters
+
+Clients can get information about the current DAL parameters from the RPC endpoint `GET /chains/main/blocks/head/context/constants` or the Smart Rollup kernel SDK function `reveal_dal_parameters`.
+These parameters include:
+
+- `number_of_slots`: The maximum number of slots in each block
+- `slot_size`: The size of each slot in bytes
+- `page_size`: The size of each page in bytes
+- `attestation_lag`: The number of blocks after a certificate is published that bakers attest that the data is available; if enough attestations are available in this block, the data becomes available to Smart Rollups
+- `redundancy_factor`: How much redundancy is used to split the data into shards; for example, a redundancy factor of 2 means that half of all shards are enough to reconstruct the original data and a redundancy factor of 4 means that 25% of all shards are required
+
+## Sending data to the DAL
+
+Sending data to the DAL is a two-step process:
+
+1. Send the data to a DAL node by passing it to its `POST /slot` endpoint, as in this example:
+
+   ```bash
+   curl -X POST http://dal-node.example.com:10732/slot --data '"Hello, world!"' -H 'Content-Type: application/json'
+   ```
+
+   The DAL node returns the commitment and proof of the data, as in this abbreviated example:
+
+   ```json
+   {
+     "commitment": "sh1u3tr3YKPDY",
+     "commitment_proof": "8229c63b8e858d9a9"
+   }
+   ```
+
+1. Send an operation to include the commitment and proof in a block by running this Octez client command, using an RPC endpoint for `$ENDPOINT` and an account alias or address for `$MY_ACCOUNT`:
+
+   ```bash
+   commitment="sh1u3tr3YKPDY"
+   proof="8229c63b8e858d9a9"
+   octez-client --endpoint ${ENDPOINT} \
+     publish dal commitment "${commitment}" from ${MY_ACCOUNT} for slot 10 \
+     with proof "${proof}"
+   ```
+
+For an example of sending larger amounts of data, see [Implement a file archive with the DAL and a Smart Rollup](../tutorials/build-files-archive-with-dal).
+
+## Getting data from the DAL
+
+Smart Rollups can use data from the DAL **only after it has been attested by the bakers**.
+Therefore, they cannot access DAL data in the current level, because not enough blocks have elapsed to allow bakers to attest the data.
+
+The latest level that Smart Rollups can access is the current level minus the attestation lag.
+They can access the data in that level with the Smart Rollup kernel SDK function `reveal_dal_page`, which accepts the level, slot, and page to receive, as in this example:
+
+```rust
+let param = host.reveal_dal_parameters();
+
+let sol = host.read_input()?.unwrap();
+
+let target_level = sol.level as usize - param.attestation_lag as usize;
+
+let mut buffer = vec![0u8; param.page_size as usize];
+
+let bytes_read = host.reveal_dal_page(target_level as i32, slot_index, 0, &mut buffer)?;
+
+if 0 < bytes_read {
+    debug_msg!(
+        host,
+        "Attested slot at index {} for level {}: {:?}\n",
+        slot_index,
+        target_level,
+        &buffer.as_slice()[0..10]
+    );
+} else {
+    debug_msg!(
+        host,
+        "No attested slot at index {} for level {}\n",
+        slot_index,
+        target_level
+    );
+}
+```
+
+## Reference
+
+For more information about the DAL, see [DAL overview](https://tezos.gitlab.io/shell/dal_overview.html) in the Octez documentation.

docs/architecture/rpc.md

@@ -2,7 +2,7 @@
 title: The RPC protocol
 authors: "Tim McMackin"
 last_update:
-  date: 6 November 2023
+  date: 6 February 2023
 ---
 
 The Tezos RPC (Remote Procedure Call) protocol is a specification for a REST API that clients use to interact with Tezos nodes and nodes use to communicate with each other.
@@ -11,7 +11,10 @@ Tezos nodes act as servers and accept HTTP requests from clients and other nodes
 
 Tezos RPC uses JSON to send and receive data, but it does not adhere to the JSON-RPC specification.
 
-For a list of the endpoints in the Tezos RPC protocol, see [Shell RPCs - Reference](https://tezos.gitlab.io/shell/rpc.html#rpc-index-shell) in the Octez documentation.
+Different parts of the software provide different RPC endpoints:
+
+- For RPC endpoints for the shell, see [Shell RPCs - Reference](https://tezos.gitlab.io/shell/rpc.html#rpc-index-shell) in the Octez documentation.
+- For RPC endpoints for the active version of the protocol, see https://tezos.gitlab.io/active/rpc.html.
 
 ## Public and private RPC nodes
 

docs/architecture/smart-rollups.mdx → docs/architecture/smart-rollups.md

@@ -5,8 +5,6 @@ last_update:
   date: 18 January 2024
 ---
 
-import LucidDiagram from '@site/src/components/LucidDiagram';
-
 Smart Rollups play a crucial part in providing high scalability on Tezos.
 They handle logic in a separate environment that can run transactions at a much higher rate and can use larger amounts of data than the main Tezos network.
 
@@ -26,7 +24,8 @@ For reference on Smart Rollups, see [Smart Optimistic Rollups](https://tezos.git
 
 This diagram shows a high-level view of how Smart Rollups interact with layer 1:
 
-<LucidDiagram width="640px" height="480px" src="https://lucid.app/documents/embedded/1e176e48-5c1a-457c-af3e-2f66d3c1b893" id="iB96FUquB~sm" />
+![Diagram of Smart Rollup architecture](/img/architecture/smart-rollup-architecture.png)
+<!-- https://lucid.app/lucidchart/1e176e48-5c1a-457c-af3e-2f66d3c1b893/edit>
 
 ## Uses for Smart Rollups
 
@@ -79,14 +78,14 @@ To transfer more than 4KB of data, rollups must use multiple pages, which may co
 
 - A rollup node can request information about the rollup, including the address and origination level of the rollup, known as _metadata requests_.
 
-{/*
+<!--
 TODO how is this data provided?
 Where does it come from?
 Do we need instructions on how to provide data?
 Eventually include:
   - importing data from a DAC certificate (which can contain anything ultimately, including a kernel to upgrade to)
   - revealing data from the (WIP) DAL
-*/}
+-->
 
 ## Smart Rollup lifecycle
 
@@ -99,7 +98,7 @@ The general flow of a Smart Rollup goes through these phases:
 
 Here is more information on each of these phases:
 
-{/* TODO diagram of commitment periods and refutation periods? */}
+<!-- TODO diagram of commitment periods and refutation periods? -->
 
 ### Origination
 

docs/dApps.mdx → docs/dApps.md

@@ -5,8 +5,6 @@ last_update:
   date: 30 January 2024
 ---
 
-import LucidDiagram from '@site/src/components/LucidDiagram';
-
 One of the main features of blockchains is _decentralization_: each transaction is verified by multiple nodes and its validation process does not rely on a single trusted third party.
 Decentralized applications (dApps or Dapps) take advantage of these features to create applications that are independent, transparent, and trustless.
 
@@ -20,7 +18,7 @@ The off-chain component can be nearly any kind of program, including a web appli
 It relies on wallets and tools to interact with the smart contracts on behalf of a user's Tezos account.
 
 ![Fundamental diagram of dApps, showing the frontend, indexer, and backend](/img/dApps/dapp-overview.png)
-{/* Source https://lucid.app/lucidchart/8caf9ef1-11e4-454a-bbb6-ef4852515959/edit?page=0_0# */}
+<!-- Source https://lucid.app/lucidchart/8caf9ef1-11e4-454a-bbb6-ef4852515959/edit?page=0_0# -->
 
 Some of these tools that allow an off-chain component to interact with smart contracts include:
 

docs/tutorials/build-files-archive-with-dal.md

@@ -0,0 +1,114 @@
+---
+title: Implement a file archive with the DAL and a Smart Rollup
+authors: 'Tezos Core Developers'
+last_update:
+  date: 9 February 2024
+---
+
+:::note Experimental
+The Data Availability Layer is an experimental feature that is not yet available on Tezos Mainnet.
+The way the DAL works may change significantly before it is generally available.
+:::
+
+The Data Availability Layer (DAL) is a companion peer-to-peer network for the Tezos blockchain, designed to provide additional data bandwidth to Smart Rollups.
+It allows users to share large amounts of data in a way that is decentralized and permissionless, because anyone can join the network and post and read data on it.
+
+In this tutorial, you will set up a file archive that stores and retrieves files with the DAL.
+You will learn:
+
+- How data is organized and shared with the DAL and the reveal data channel
+- How to read data from the DAL in a Smart Rollup
+- How to host a DAL node
+- How to publish data and files with the DAL
+
+Because the DAL is not yet available on Tezos Mainnet, this tutorial uses the [Weeklynet test network](https://teztnets.com/weeklynet-about), which runs on a newer version of the protocol that includes the DAL.
+Weeklynet runs just like other Tezos networks like Mainnet and Ghostnet, with its own nodes, bakers, and accusers, so you don't need to run your own nodes and bakers.
+
+See these links for more information about the DAL:
+
+- For technical information about how the DAL works, see [Data Availability Layer](https://tezos.gitlab.io/shell/dal.html) in the Octez documentation.
+- For more information about the approach for the DAL, see [The Rollup Booster: A Data-Availability Layer for Tezos](https://research-development.nomadic-labs.com/data-availability-layer-tezos.html).
+
+## Tutorial applications
+
+In this tutorial, you set up these components:
+
+- The Octez client, which you use to manage a local wallet, deploy a Smart Rollup, and send data to the DAL
+- A Data Availability Layer node (not to be confused with a layer 1 node), which stores data temporarily and distributes it to Smart Rollups
+- A Smart Rollup that listens for data published to the DAL, retrieves it from the DAL node, and stores it locally
+- A Smart Rollup node that runs your Smart Rollup
+
+For simplicity, you do not set up a layer 1 node or a baker, which are responsible for verifying that the data is available before Smart Rollups can access it.
+Instead, you use the existing nodes and bakers that are running on Weeklynet.
+
+## Tutorial diagram
+
+Here is a diagram that shows the components that you set up in this tutorial in a light blue background:
+
+![A diagram of the DAL file tutorial, highlighting the Octez client, DAL node, and Smart Rollup that you create with a light blue background to distinguish them from the existing DAL nodes, layer 1 nodes, and bakers](/img/tutorials/dal-file-tutorial-setup.png)
+
+<!-- https://lucid.app/lucidchart/58f5577e-91b5-4237-89c4-a8cdf81c71ad/edit -->
+
+## Prerequisites
+
+This article assumes some familiarity with Smart Rollups.
+If you are new to Smart Rollups, see the tutorial [Deploy a Smart Rollup](./smart-rollup).
+
+## Why the DAL?
+
+The DAL has earned the nickname "Rollup Booster" from its ability to address
+the last bottleneck Smart Rollups developers could not overcome without
+sacrificing decentralization: block space. Smart Rollups offload
+*computation* from layer 1, but the transactions that they process still need to
+originate from somewhere.
+
+By default, that "somewhere" is the layer 1 blocks, yet the size of a Tezos
+block is limited to around 500KBytes. In this model, while Smart Rollups do not
+compete for layer 1 gas anymore, they still compete for block space.
+
+The DAL allows third parties to publish data and have bakers attest that the data is available.
+When enough bakers have attested that the data is available, Smart Rollups can retrieve the data without the need for additional trusted third-parties.
+
+## How the DAL works
+
+In this tutorial, you create a file archive application that allows clients to upload data to the DAL.
+You also create a Smart Rollup that listens to the DAL and responds to that data.
+
+The DAL works like this:
+
+1. Users post data to a DAL node.
+1. The DAL node returns a certificate, which includes two parts:
+
+   - The _commitment_ is like a hash of the data but has the additional ability to identify individual shards of the data and reconstruct the original data from a certain percentage of the shards.
+   The number of shards needed depends on how the data is spread across shards, which is controlled by a parameter called the _redundancy factor_.
+   - The _proof_ certifies the length of the data to prevent malicious users from overloading the layer with data.
+
+1. Users post the certificate to Tezos layer 1 via the Octez client.
+1. When the certificate is confirmed in a block, the DAL splits the data into shards and shares it through the peer-to-peer network.
+1. Layer 1 assigns the shards to bakers.
+1. Bakers verify that they are able to download the shards that they are assigned to.
+1. Bakers attest that the data is available in their usual block attestations to layer 1.
+
+   Each Tezos network has a delay of a certain number of blocks known as the _attestation lag_.
+   This number of blocks determines when bakers attest that the data is available and when the data becomes available to Smart Rollups.
+   For example, if a certificate is included in level 100 and the attestation lag is 4, bakers must attest that the data is available in level 104, along with their usual attestations that build on level 103.
+
+   If enough shards are attested in that level, the data becomes available to Smart Rollups at the end of layer 104.
+   If not enough shards are attested in that level, the certificate is considered bogus and the related data is dropped.
+
+1. The Smart Rollup node monitors the blocks and when it sees attested DAL data, it connects to a DAL node to request the data.
+Smart Rollups must store the data if they need it because it is available on the DAL for only a short time.
+
+The overall workflow is summarized in the following figure:
+
+![Overall diagram of the workflow of the Data Availability Layer](/img/architecture/dal-workflow.png)
+<!-- https://lucid.app/lucidchart/cc422278-7319-4a2f-858a-a7b72e1ea3a6/edit -->
+
+There are many steps in the DAL process, but the most complicated parts (storing and sharing data) are handled automatically by the various daemons in the Octez suite.
+
+:::note
+When you install a Smart Rollup, you provide only the installer kernel on layer 1 and the full kernel via the reveal data channel.
+Currently, you cannot send the full kernel data over the Data Availability Layer, so this tutorial relies on the reveal data channel to install the kernel as usual.
+:::
+
+When you are ready, get started by going to [Part 1: Setting up an environment](./build-files-archive-with-dal/set-up-environment).