Add HTTP Gateway Specs (#283)

* feat: initial HTTP gateway specs

This adds gateway specs under ./http-gateways directory.

The aim is to document _current_ behavior (implementation in go-ipfs 0.13)
and switch the way we do the gateway work to be specs-driven.

Long term goal is to provide language and implementation agnostic
specification that anyone can use to implement compatible gateways.

* gateway: add Content-Range

* gateway: registerProtocolHandler uri router

* CODEOWNERS: add lidel for ./http-gateways

* gateway: resolving an advanced DNSLink chain

* gateway: only-if-cached HEAD behavior

* gateway: suggestions from reviewers

Co-authored-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
Co-authored-by: Vasco Santos <vasco.santos@moxy.studio>
Co-authored-by: Oli Evans <oli.evans@gmail.com>

* gateway: include CIDv1 node in summary

* gateway: reorder URI router section

As suggested in #283 (comment)

* gateway: add Denylists section

* gateway: switch only-if-cached miss to 412

Rationale: ipfs/kubo#8783 (comment)

* gateway: apply suggestions from review

Co-authored-by: Thibault Meunier <thibmeu@users.noreply.github.com>

* gateway: apply suggestions from Cloudflare

#283 (review)

* gateway: add X-Content-Type-Options

* gateway: simplify dnslink summary

https://github.com/ipfs/specs/pull/283/files#r898709569

* gateway: document 412 Precondition Failed

https://github.com/ipfs/specs/pull/283/files#r898686654

* gateway: link to ipld codecs explainer

https://github.com/ipfs/specs/pull/283/files#r898687052

* gateway: stub about handling traversal errors

https://github.com/ipfs/specs/pull/283/files#r892845860

* gateway: expand HTTP caching considerations

* gateway: editorial fixes

Co-authored-by: Steve Loeppky <stvn@loeppky.com>

* gateway: expand on Host header parsing

https://github.com/ipfs/specs/pull/283/files#r898703765

* gateway: editorial fixes

* gateway: X-Forwarded-Proto and X-Forwarded-Host

* gateway: editorial fixes

* gateway: X-Trace-Id

optional header suggested in:
#283 (comment)

rationale: having specific name as a suggestion of 'best practice' in
the specs will simplify debugging across ecosystem

* gateway: Generated HTML with directory index

Synthesis of ideas from:
ipfs/kubo#8455
and
ipfs/kubo#9058

Co-authored-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
Co-authored-by: Vasco Santos <vasco.santos@moxy.studio>
Co-authored-by: Oli Evans <oli.evans@gmail.com>
Co-authored-by: Thibault Meunier <thibmeu@users.noreply.github.com>
Co-authored-by: Steve Loeppky <stvn@loeppky.com>

.github/CODEOWNERS

@@ -0,0 +1,4 @@
+# Spec Stewards defined below are automatically requested for review when
+# someone opens a pull request that modifies area of their interest.
+
+http-gateways/  @lidel

README.md

@@ -29,14 +29,12 @@ The specs contained in this repository are:
   - [Protocol Architecture Overview](./ARCHITECTURE.md) - the top-level spec and the stack
   - [Other IPFS Overviews](/overviews) - quick overviews of the various parts of IPFS
 - **User Interface (aka Public APIs):**
-  - [Core API (aka using IPFS as a package/module)](./API_CORE.md)
-    - [JavaScript Interface](https://github.com/ipfs/interface-js-ipfs-core)
-    - [Golang Interface](https://github.com/ipfs/interface-go-ipfs-core)
-  - [CLI (the ipfs daemon API)](./API_CLI.md)
-  - [HTTP API](./API_HTTP.md)
-  - HTTP Gateway
+  - [HTTP Gateways](./http-gateways/README.md) - implementation agnostic interfaces for accessing content-addressed data over HTTP
+  - IPFS implementations may provide additional interfaces, for example:
+    - [HTTP RPC API exposed by go-ipfs](https://docs.ipfs.io/reference/http/api/)
+    - [Programmatic Core API for JavaScript](https://github.com/ipfs/js-ipfs/tree/master/docs/core-api#readme)
 - **Data Formats:**
-  - [IPLD](https://github.com/ipld/spec) - InterPlanetary Linked Data.
+  - [IPLD](https://ipld.io/specs/) - InterPlanetary Linked Data.
   - [Merkle DAG (Deprecated)](./MERKLE_DAG.md)
   - Self Describing Formats ([multiformats](http://github.com/multiformats/multiformats)):
     - [multihash](https://github.com/multiformats/multihash) - self-describing hash digest format.

http-gateways/DNSLINK_GATEWAY.md

@@ -0,0 +1,100 @@
+# DNSLink Gateway Specification
+
+![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square)
+
+**Authors**:
+
+- Marcin Rataj ([@lidel](https://github.com/lidel))
+
+----
+
+**Abstract**
+
+DNSLink Gateway is an extension of
+[PATH_GATEWAY.md](./PATH_GATEWAY.md)
+that enables hosting a specific content path under a specific DNS name.
+
+This document describes the delta between [PATH_GATEWAY.md](./PATH_GATEWAY.md) and this gateway type.
+
+In short:
+
+- HTTP request includes a valid [DNSLink](https://dnslink.dev/) name in `Host` header
+- gateway decides if DNSlink name is allowed
+- gateway resolves DNSLink to an immutable content root identified by a CID
+- HTTP response includes the data for the CID
+
+# Table of Contents
+
+- [DNSLink Gateway Specification](#dnslink-gateway-specification)
+- [Table of Contents](#table-of-contents)
+- [HTTP API](#http-api)
+  - [`GET /[{path}][?{params}]`](#get-pathparams)
+  - [`HEAD /[{path}][?{params}]`](#head-pathparams)
+- [HTTP Request](#http-request)
+  - [Request headers](#request-headers)
+    - [`Host` (request header)](#host-request-header)
+- [HTTP Response](#http-response)
+- [Appendix: notes for implementers](#appendix-notes-for-implementers)
+  - [Leveraging DNS for content routing](#leveraging-dns-for-content-routing)
+
+# HTTP API
+
+## `GET /[{path}][?{params}]`
+
+Downloads data at specified path under the content path for DNSLink name provided in `Host` header. 
+
+- `path` – optional path to a file or a directory under the content root sent in `Host` HTTP header
+    - Example: if `Host: example.com` then the content path to resolve is `/ipns/example.com/{path}`
+
+## `HEAD /[{path}][?{params}]`
+
+Same as GET, but does not return any payload.
+
+# HTTP Request
+
+Below MUST be implemented **in addition** to the [HTTP Request section from `PATH_GATEWAY.md`](./PATH_GATEWAY.md#http-request).
+
+## Request headers
+
+### `Host` (request header)
+
+Defines the [DNSLink](https://docs.ipfs.io/concepts/glossary/#dnslink) name
+to RECURSIVELY resolve into an immutable `/ipfs/{cid}/` prefix that should
+be prepended to the `path` before the final IPFS content path resolution
+is performed.
+
+Implementations MUST ensure DNSLink resolution is safe and correct:
+- each DNSLink may include an additional path segment, which MUST be preserved
+- each DNSLink may point at other DNSLink, which means there MUST be a hard
+  recursion limit (e.g. 32) and HTTP 400 Bad Request error MUST be returned
+  when the limit is reached.
+
+**Example: resolving an advanced DNSLink chain**
+
+To illustrate, given DNSLink records:
+
+- `_dnslink.a.example.com` TXT record: `dnslink=/ipns/b.example.net/path-b`
+- `_dnslink.b.example.net` TXT record: `dnslink=/ipfs/bafy…qy3k/path-c`
+
+HTTP client sends `GET /path-a` request with  `Host: a.example.com` header
+which recursively resolves all DNSLinks and produces the final immutable
+content path:
+
+1. `Host` header + `/path-a` → `/ipns/a.example.net/path-a`
+2. Resolving DNSlink at `a.example.net` replaces `/ipns/a.example.net` with `/ipns/b.example.net/path-b`
+3. Resolving DNSlink at `b.example.net` replaces `/ipns/b.example.net` with `/ipfs/bafy…qy3k/path-c`
+4. The immutable content path is `/ipfs/bafy…qy3k/path-c/path-b/path-a`
+
+# HTTP Response
+
+Same as [HTTP Response section in `PATH_GATEWAY.md`](./PATH_GATEWAY.md#http-response).
+
+# Appendix: notes for implementers
+
+## Leveraging DNS for content routing
+
+- It is a good idea to publish
+  [DNSAddr](https://github.com/multiformats/multiaddr/blob/master/protocols/DNSADDR.md)
+  TXT records with known content providers for the data behind a DNSLink. IPFS
+  clients will be able to detect DNSAddr and preconnect to known content
+  providers, removing the need for expensive DHT lookup.