Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Check TS in docs using typescript-docs-verifier #1505

Merged
merged 21 commits into from
Jul 4, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5bf5624
docs: added aegir doc-check command to compile Typescript in document…
maschad Jan 13, 2023
a351954
doc: updated limits doc according to ts doc verifier errors (wip)
maschad Jan 13, 2023
61cc783
Merge branch 'master' into docs/check-ts-readme
achingbrain Jan 17, 2023
62706de
Merge branch 'master' into docs/check-ts-readme
maschad Apr 21, 2023
0ad1323
docs: updated linting for ts on docs (#1228)
maschad Apr 21, 2023
e5ab618
Merge branch 'master' into docs/check-ts-readme
maschad Apr 24, 2023
bbbf330
Merge branch 'master' into docs/check-ts-readme
maschad Apr 25, 2023
f3c95ce
Merge branch 'master' into docs/check-ts-readme
maschad May 1, 2023
8a4b309
docs: update metrics documentation (#1228)
maschad May 3, 2023
924e95c
doc: linting fixes (#1228)
maschad May 3, 2023
999575a
docs: move config to aegir repo (#1228)
maschad May 3, 2023
21a7087
Merge branch 'master' into docs/check-ts-readme
maschad May 29, 2023
f485990
deps: upgraded aegir to 39.0.9 to use latest doc-check (#1228)
maschad May 29, 2023
f9a5840
docs: ignore metrics dependency as it is needed for docs (#1228)
maschad May 29, 2023
9a43868
Merge branch 'master' into docs/check-ts-readme
maschad Jun 21, 2023
2b42331
docs: added doc check script (#1228)
maschad Jun 21, 2023
bb75005
Merge branch 'master' into docs/check-ts-readme
maschad Jun 22, 2023
ba36b95
Merge branch 'master' into docs/check-ts-readme
maschad Jun 28, 2023
ef3e569
docs: added doc-check to doc workspace (#1228)
maschad Jun 28, 2023
0948c94
docs: added doc-check to doc workspace (#1228)
maschad Jun 28, 2023
ec3ed56
Add private flag to avoid publishing to npm
maschad Jul 4, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ jobs:
- uses: ipfs/aegir/actions/cache-node-modules@master
- run: npm run --if-present lint
- run: npm run --if-present dep-check
- run: npm run --if-present doc-check

test-node:
needs: check
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,6 @@ dist
.coverage
package-lock.json
yarn.lock

# IDE
.DS_STORE
141 changes: 81 additions & 60 deletions doc/LIMITS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ This is important for [DoS](https://en.wikipedia.org/wiki/Denial-of-service_atta

It's possible to limit the total amount of connections a node is able to make (combining incoming and outgoing). When this limit is reached and an attempt to open a new connection is made, existing connections may be closed to make room for the new connection (see [Closing connections][#closing-connections]).

* Note: there currently isn't a way to specify different limits for incoming vs. outgoing. Connection limits are applied across both incoming and outgoing connections combined. There is a backlog item for this [here](https://github.com/libp2p/js-libp2p/issues/1508).
- Note: there currently isn't a way to specify different limits for incoming vs. outgoing. Connection limits are applied across both incoming and outgoing connections combined. There is a backlog item for this [here](https://github.com/libp2p/js-libp2p/issues/1508).

We can also limit the number of connections in a "pending" state. These connections have been opened by a remote peer but peer IDs have yet to be exchanged and/or connection encryption and multiplexing negotiated. Once this limit is hit further connections will be closed unless the remote peer has an address in the [allow list](#allowdeny-lists).

Expand All @@ -35,32 +35,36 @@ const node = await createLibp2pNode({
/**
* The total number of connections allowed to be open at one time
*/
maxConnections: number
maxConnections: number,
maschad marked this conversation as resolved.
Show resolved Hide resolved

/**
* If the number of open connections goes below this number, the node
* will try to connect to randomly selected peers from the peer store
*/
minConnections: number
minConnections: number,

/**
* How many connections can be open but not yet upgraded
*/
maxIncomingPendingConnections: number
}
})
maxIncomingPendingConnections: number,
},
});

export {};
maschad marked this conversation as resolved.
Show resolved Hide resolved
```

## Closing connections

When choosing connections to close the connection manager sorts the list of connections by the value derived from the tags given to each peer. The values of all tags are summed and connections with lower valued peers are eligible for closing first. If there are tags with equal values, the shortest-lived connection will be closed first.

```js
```ts
// tag a peer
await libp2p.peerStore.tagPeer(peerId, 'my-tag', {
await libp2p.peerStore.tagPeer(peerId, "my-tag", {
value: 50, // 0-100 is the typical value range
ttl: 1000 // optional field, this tag will be deleted after this many ms
})
ttl: 1000, // optional field, this tag will be deleted after this many ms
});

export {};
```

## Inbound connection threshold
Expand All @@ -76,9 +80,11 @@ const node = await createLibp2pNode({
* A remote peer may attempt to open up to this many connections per second,
* any more than that will be automatically rejected
*/
inboundConnectionThreshold: number
}
})
inboundConnectionThreshold: number,
},
});

export {};
```

## Data transfer and Event Loop limits
Expand All @@ -92,7 +98,7 @@ All fields are optional. The default values are defined in [src/connection-manag
```ts
const node = await createLibp2pNode({
metrics: {
enabled: true
enabled: true,
},
connectionManager: {
/**
Expand All @@ -102,7 +108,7 @@ const node = await createLibp2pNode({
*
* It is bytes per second.
*/
maxData: number
maxData: number,

/**
* If the node sends more than this amount of data in bytes/second
Expand All @@ -111,7 +117,7 @@ const node = await createLibp2pNode({
*
* It is bytes per second.
*/
maxSentData: number
maxSentData: number,

/**
* If the node receives more than this amount of data in bytes/second
Expand All @@ -120,17 +126,18 @@ const node = await createLibp2pNode({
*
* It is bytes per second.
*/
maxReceivedData: number
maxReceivedData: number,

/**
* If the event loop takes longer than this many ms to run, connections
* to untagged peers or those not in the allow list may be closed.
*
* It is milliseconds.
*/
maxEventLoopDelay: number
}
})
maxEventLoopDelay: number,
},
});
export {};
```

## Stream limits
Expand All @@ -152,31 +159,32 @@ const node = await createLibp2pNode({
/**
* The total number of inbound protocol streams that can be opened on a given connection
*/
maxInboundStreams: number
maxInboundStreams: number,

/**
* The total number of outbound protocol streams that can be opened on a given connection
*/
maxOutboundStreams: number
maxOutboundStreams: number,

/**
* How much incoming data in bytes to buffer while attempting to parse messages - peers sending many small messages in batches may cause this buffer to grow
*/
maxUnprocessedMessageQueueSize: number
maxUnprocessedMessageQueueSize: number,

/**
* How much message data in bytes to buffer after parsing - slow stream consumers may cause this buffer to grow
*/
maxStreamBufferSize: number
maxStreamBufferSize: number,

/**
* Mplex does not support backpressure so to protect ourselves, if `maxInboundStreams` is
* hit and the remote opens more than this many streams per second, close the connection
*/
disconnectThreshold: number
})
]
})
disconnectThreshold: number,
}),
],
});
export {};
```

### Yamux
Expand All @@ -194,17 +202,19 @@ const node = await createLibp2pNode({
*
* This field is optional, the default value is shown
*/
maxInboundStreams: number
maxInboundStreams: number,

/**
* The total number of outbound protocol streams that can be opened on a given connection
*
* This field is optional, the default value is shown
*/
maxOutboundStreams: number
})
]
})
maxOutboundStreams: number,
}),
],
});

export {};
```

### Protocol limits
Expand All @@ -218,12 +228,19 @@ Since incoming stream data is buffered until it is consumed, you should attempt
All fields are optional. The default values are defined in [src/registrar.ts](https://github.com/libp2p/js-libp2p/blob/master/src/registrar.ts) - please see that file for the current values.

```ts
libp2p.handle('/my-protocol/1.0.0', (streamData) => {
// ..handle stream
}, {
maxInboundStreams: number
maxOutboundStreams: number
})
const node = await createLibp2pNode({});

node.handle(
"/my-protocol/1.0.0",
(streamData) => {
// ..handle stream
},
{
maxInboundStreams: number,
maxOutboundStreams: number,
}
);
export {};
```

## Transport specific limits
Expand All @@ -245,29 +262,31 @@ const node = await createLibp2pNode({
/**
* Inbound connections with no activity in this time frame (ms) will be closed
*/
inboundSocketInactivityTimeout: number
inboundSocketInactivityTimeout: number,

/**
* Outbound connections with no activity in this time frame (ms) will be closed
*/
outboundSocketInactivityTimeout: number
outboundSocketInactivityTimeout: number,

/**
* Once this many connections are open on this listener any further connections
* will be rejected - this will have no effect if it is larger than the value
* configured for the ConnectionManager maxConnections parameter
*/
maxConnections: number
})
]
})
maxConnections: number,
}),
],
});

export {};
```

## Allow/deny lists

It is possible to configure some hosts to always accept connections from and some to always reject connections from.

```js
```ts
const node = await createLibp2pNode({
connectionManager: {
/**
Expand All @@ -276,24 +295,26 @@ const node = await createLibp2pNode({
* all connection limits
*/
allow: [
'/ip4/43.123.5.23/tcp/3984',
'/ip4/234.243.64.2',
'/ip4/52.55',
"/ip4/43.123.5.23/tcp/3984",
maschad marked this conversation as resolved.
Show resolved Hide resolved
"/ip4/234.243.64.2",
"/ip4/52.55",
// etc
],

/**
* Any connection with a `remoteAddress` property that has any of these
* addresses as a prefix will be immediately rejected
*/
deny: [
'/ip4/132.14.52.64/tcp/3984',
'/ip4/234.243.64.2',
'/ip4/34.42',
deny: [
"/ip4/132.14.52.64/tcp/3984",
"/ip4/234.243.64.2",
"/ip4/34.42",
// etc
]
}
})
],
},
Comment on lines +287 to +288
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These comments should follow the same linting rules as the main codebase; comma-dangle is disallowed.

Suggested change
],
},
]
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the doc-check command which utilizes https://github.com/bbc/typescript-docs-verifier only checks that Typescript code snippets compile, it's not a linter. That could be added in a follow up PR but wasn't the original intention of this.

});
maschad marked this conversation as resolved.
Show resolved Hide resolved

export {};
```

## How much memory will be used for buffering?
Expand All @@ -302,10 +323,10 @@ There is no a single config value to control the amount of memory js-libp2p uses

Important details for ascertaining this are:

* Each connection has a multiplexer
* Each multiplexer has a buffer for raw incoming data (`muxer.maxUnprocessedMessageQueueSize`)
* The incoming data is parsed into messages for each stream and queued (`muxer.maxStreamBufferSize`)
* Each multiplexer has a stream limit for number of streams (`muxer.maxInboundStreams`, `muxer.maxOutboundStreams`).
- Each connection has a multiplexer
- Each multiplexer has a buffer for raw incoming data (`muxer.maxUnprocessedMessageQueueSize`)
- The incoming data is parsed into messages for each stream and queued (`muxer.maxStreamBufferSize`)
- Each multiplexer has a stream limit for number of streams (`muxer.maxInboundStreams`, `muxer.maxOutboundStreams`).

As a result, the max amount of memory buffered by libp2p is approximately:

Expand Down
17 changes: 17 additions & 0 deletions doc/tsconfig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
maschad marked this conversation as resolved.
Show resolved Hide resolved
"compilerOptions": {
/* Language and Environment */
"target": "esnext" /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */,
/* Modules */
"module": "esnext" /* Specify what module code is generated. */,
"esModuleInterop": true /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */,
// "preserveSymlinks": true, /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */
"forceConsistentCasingInFileNames": true /* Ensure that casing is correct in imports. */,
/* Type Checking */
"strict": true /* Enable all strict type-checking options. */,
/* Completeness */
"skipLibCheck": true /* Skip type checking all .d.ts files. */,
"noImplicitAny": false,
"isolatedModules": false
}
}
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@
"clean": "aegir clean",
"lint": "aegir lint",
"dep-check": "aegir dep-check",
"doc-check": "aegir doc-check --inputFiles doc/*.md --tsConfigPath doc/",
"prepublishOnly": "node scripts/update-version.js",
"build": "aegir build",
"docs": "aegir docs",
Expand Down Expand Up @@ -187,7 +188,7 @@
"@types/p-fifo": "^1.0.0",
"@types/varint": "^6.0.0",
"@types/xsalsa20": "^1.1.0",
"aegir": "^37.5.6",
"aegir": "^38.1.0",
"cborg": "^1.8.1",
"delay": "^5.0.0",
"execa": "^6.1.0",
Expand Down