feat(rpc, api): add `gossipsub` methods and expose the p2p `network_events` to the `Api` #669

ramfox · 2023-01-03T05:23:04Z

Gets iroh-api ready for use in iroh-share (via iroh-embed):

p2p.network_events - exposes stream of NetworkEvents
p2p.gossipsub_subscribe
p2p.gossipsub_publish
p2p.gossipsub_add_peers
p2p.peer_id - the local peer id
p2p.addrs - the addresses of the local node
resolver - exposes the resolver
store - exposes the store methods

This focuses on creating tests for the `iroh get` CLI command. The Iroh API is also improved along the way. This required more work in the underlying API to make it more testable. What's tested here is everything *except* the actual IPFS behavior -- the focus is on the behavior of the CLI and API and its interactions with the filesystem. ## trycmd tests In `iroh/tests/cmd` you can see quite a few `.trycmd` files. These describes the command-line interaction. Stuff behind `$ is a command, there's a special `? failed` indicator if the command is considered to have failed, and the output of the command is shown. New are the `.out` and `.in` directories with the same names as the `.trycmd` files. These describe the filesystem before the command runs (may be missing), and the filesystem after the command has run. This way we can describe the effects of the `iroh get` command - directories and files are supposed to be created. We can also test failure scenarios where we refuse to overwrite a directory that already exists. #269 tracks various test cases. ## `get_stream` The `get` high level CLI method has been removed from the mockable `Api` trait (read on to see where it went). Instead, a more low level but still useful API method `get_stream` has been added to the `Api` trait. This gets a stream of relative paths and `OutType`, describing the directories and files you can create. The big difference with what was there before is that it returns relative paths and doesn't calculate final destination paths -- that's up to the user of the API. ## No `async_trait` macro for `Api` trait The interactions between the `Api` trait, `mockall` macro and `async_trait` were getting so hairy I couldn't figure out how to express things anymore once I wanted to add `get_stream`. For my sanity and also to learn better how this really works underneath, I've rewritten the `Api` trait to describe itself explicitly in terms of `(Local)BoxFuture` and `(Local)_BoxStream`. It's more verbose but functionally equivalent and I could express what I wanted. ## `ApiExt` trait The `ApiExt` trait is a trait that implements the high level `get` command. It puts everything together: it handles various error conditions, accesses the stream and then writes the stream to the filesystem. It's basically `iroh get`. The `ApiExt` trait is solely intended to contain default trait methods, and is automatically available when the `Api` contract is fulfilled (if you `use` it). Factored out `save_get_stream` from `getadd.rs` to be solely concerned with turning a stream into files and directories on the files system. That makes it possible to test its behavior in isolation. ## test fixture The `get` test fixture now mocks `get_stream` and returns a fake stream made from a `Vec`. This defines the actual stuff that the CLI writes to disk. ## relative_path Now depend on the [`relative-path` crate](https://crates.io/crates/relative-path) because what the stream returns are clearly relative paths, and we want to force the user to do something with them before being able to actually write stuff.

* feat: range request support * cr * tests & fixes * fix etag range headers * rebase cleanup

…iroh` commands (#348) * docs: add full help text for `lookup`, `connect`, `get`, `p2p`, and `iroh` commands * move long descriptions into constants Co-authored-by: b5 <sparkle_pony_2000@qri.io>

The store does not use the RPC client at all. Remove it.

We used to use the same config for the service and server (aka the binary). This is confusing when creating configs to use with e.g. iroh-one, iroh-embed or iroh-share because some fields are not used. This splits off the config structs to avoid this problem, services now have a Config and binaries a ServerConfig. This allows creating the services standalone without all the baggage a server needs. While this isn't many fields yet, this will get worse as we add more features (this is split off from another PR where this seemed useful).

iroh-api/src/p2p.rs

This removes the use of mockall which is causing a lot of trouble with the cargo features intricacies it brings with it. A lot of the tooling struggles with these different versions of the structs which also behave differently in surprising ways. The tests affected by this will be converted into end-to-end tests as part of is now unused. This is possibly only temprorary and that might come back when the end-to-end tests are added.

flub

If you'd like to punt on the API musings I have that's fine. We don't have to have a perfect API straight away and are probably free to keep breaking it for a while.

iroh-api/src/api.rs

flub · 2023-01-05T12:00:26Z

iroh-api/src/api.rs

@@ -117,6 +122,11 @@ impl Api {
        Ok(P2pApi::new(p2p_client))
    }

+    pub fn store(&self) -> Result<StoreApi> {


Even if relatively trivial, I really appreciate doc comments especially on things that end up in the public API of iroh-embed. E.g. this would be a good place to give a summary of what a the store is and why you might want to use it directly.

flub · 2023-01-05T12:06:08Z

iroh-api/src/p2p.rs

+    }
+
+    /// The list of [`Multiaddr`] that the Iroh p2p node is listening on
+    pub async fn addrs(&self) -> Result<Vec<Multiaddr>> {


From similar APIs in e.g. hyper I'm mildly tempted to call this local_addrs. Though I don't mind this name either.

iroh-api/src/p2p.rs

flub · 2023-01-05T12:13:18Z

iroh-api/src/p2p.rs

@@ -56,9 +85,69 @@ impl P2p {
            .await
            .map_err(|e| map_service_error("p2p", e))
    }
+
+    /// Get a stream of [`NetworkEvent`].
+    pub async fn network_events(&self) -> Result<BoxStream<'static, Result<NetworkEvent>>> {


I'd be tempted to wrap this return type in a newtype for the API purposes.

iroh-api/src/p2p.rs

iroh-api/src/store.rs

iroh-p2p/src/rpc.rs

…k_events` method

Add `api.store()` method as way to get `StoreApi` from the API. This exposes the `put` method, which is needed in `iroh-share`. also moves CLI specific `connect` method to `iroh` and adds more general `connect` method to `iroh-api`

add, get, & gossipsub methods This will get refactored to not be gross.

ramfox · 2023-01-11T02:43:25Z

iroh-p2p/src/rpc.rs

+    fn gossipsub_subscribe(self, req: GossipsubSubscribeRequest) -> GossipsubEventStream {
+        async move {
+            self.gossipsub_subscribe_0(req)
+                .await
+                .expect("FIX THIS - NEEDS TO BE ABLE TO RETURN A RESULT")
+        }
+        .flatten_stream()
+        .boxed()
+    }


@rklaehn I'm coming into conflict with this method in two ways.

For context, a GossipsubEventStream is BoxStream<'static, Box<GossipsubSubscribeResponse>>

First question/problem: I can't figure out how to handle the possible error from gossipsub_subscribe_0 When I try to use ? instead of expect here, I keep running into

cannot use the `?` operator in an async block that returns `Pin<Box<dyn Stream<Item = Box<iroh_rpc_types::p2p::GossipsubSubscribeResponse>> + std::marker::Send>>` | this function should return `Result` or `Option` to accept `?

And when I looked up how to solve this, I didn't understand the syntax that the solutions suggested. Would love some help!

The second conflict I'm running into is with my "preferred" return type. My "dream" scenario is that the p2p RpcClient's gossipsub_subscribe can have a return type of:
Result<impl Stream<Item=GossipsubEvent>>

But, it seems like I'm forced to return a Result<GossipsubEvent> Item, based on the expectations of server_streaming. This isn't critical, but It would be nice to streamline the API in this way.

A server streaming interaction returns a stream, not a result of stream. So if you have a situation where you have a possible error on creation of the stream, but then no errors once the stream is created, you have no choice but to have the stream item be a Result. Otherwise the try_flatten_stream thing won't work.

So the result must be impl Stream<Item=Result<GossipsubEvent>>, or we must extend quic-rpc to support a result of stream. Since it is all just decorators and most of the critical work is in the transports, this should be possible. Just a bit of work, so the question is how strongly you feel about this.

Is it really the case that once you have subscribed there is nothing whatsoever that can go wrong? That is usually quite rare. What I saw often is to have Result<impl Stream<Item = Result<...>>, in which case I figured you might as well flatten it.

ramfox · 2023-02-20T16:06:43Z

Hello! The code that was previously hosted in this repository has been moved to n0-computer/beetle. beetle is in maintenance mode, but will still accept PRs.

If you are still interested in getting your PR merged, please re-open your PR on n0-computer/beetle.

Check out our blog post for more info on our new direction for iroh: https://n0.computer/blog/a-new-direction-for-iroh/

This commit represents the state of n0-computer#669 before the repo changes.

This commit represents the state of n0-computer/iroh#669 before the repo changes.

faassen and others added 30 commits October 14, 2022 18:46

feat: improve CLI texts and flags (#342)

9e0a03e

feat: range request support (#330)

074f0c6

* feat: range request support * cr * tests & fixes * fix etag range headers * rebase cleanup

chore: add dev-install xtask (#347)

5937924

reduce invalid stored connections

31cb29c

metrics: bitswap loop & poll ticks

a160805

feat: race a http gateway with p2p in iroh-one's content loader (#338)

40cb232

docs: add full help text for lookup, connect, get, p2p, and `…

b14a7a8

…iroh` commands (#348) * docs: add full help text for `lookup`, `connect`, `get`, `p2p`, and `iroh` commands * move long descriptions into constants Co-authored-by: b5 <sparkle_pony_2000@qri.io>

allow for more dialing out by default

4570621

refactor message queue actor

0904280

try: trigger connects for non bitswap peers

a95d0f5

improve gc and stopping of message queue

56f1df9

fix client loop metric

381f934

track peers

09a5155

improve peer state tracking

71eaab3

only skip real dials when dialing is paused

671d178

fixup triggers

a82ade2

improve connection management

8191cb7

more counters and rely on disconnects for cleanup

0afefe5

connect with identify

16c40c9

misc fixes and debugging

8e3f831

refactor: lock free peer manager

733601d

remove client unused client counters

cd78bd3

serialize incoming message processing

ced8e05

log backpressure for network actions

00a55c6

more evil

4bdec19

more detailed message metrics

a8303df

more debugging

6841992

getting closer

6046da1

bigger buffers

e4b6beb

flub added 2 commits January 4, 2023 18:08

ref(store): Remove unused RPC Client (#672)

1d30916

The store does not use the RPC client at all. Remove it.

ramfox force-pushed the ramfox/iroh_share branch 2 times, most recently from ac5ed16 to f2f42fd Compare January 5, 2023 05:56

ramfox changed the title ~~WIP refactor: iroh-share should used iroh-embed~~ feat(rpc, api): add gossipsub methods and expose the p2p network_events to the Api Jan 5, 2023

ramfox marked this pull request as ready for review January 5, 2023 06:18

ramfox force-pushed the ramfox/iroh_share branch from f2f42fd to 7baba0c Compare January 5, 2023 06:22

ramfox commented Jan 5, 2023

View reviewed changes

iroh-api/src/p2p.rs Outdated Show resolved Hide resolved

ramfox requested review from flub and dignifiedquire January 5, 2023 06:27

flub reviewed Jan 5, 2023

View reviewed changes

ramfox added 4 commits January 5, 2023 10:12

feat(rpc, p2p): expose a stream of p2p network events via rpc `networ…

e869a7f

…k_events` method

feat(api): expose network_events and gossipsub methods over the API

6c05484

clean up & add documentation

b2414c6

add more gossipsub methods to api

39f6804

Add `api.store()` method as way to get `StoreApi` from the API. This exposes the `put` method, which is needed in `iroh-share`. also moves CLI specific `connect` method to `iroh` and adds more general `connect` method to `iroh-api`

ramfox force-pushed the ramfox/iroh_share branch from 7baba0c to 378c805 Compare January 6, 2023 00:50

Proof of concept iroh-share using iroh-api methods

8517e1d

add, get, & gossipsub methods This will get refactored to not be gross.

ramfox force-pushed the ramfox/iroh_share branch from 378c805 to 8517e1d Compare January 6, 2023 00:55

ramfox added 2 commits January 10, 2023 18:48

POC iroh-share

f23b895

refactor: iroh-share sender

3f9bbc9

ramfox force-pushed the ramfox/iroh_share branch from 9adc9c6 to 3f9bbc9 Compare January 11, 2023 02:25

ramfox commented Jan 11, 2023

View reviewed changes

dignifiedquire closed this Feb 20, 2023

dignifiedquire force-pushed the main branch from 6527ee4 to 3a3fc46 Compare February 20, 2023 15:47

dignifiedquire deleted the ramfox/iroh_share branch February 20, 2023 15:48

nathanielc pushed a commit to nathanielc/iroh that referenced this pull request Mar 7, 2023

feat(rpc, api): add gossipsub methods to Api

3f39db1

This commit represents the state of n0-computer#669 before the repo changes.

nathanielc mentioned this pull request Mar 7, 2023

feat(rpc, api): add gossipsub IPC methods n0-computer/beetle#296

Open

nathanielc pushed a commit to nathanielc/beetle that referenced this pull request Mar 7, 2023

feat(rpc, api): add gossipsub methods to Api

3467da1

This commit represents the state of n0-computer/iroh#669 before the repo changes.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(rpc, api): add `gossipsub` methods and expose the p2p `network_events` to the `Api` #669

feat(rpc, api): add `gossipsub` methods and expose the p2p `network_events` to the `Api` #669

ramfox commented Jan 3, 2023 •

edited

Loading

flub left a comment

flub Jan 5, 2023

flub Jan 5, 2023

flub Jan 5, 2023

ramfox Jan 11, 2023

rklaehn Jan 11, 2023 •

edited

Loading

ramfox commented Feb 20, 2023

feat(rpc, api): add gossipsub methods and expose the p2p network_events to the Api #669

feat(rpc, api): add gossipsub methods and expose the p2p network_events to the Api #669

Conversation

ramfox commented Jan 3, 2023 • edited Loading

flub left a comment

Choose a reason for hiding this comment

flub Jan 5, 2023

Choose a reason for hiding this comment

flub Jan 5, 2023

Choose a reason for hiding this comment

flub Jan 5, 2023

Choose a reason for hiding this comment

ramfox Jan 11, 2023

Choose a reason for hiding this comment

rklaehn Jan 11, 2023 • edited Loading

Choose a reason for hiding this comment

ramfox commented Feb 20, 2023

feat(rpc, api): add `gossipsub` methods and expose the p2p `network_events` to the `Api` #669

feat(rpc, api): add `gossipsub` methods and expose the p2p `network_events` to the `Api` #669

ramfox commented Jan 3, 2023 •

edited

Loading

rklaehn Jan 11, 2023 •

edited

Loading