Hoover - Decentralized Social Network Data Aggregator

Hoover is a data aggregator built for Arweave developers to collect and unify event data from decentralized social networks like Nostr, Bluesky, and Farcaster. By consolidating this data on Arweave, Hoover provides a unified feed that developers can leverage for various decentralized applications, helping to bridge the gap between fragmented communities spread across different social media protocols.

Key Features

• Aggregates data from Nostr, Bluesky, and Farcaster into a common schema.
• Stores data on Arweave for long-term, decentralized storage.
• Supports five primary event types: Post, Repost, Like, Follow, Profile.
• Includes a basic event browser for viewing and searching indexed data. Additionally, this browser performs in-place signature validations of posts ensuring their integrity.

Supported Social Networks

• Nostr: A minimalist, decentralized protocol with a publish-subscribe architecture.
• Bluesky: A federated protocol developed by Bluesky, Inc., with strict control over the specification.
• Farcaster: Built on Ethereum, Farcaster uses a blockchain-based model for social interactions.

Getting Started

To get started with Hoover, follow these steps:

1. Install Go (version 1.22.6 or higher) Go Download

2. Generate Key: Generate new RSA key or load an old one for Arweave uploading

   Run the following command from the project root:

   go run ./cmd/keygen/. > keyfile.json

3. Run the Hoover: You can now start Hoover:

   ARWEAVE_GATEWAYS=https://up.arweave.net WALLET_FILE=keyfile.json go run ./cmd/hoover/.

4. Start the Browser Interface: Finally, start the browser to view and interact with the posts pulled from Arweave (previously uploaded by th Hoover):

   cd browser
   npm install
   npm run dev

   Upon running this command, you should be presented with a clickable http link which allows you to view the Hoover's output.

   To learn more about the browser, click here

To run the hoover on arlocal for testing purposes, see the Testing Environment Setup

Architecture and Data Formats

Hoover works by aggregating data from decentralized social networks and formatting it for storage on Arweave. It supports five primary event types:

• Post - Basic text content, potentially containing links and references to other users or events.
• Repost - An event re-published by a user for their followers.
• Like - A user’s positive reaction to a post or event.
• Follow - A public declaration of subscribing to another user.
• Profile - User metadata, such as display name, avatar, and other contact information.

For full details on data formats and bundling principles, see the Data Specification.

Project Milestones

Our project milestones cover the development process, goals, and task completion. Check out the detailed milestones in the Project Milestones Document.

Credits

This project was developed by Hubmaker Labs, a company specializing in decentralized applications and infrastructure, under contract with Arweave, whose support and funding made this work possible.

Team:

• David Vennik (Nostr & Bluesky Aggregation, Arweave Test Harness) - me@mleku.dev
• Akash Kumar (Farcaster & Bluesky Aggregation, Arweave Integration, Documentation) - akashkumar1691@gmail.com
• Selami Cetinguney (Data Browser example app) - selami.c@sbytes-it.com

We are grateful to Arweave for enabling the development of Hoover and its role in advancing decentralized, long-term data storage.