Skip to content

Commit

Permalink
add QueryAPI pages (#1479)
Browse files Browse the repository at this point in the history
* add page

* update

* how it works

* Update docs/bos/queryapi/how-works.md

Co-authored-by: Morgan McCauley <morgan@mccauley.co.nz>



---------

Co-authored-by: Morgan McCauley <morgan@mccauley.co.nz>
  • Loading branch information
bucanero and morgsmccauley authored Sep 7, 2023
1 parent 9082533 commit fc2568f
Show file tree
Hide file tree
Showing 4 changed files with 144 additions and 14 deletions.
76 changes: 76 additions & 0 deletions docs/bos/queryapi/component.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
id: bos-components
title: Components handling Historical data
sidebar_label: Components & Historical data
---

Building components that handle historical blockchain data require dedicated solutions that manage the data and reduce the latency of requests, as it's not possible to scan the whole blockchain when a user makes a request.

A simple solution for developers building on NEAR BOS is using [QueryAPI](intro.md), a fully managed solution to build indexer functions, extract on-chain data, store it in a database, and be able to query it using GraphQL endpoints.

## QueryAPI

:::tip
Learn more about QueryAPI in this [QueryAPI Overview](intro.md) article.
:::

#### Tutorials

For a technical implementation deep-dive, check these QueryAPI tutorials:

- [Posts Indexer tutorial](../tutorial/indexer-tutorials/posts-indexer.md): this indexer creates a new row in a pre-defined database for every new BOS post found on the blockchain.
- [Hype Indexer tutorial](../tutorial/indexer-tutorials/hype-indexer.md): this indexer creates a new row in a pre-defined database for every new BOS post or comment found on the blockchain that contains either `PEPE` or `DOGE` in the contents.
- [BOS Feed Indexer tutorial](../tutorial/indexer-tutorials/feed-indexer.md): this indexer keeps track of new posts, comments, and likes on BOS, so a social feed can be rendered quickly.

## GraphQL queries

Using [QueryAPI's GraphiQL](index-function.md#mutations-in-graphql) tab, you can access the GraphiQL Explorer that provides a user friendly GraphQL playground, where you can view and create queries and mutations based on the DB schema that you defined for the indexer.

![QueryAPI Indexer Dashboard](/docs/assets/QAPIgraphiql.png)

You can easily set some fields and select the returning data
that you want, and the tool will build a query on the mutation panel on the right.
Then you can copy the resulting query, either in your JavaScript code so that you pass actual
data manually, or you pass in the mutation data object as a second parameter.

For example, if you go and add a new mutation, click <kbd>+</kbd>, then you can do a bunch of actions here, such as creating, deleting, or inserting posts into your table.

![Playground](/docs/assets/QAPIScreen.gif)

If you want to test your mutation, using [Debug Mode](index-function.md#local-debug-mode) you can add a specific
block to the list, and then play it to see how it works.
Based on the indexer logic you defined, you'll get a call to the GraphQL mutation with the object
and data passed into it.

:::tip Video Walkthrough

**Tip:** watch the video on how to [create mutations in GraphQL](https://www.youtube.com/watch?v=VwO6spk8D58&t=781s).

:::

## Generate a BOS component using Playground

Creating a BOS component from a GraphQL query is simple when using QueryAPI's GraphQL Playground. Just follow these steps:

- go to the GraphiQL tab
- select the query that you want to use
- click on the <kbd>Show GraphiQL Code Exporter</kbd> button
- get some default code here, copy it,
- go to the BOS sandbox, paste it.


This will set up some boilerplate code to execute the GraphQL query, add the query that you had
in your playground and then call that query, extract the data and render it using the
render data function.

Once you have the BOS component code, you can test it out by going to [the sandbox](https://near.org/sandbox),
pasting the generated code, and then selecting <kbd>Component Preview</kbd>.
Next, you can create a nice UI over this boilerplate code, and publish your new BOS component.

### Component Examples

- [Activity Feed widget](https://near.org/near/widget/ComponentDetailsPage?src=roshaan.near/widget/user-activity-feed&tab=source) running on [near.org](https://near.org)

<!--
- Example of BOS component using BigQuery
-->
48 changes: 48 additions & 0 deletions docs/bos/queryapi/how-works.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
id: how-it-works
title: How QueryAPI works
sidebar_label: How it works
---

QueryApi is a streaming indexer implementation that executes custom indexing logic written by developers on the NEAR blockchain.
QueryApi allows hosted execution of complex queries (ones that can’t be answered by a [simple RPC](../../5.api/rpc/introduction.md) or [Enhanced API](https://docs.pagoda.co/api) call), data hosting, and its exposure via GraphQL endpoints.


## Components involved

The QueryApi implementation integrates many different components in a single and streamlined solution.
In a high-level overview, the main components are:

:::info Components
`NEAR Protocol` -> `NEAR Lake` -> `Coordinator` -> `Runner` -> `Database` -> `API`
:::


### Detailed overview

An in-depth, detailed overview of the QueryApi components:

[![QueryAPI](/docs/qapi-components.png)](/docs/qapi-components.png)


### Description

- **Protocol:** the underlying NEAR Layer-1 Blockchain, where data `Blocks` and `Chunks` are produced.
- **NEAR Lake:** an indexer which watches the Layer-1 network and stores all the events as JSON files on AWS S3. Changes are indexed as new `Blocks` arrive.
- **Coordinator:** the QueryApi coordinator indexer filters matching data `Blocks`, runs historical processing threads, and queues developer's JS code to be indexed with these matched blocks.
- **Runner:** executes the user's indexer code, which outputs the data to the database.
- **Database:** a Postgres database where the developer's indexer data is stored, using a logical DB per user, and a logical schema per indexer function.
- **API:** a Hasura server running on Google Cloud Platform exposes a GraphQL endpoint so users can access their data from anywhere.

## Who hosts QueryAPI

[Pagoda Inc.](https://pagoda.co) runs and manages all the infrastructure of QueryAPI, including NEAR Lake nodes to store the data in JSON format on AWS S3.

- NEAR Lake indexing is hosted on AWS S3 buckets.
- Coordinator is hosted on GCP.
- Runners are hosted on AWS SQS and AWS Lambda.
- Hasura GraphQL API server is hosted on GCP.

:::caution Pricing
QueryAPI is currently free. Pagoda doesn't charge for storage of your indexer code and data as well as running the indexer, but usage pricing will be introduced once QueryApi is out of beta.
:::
34 changes: 20 additions & 14 deletions website/sidebars.json
Original file line number Diff line number Diff line change
Expand Up @@ -580,20 +580,6 @@
"bos/api/storage"
]
},
{
"Indexing": [
"bos/queryapi/intro",
"bos/queryapi/index-functions",
"bos/community/indexers",
{
"Tutorials": [
"bos/tutorial/indexer-tutorials/posts-indexer",
"bos/tutorial/indexer-tutorials/hype-indexer",
"bos/tutorial/indexer-tutorials/feed-indexer"
]
}
]
},
{
"Examples & Tutorials": [
"bos/tutorial/quickstart",
Expand All @@ -606,6 +592,7 @@
"bos/tutorial/lido"
]
},
"bos/queryapi/bos-components",
{
"type": "html",
"value": "<hr/>"
Expand Down Expand Up @@ -684,6 +671,25 @@
"type": "html",
"value": "<hr/>"
},
{
"type": "html",
"value": "<span class='menu__link'><b><small> QueryAPI </small></b></span>"
},
"bos/queryapi/intro",
"bos/queryapi/how-it-works",
"bos/queryapi/index-functions",
"bos/community/indexers",
{
"Tutorials": [
"bos/tutorial/indexer-tutorials/posts-indexer",
"bos/tutorial/indexer-tutorials/hype-indexer",
"bos/tutorial/indexer-tutorials/feed-indexer"
]
},
{
"type": "html",
"value": "<hr/>"
},
{
"type": "html",
"value": "<span class='menu__link'><b><small> NEAR Lake Framework </small></b></span>"
Expand Down
Binary file added website/static/docs/qapi-components.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit fc2568f

Please sign in to comment.