Public Helium API

[PUSHRCDN]

Project:

Public Helium API

Elevator Pitch:

This submission aims to deliver a high-performance, scalable and highly available endpoint for the public Helium API.

Total fiat/hnt ask:

$50 000 as listed in the published contract work opportunity

Team website:

This service will be built, maintained and operated by Net Stack Ltd, the company behind PUSHR

Project Details:

This API service is built on top of pairs of bare-metal 16-core AMD Ryzen servers powered by NVMe drives. Each server in a pair is load balanced and has an proactive failover setup with realtime monitoring. All pairs are load balanced with proactive failover capability on per-pair basis. Scalability is simplified and there is no practical limit on the pairs of servers that can be added, as well as no limit on their geographical distribution. The API is proxied via a global network of edge servers with proactive failover in the event of edge failure. Custom Layer 7 attack mitigation and rate limiting rules are applied on the network edge.
This service is designed to be commercialized for heavy use.

Roadmap:

Milestone + Date	Deliverable	Summary	Cost
MS1, 05 May, 2022	API service start	Bring the service up with a single pair of API servers. Confirm operational state. Start tuning the configuration of the rest of the infrastructure	0 USD
MS2, 16 May, 2022	Ready for demonstrative period	API servers and rest of infrastructure are ready to start a 3-month demonstrative period with public access for limited calls	0 USD
MS3, 16 August, 2022	Ready for heavy use	End of demonstrative period. Service is ready to scale up to demand for heavy use.	$50 000 USD

[chrisabruce]

Is this proposal just to use your CDN as a front to the Helium Public API? Or is this intended to attempt to run full ETL/Postgres/API?

[PUSHRCDN]

Is this proposal just to use your CDN as a front to the Helium Public API? Or is this intended to attempt to run full ETL/Postgres/API?

Hello, we intend to run full ETL, database and API, and will be proxying it via our CDN.

[chrisabruce]

Who is the intended audience of this and what amount of traffic are you expecting to handle? Is this expected to replace Helium public API? How are the ongoing operational costs to be calculated and paid?

The public API does well over 3bn requests/month and growing fast (it was under 1bn req/month 6 months ago). It bursts up to 300,000 req/sec (not a typo). It is running on over 1200 CPU cores for just the databases (20 replicas) and growing. All bare metal 60/100 core Ryzens. It currently uses Fastly for CDN. The API servers are also running on a cluster of servers. And all this is even with 1 req/sec rate limiter in place.

I am not sure, but the load, traffic, and amount of management effort might be being way underestimated?

[PUSHRCDN]

I guess there might be some misunderstanding. This proposal is based on https://gist.github.com/dansku/17daa20314610ab0f8b3a73a8dec38b8 , which is linked to from https://heliumfoundation.notion.site/The-Helium-Foundation-Grant-Program-18a0484cf8a1494bae9eb3497b77a3cf , which appears to seek decentralization for the API and to incentivize more public API endpoints to be created. The purpose is not to replace Helium's endpoint, If I understand it correctly, but to add another endpoint which could be used publicly to help offloading Helium's API. Operational costs are not mentioned in the gist, for which I have assumed that is subject open to discussion based on another document here https://docs.google.com/document/d/1KOQchoCyqnZYDUfYxMa1B7UL-oVUIqHCLPdUd-fNRaw/edit , under the Work opportunities section, which lists the option to commercialize for heavy use as a requirement. Please correct me if I am wrong.

[chrisabruce]

Can you try and estimate the load and duration that this setup would support for the project amount?