Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dune mock api service #2

Merged
merged 58 commits into from
Sep 17, 2024
Merged
Show file tree
Hide file tree
Changes from 53 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
489f9b1
added controller,service and module for events-webhook route
GuticaStefan Aug 21, 2024
e154c97
Work in progress
GuticaStefan Aug 21, 2024
e6c35fa
Work in progress
GuticaStefan Aug 22, 2024
1a82482
Computed liquidity and data formated for addLiquidity/removeLiquidity…
GuticaStefan Aug 22, 2024
de8a197
Add events parsing and liquidity computation
MogageNicolae Aug 23, 2024
fb9c12d
Add prices fetch
MogageNicolae Aug 23, 2024
1bf2bcf
Start even fetching script
MogageNicolae Aug 23, 2024
6bbd85c
Finish script
MogageNicolae Aug 26, 2024
ead0706
Added data service to fetch token price by day and cache it
GuticaStefan Aug 26, 2024
8f3c391
Improvements
GuticaStefan Aug 26, 2024
9236dac
Fetch events script update
GuticaStefan Aug 26, 2024
2545b15
Fix minor bugs
MogageNicolae Aug 27, 2024
a5f058b
Working on dune sender service
GuticaStefan Aug 27, 2024
8833d74
created class manager for and automated creating tables and sending d…
GuticaStefan Aug 28, 2024
5e49432
Add redis cache for records
MogageNicolae Aug 29, 2024
cf2a141
Change records sending to dune logic
MogageNicolae Aug 29, 2024
8089f53
Add OriginLogger
MogageNicolae Aug 29, 2024
196e8d2
Bug fixes
GuticaStefan Aug 30, 2024
905cd3c
WIP
GuticaStefan Aug 30, 2024
c602f54
.
GuticaStefan Aug 30, 2024
7fc5778
WIP
GuticaStefan Aug 30, 2024
3221b91
Parse events fields from hex to base64
GuticaStefan Aug 30, 2024
00066ae
Add default price for tokens if not available on data-api
GuticaStefan Aug 30, 2024
17bb653
Project clean
GuticaStefan Aug 30, 2024
acc98dc
Bug fix
GuticaStefan Aug 30, 2024
f0fb545
Create Dune Mock service WIP
GuticaStefan Aug 30, 2024
cefefc0
Added dune mock app API
GuticaStefan Sep 2, 2024
c130a60
Added script to generate area chart for each pair present in mongodb
GuticaStefan Sep 2, 2024
371dc2a
store any type of csv in mongo and code improvements
GuticaStefan Sep 3, 2024
b038eb4
Linter update
GuticaStefan Sep 3, 2024
7684f18
dune-mock renamed to dune-simulator, update linter, code improvements
GuticaStefan Sep 3, 2024
d14726f
update package.json for dune-simulator renaming
GuticaStefan Sep 3, 2024
22b4cf3
replicate dune insert data into table on dune simulator
GuticaStefan Sep 3, 2024
5a00402
linter fixes
GuticaStefan Sep 3, 2024
f0d03b0
replicate dune create table on dune simulator
GuticaStefan Sep 4, 2024
499a7d4
replicate dune create table on dune simulator
GuticaStefan Sep 4, 2024
1bc3e90
dune api url from env accepts either dune url or dune simulator url
GuticaStefan Sep 4, 2024
adf97b9
route path for create table fix
GuticaStefan Sep 4, 2024
5f79a83
Change Locker.lock with RedLockService for csv-records class manager
GuticaStefan Sep 5, 2024
3def0cc
Created endpoint that returns a chart image for a specified pair (not…
GuticaStefan Sep 5, 2024
37e1b0f
chart png can be created regardless the amount of points, added new e…
GuticaStefan Sep 6, 2024
ae37c39
Added service parser for HATOM borrowing events + bug fixes
GuticaStefan Sep 9, 2024
dc3be7d
denominate usdt
GuticaStefan Sep 12, 2024
9c59b40
test
GuticaStefan Sep 12, 2024
bd28865
test
GuticaStefan Sep 12, 2024
cbcf96b
add retry interval for redlock
GuticaStefan Sep 12, 2024
9d758af
synchronization bug fix on hatom events
GuticaStefan Sep 12, 2024
d177309
parse events into a table
GuticaStefan Sep 13, 2024
7e254dd
test
GuticaStefan Sep 13, 2024
2d59ecd
cron every 1 minute
GuticaStefan Sep 13, 2024
88b0d37
test
GuticaStefan Sep 13, 2024
006c84f
store csv files as dynamic collections
GuticaStefan Sep 16, 2024
96c57fa
added xaxis and yaxis as url param to generate custom charts
GuticaStefan Sep 16, 2024
481d55e
Rename hatomEventsService to hatomBorrowEventsService
GuticaStefan Sep 17, 2024
7a74378
added util joinCsvAttributes function
GuticaStefan Sep 17, 2024
96d1d06
convert all headers and csvFileNames to snakeCase in csvRecords class…
GuticaStefan Sep 17, 2024
db9b9ce
decode event identifier and compare to readable string borrow_event
GuticaStefan Sep 17, 2024
cd5e35b
code improvements
GuticaStefan Sep 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .env.devnet
Original file line number Diff line number Diff line change
@@ -1,2 +1,8 @@
NETWORK=devnet
API_URL=https://devnet-api.multiversx.com
NETWORK=mainnet
API_URL=https://api.multiversx.com
DATA_API_CEX_URL=https://data-api.multiversx.com/v1/quotes/cex
DATA_API_XEXCHANGE_URL=https://data-api.multiversx.com/v1/quotes/xexchange
# DUNE_API_URL=http://localhost:3001/api/v1/table
DUNE_API_URL=https://api.dune.com/api/v1/table
DUNE_NAMESPACE=stefanmvx
DUNE_API_KEY=
6 changes: 6 additions & 0 deletions .env.mainnet
Original file line number Diff line number Diff line change
@@ -1,2 +1,8 @@
NETWORK=mainnet
API_URL=https://api.multiversx.com
DATA_API_CEX_URL=https://data-api.multiversx.com/v1/quotes/cex
DATA_API_XEXCHANGE_URL=https://data-api.multiversx.com/v1/quotes/xexchange
DUNE_API_URL=http://localhost:3001/api/v1/table
# DUNE_API_URL=https://api.dune.com/api/v1/table
DUNE_NAMESPACE=stefanmvx
DUNE_API_KEY=test
10 changes: 8 additions & 2 deletions .env.testnet
Original file line number Diff line number Diff line change
@@ -1,2 +1,8 @@
NETWORK=testnet
API_URL=https://testnet-api.multiversx.com
NETWORK=mainnet
API_URL=https://api.multiversx.com
DATA_API_CEX_URL=https://data-api.multiversx.com/v1/quotes/cex
DATA_API_XEXCHANGE_URL=https://data-api.multiversx.com/v1/quotes/xexchange
# DUNE_API_URL=http://localhost:3001/api/v1/table
DUNE_API_URL=https://api.dune.com/api/v1/table
DUNE_NAMESPACE=stefanmvx
DUNE_API_KEY=
26 changes: 5 additions & 21 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -28,16 +28,8 @@ module.exports = {
pattern: 'apps/api',
},
{
type: 'apps/cache-warmer',
pattern: 'apps/cache-warmer',
},
{
type: 'apps/queue-worker',
pattern: 'apps/queue-worker',
},
{
type: 'apps/transactions-processor',
pattern: 'apps/transactions-processor',
type: 'apps/dune-simulator',
pattern: 'apps/dune-simulator',
},
{
type: 'libs/common',
Expand Down Expand Up @@ -82,24 +74,16 @@ module.exports = {
allow: ['libs/common', 'libs/entities', 'libs/services']
},
{
from: 'apps/cache-warmer',
allow: ['libs/common', 'libs/entities', 'libs/services']
},
{
from: 'apps/queue-worker',
allow: ['libs/common', 'libs/entities', 'libs/services']
},
{
from: 'apps/transactions-processor',
from: 'apps/dune-simulator',
allow: ['libs/common', 'libs/entities', 'libs/services']
},
{
from: 'libs/database',
allow: ['libs/common', 'libs/entities']
allow: ['libs/common', 'libs/entities', 'apps/dune-simulator']
},
{
from: 'libs/services',
allow: ['libs/common', 'libs/entities', 'libs/database']
allow: ['libs/common', 'libs/entities', 'libs/database', 'apps/api', 'apps/dune-simulator']
},
{
from: 'libs/common',
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -47,3 +47,6 @@ lerna-debug.log*
# Environment
.env
.env.custom
/scripts/insertCsv.ts
scripts/
out_csv/
18 changes: 11 additions & 7 deletions .multiversx/config/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,18 @@ apps:
port: 3000
privatePort: 4000
useCachingInterceptor: true
cacheWarmer:
port: 4001
queueWorker:
port: 4002
transactionsProcessor:
port: 4003
maxLookBehind: 100
duneSimulator:
port: 3001
privatePort: 4001
useCachingInterceptor: true
libs:
common:
network: ${NETWORK}
urls:
api: ${API_URL}
dataApiCex: ${DATA_API_CEX_URL}
dataApiXexchange: ${DATA_API_XEXCHANGE_URL}
duneApi: ${DUNE_API_URL}
database:
host: 'localhost'
port: 27017
Expand All @@ -25,6 +25,10 @@ libs:
redis:
host: '127.0.0.1'
port: 6379
features:
dune:
namespace: ${DUNE_NAMESPACE}
apiKey: ${DUNE_API_KEY}
nativeAuth:
# maxExpirySeconds:
acceptedOrigins:
Expand Down
22 changes: 0 additions & 22 deletions TODO

This file was deleted.

24 changes: 24 additions & 0 deletions apps/api/src/config/app-config.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,28 @@ import { Injectable } from "@nestjs/common";
@Injectable()
export class AppConfigService {
readonly config = configuration().apps.api;

getDuneNamespace(): string {
return configuration().libs.common.features.dune.namespace ?? "";
}

getDuneApiKey(): string {
return configuration().libs.common.features.dune.apiKey ?? "";
}

getApiUrl(): string {
return configuration().libs.common.urls.api ?? "";
}

getDataApiCexUrl(): string {
return configuration().libs.common.urls.dataApiCex ?? "";
}

getDataApiXexchangeUrl(): string {
return configuration().libs.common.urls.dataApiXexchange ?? "";
}

getDuneApiUrl(): string {
return configuration().libs.common.urls.duneApi ?? "";
}
}
16 changes: 0 additions & 16 deletions apps/api/src/endpoints/auth/auth.controller.ts

This file was deleted.

13 changes: 0 additions & 13 deletions apps/api/src/endpoints/auth/auth.module.ts

This file was deleted.

10 changes: 2 additions & 8 deletions apps/api/src/endpoints/endpoints.module.ts
Original file line number Diff line number Diff line change
@@ -1,16 +1,10 @@
import { Module } from "@nestjs/common";
import { DynamicModuleUtils } from "@libs/common";
import { AuthModule } from "./auth/auth.module";
import { TokenModule } from "./token/token.module";
import { UserModule } from "./user/user.module";
import { ExampleModule } from "./example/example.module";
import { EventsModule } from "./events/events.module";

@Module({
imports: [
AuthModule,
TokenModule,
UserModule,
ExampleModule,
EventsModule,
],
providers: [
DynamicModuleUtils.getNestJsApiConfigService(),
Expand Down
48 changes: 48 additions & 0 deletions apps/api/src/endpoints/events/entities/event.log.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import { ApiProperty } from "@nestjs/swagger";

export class EventLog {
constructor(init?: Partial<EventLog>) {
Object.assign(this, init);
}
@ApiProperty()
eventId: string = '';

@ApiProperty()
txHash: string = '';

@ApiProperty()
shardId: number = 0;

@ApiProperty()
timestamp: number = 0;

@ApiProperty()
address: string = '';

@ApiProperty()
identifier: string = '';

@ApiProperty()
topics: string[] = [];

@ApiProperty()
data: string = '';

@ApiProperty()
additionalData: string[] = [];

@ApiProperty()
txOrder: number = 0;

@ApiProperty()
eventOrder: number = 0;
}

export class EventsLog {
constructor(init?: Partial<EventsLog>) {
Object.assign(this, init);
}

@ApiProperty({ type: EventLog, isArray: true })
events: EventLog[] = [];
}
1 change: 1 addition & 0 deletions apps/api/src/endpoints/events/entities/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from "./event.log";
29 changes: 29 additions & 0 deletions apps/api/src/endpoints/events/events.controller.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import { Body, Controller, Param, Post } from "@nestjs/common";
import { ApiTags } from "@nestjs/swagger";
import { EventLog } from "./entities/event.log";
import { HatomEventsService, LiquidityEventsService } from "@libs/services/events";

@Controller('/events')
@ApiTags('events')
export class EventsController {
constructor(
private readonly liquidityService: LiquidityEventsService,
private readonly hatomService: HatomEventsService,
) { }

@Post("/liquidity-webhook")
async liquidityWebhook(
@Body() body: EventLog[],
): Promise<void> {
await this.liquidityService.liquidityWebhook(body);
}

@Post("/hatom-webhook/:borrowed_token")
async hatomWebhook(
@Body() body: EventLog[],
@Param('borrowed_token') borrowedToken: string,
): Promise<void> {
await this.hatomService.hatomWebhook(body, borrowedToken);
}

}
13 changes: 13 additions & 0 deletions apps/api/src/endpoints/events/events.module.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import { Module } from "@nestjs/common";
import { ServicesModule } from "@libs/services/services.module";
import { EventsController } from "./events.controller";

@Module({
imports: [
ServicesModule,
],
controllers: [
EventsController,
],
})
export class EventsModule { }
16 changes: 0 additions & 16 deletions apps/api/src/endpoints/example/entities/event.log.ts

This file was deleted.

58 changes: 0 additions & 58 deletions apps/api/src/endpoints/example/example.controller.ts

This file was deleted.

Loading
Loading