Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add syncing_status query parameter to eth/v1/node/health #5790

Merged
merged 12 commits into from
Aug 1, 2023
22 changes: 16 additions & 6 deletions packages/api/src/beacon/routes/node.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,10 @@ export enum NodeHealth {
NOT_INITIALIZED_OR_ISSUES = HttpStatusCode.SERVICE_UNAVAILABLE,
}

export type NodeHealthOptions = {
syncingStatus?: number;
};

/**
* Read information about the beacon node.
*/
Expand Down Expand Up @@ -120,13 +124,13 @@ export type Api = {
/**
* Get health check
* Returns node health status in http status codes. Useful for load balancers.
*
* NOTE: This route does not return any value
*/
getHealth(): Promise<
getHealth(
options?: NodeHealthOptions
): Promise<
ApiClientResponse<
{[HttpStatusCode.OK]: void; [HttpStatusCode.PARTIAL_CONTENT]: void},
HttpStatusCode.SERVICE_UNAVAILABLE
HttpStatusCode.BAD_REQUEST | HttpStatusCode.SERVICE_UNAVAILABLE
>
>;
};
Expand All @@ -150,7 +154,7 @@ export type ReqTypes = {
getPeerCount: ReqEmpty;
getNodeVersion: ReqEmpty;
getSyncingStatus: ReqEmpty;
getHealth: ReqEmpty;
getHealth: {query: {syncing_status?: number}};
};

export function getReqSerializers(): ReqSerializers<Api, ReqTypes> {
Expand All @@ -171,7 +175,13 @@ export function getReqSerializers(): ReqSerializers<Api, ReqTypes> {
getPeerCount: reqEmpty,
getNodeVersion: reqEmpty,
getSyncingStatus: reqEmpty,
getHealth: reqEmpty,
getHealth: {
writeReq: (options) => ({
query: options?.syncingStatus !== undefined ? {syncing_status: options.syncingStatus} : {},
}),
parseReq: ({query}) => [{syncingStatus: query.syncing_status}],
schema: {query: {syncing_status: Schema.Uint}},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend moving the range check in the schema.

Copy link
Member Author

@nflaig nflaig Jul 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have an example API that does this? Based on current types this is not possible as far as I can see, we only support pre-defined schemas

export enum Schema {

We enforce basic validation through schema but the error response and logging is not ideal in my opinion

JSON API response

Schema validation error

[
  {
    "instancePath": "/syncing_status",
    "schemaPath": "#/properties/syncing_status/minimum",
    "keyword": "minimum",
    "params": {
      "comparison": ">=",
      "limit": 0
    },
    "message": "must be >= 0"
  }
]

API error

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Invalid syncing status code: 600"
}

Error log

Schema validation error

Eph 0/2 1.126[rest]            error: Req req-o getHealth error  querystring/syncing_status must be >= 0
Error: querystring/syncing_status must be >= 0
    at defaultSchemaErrorFormatter (/home/nico/projects/ethereum/lodestar/node_modules/fastify/lib/context.js:108:10)
    at wrapValidationError (/home/nico/projects/ethereum/lodestar/node_modules/fastify/lib/validation.js:228:17)
    at validate (/home/nico/projects/ethereum/lodestar/node_modules/fastify/lib/validation.js:157:16)
    at preValidationCallback (/home/nico/projects/ethereum/lodestar/node_modules/fastify/lib/handleRequest.js:92:25)
    at handler (/home/nico/projects/ethereum/lodestar/node_modules/fastify/lib/handleRequest.js:76:7)
    at handleRequest (/home/nico/projects/ethereum/lodestar/node_modules/fastify/lib/handleRequest.js:24:5)
    at runPreParsing (/home/nico/projects/ethereum/lodestar/node_modules/fastify/lib/route.js:569:5)
    at next (/home/nico/projects/ethereum/lodestar/node_modules/fastify/lib/hooks.js:179:7)
    at handleResolve (/home/nico/projects/ethereum/lodestar/node_modules/fastify/lib/hooks.js:196:5)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

API error

Eph 0/0 5.017[rest]             warn: Req req-9 getHealth failed reason=Invalid syncing status code: 600

In both cases as it is right now, it is better to use API error. Schema validation could be made more flexible but there are not that many values in the spec that enforce things like minimum / maximum.

},
};
}

Expand Down
17 changes: 7 additions & 10 deletions packages/api/src/beacon/server/node.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
import {ChainForkConfig} from "@lodestar/config";
import {Api, ReqTypes, routesData, getReturnTypes, getReqSerializers} from "../routes/node.js";
import {ServerRoutes, getGenericJsonServer} from "../../utils/server/index.js";
import {HttpStatusCode} from "../../utils/client/httpStatusCode.js";
import {ServerApi} from "../../interfaces.js";

export function getRoutes(config: ChainForkConfig, api: ServerApi<Api>): ServerRoutes<ServerApi<Api>, ReqTypes> {
// All routes return JSON, use a server auto-generator
const reqSerializers = getReqSerializers();

const serverRoutes = getGenericJsonServer<ServerApi<Api>, ReqTypes>(
{routesData, getReturnTypes, getReqSerializers},
config,
Expand All @@ -18,14 +18,11 @@ export function getRoutes(config: ChainForkConfig, api: ServerApi<Api>): ServerR
getHealth: {
...serverRoutes.getHealth,
handler: async (req, res) => {
try {
await api.getHealth();
res.raw.writeHead(HttpStatusCode.OK);
res.raw.end();
} catch (e) {
res.raw.writeHead(HttpStatusCode.INTERNAL_SERVER_ERROR);
res.raw.end();
}
const args = reqSerializers.getHealth.parseReq(req);
// Note: This type casting is required as per route definition getHealth
// does not return a value but since the internal API does not have access
// to response object it is required to set the HTTP status code here.
res.statusCode = (await api.getHealth(...args)) as unknown as number;
},
},
};
Expand Down
11 changes: 5 additions & 6 deletions packages/api/src/interfaces.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ export type APIClientHandler = (...args: any) => PromiseLike<ApiClientResponse>;
export type APIServerHandler = (...args: any) => PromiseLike<unknown>;

export type ApiClientSuccessResponse<S extends keyof any, T> = {ok: true; status: S; response: T; error?: never};
export type ApiClientSErrorResponse<S extends Exclude<HttpStatusCode, HttpSuccessCodes>> = {
export type ApiClientErrorResponse<S extends Exclude<HttpStatusCode, HttpSuccessCodes>> = {
ok: false;
status: S;
response?: never;
Expand All @@ -18,17 +18,16 @@ export type ApiClientResponse<
E extends Exclude<HttpStatusCode, HttpSuccessCodes> = Exclude<HttpStatusCode, HttpSuccessCodes>,
> =
| {[K in keyof S]: ApiClientSuccessResponse<K, S[K]>}[keyof S]
| {[K in E]: ApiClientSErrorResponse<K>}[E]
| ApiClientSErrorResponse<HttpStatusCode.INTERNAL_SERVER_ERROR>;
| {[K in E]: ApiClientErrorResponse<K>}[E]
| ApiClientErrorResponse<HttpStatusCode.INTERNAL_SERVER_ERROR>;

export type ApiClientResponseData<T extends ApiClientResponse> = T extends {ok: true; response: infer R} ? R : never;

export type GenericRequestObject = Record<string, unknown>;
export type GenericResponseObject = {code: (code: number) => void};
export type GenericOptions = Record<string, unknown>;

export type ServerApi<T extends Record<string, APIClientHandler>> = {
[K in keyof T]: (
...args: [...args: Parameters<T[K]>, req?: GenericRequestObject, res?: GenericResponseObject]
...args: [...args: Parameters<T[K]>, opts?: GenericOptions]
Copy link
Member Author

@nflaig nflaig Jul 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed req / res but added generic opts instead, we have API implementations that pass additional options, like publishBlock. Might not be a bad thing in general to allow this as API implementations are called internally as well.

return this.publishBlock(signedBlockOrContents, {ignoreIfKnown: true});

) => Promise<ApiClientResponseData<Resolves<T[K]>>>;
};

Expand Down
11 changes: 7 additions & 4 deletions packages/api/src/utils/client/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ export function getFetchOptsSerializer<Fn extends (...args: any) => any, ReqType
return {
url: urlFormater(req.params ?? {}),
method: routeDef.method,
query: req.query,
query: Object.keys(req.query ?? {}).length ? req.query : undefined,
Copy link
Member Author

@nflaig nflaig Jul 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes appending '?' to URL if query params are omitted, e.g. /eth/v1/node/health instead of /eth/v1/node/health?.

I would assume most/(all) HTTP servers will ignore this anyways, but it might be misleading if req.url is logged on the server.

body: req.body as unknown,
headers: req.headers,
routeId,
Expand Down Expand Up @@ -80,14 +80,17 @@ export function generateGenericJsonClient<
}
} catch (err) {
if (err instanceof HttpError) {
return {ok: false, error: {code: err.status, message: err.message, operationId: routeId}} as ReturnType<
Api[keyof Api]
>;
return {
ok: false,
status: err.status,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming this was just missed previously as the code here is not really type-safe, didn't find a quick solution to fix that but we should always set status in client response.

error: {code: err.status, message: err.message, operationId: routeId},
} as ReturnType<Api[keyof Api]>;
}

if (err instanceof TimeoutError) {
return {
ok: false,
status: HttpStatusCode.INTERNAL_SERVER_ERROR,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This status code is kinda wrong but 408 would also be wrong as it would indicate a server timeout whereas this error happens due to client timeout.

If server doesn't respond within client timeout we can assume a server issue, setting 500 here is the most accurate. Alternative would be to just throw the error here but this makes the error logs less informative if a timeout happens.

error: {code: HttpStatusCode.INTERNAL_SERVER_ERROR, message: err.message, operationId: routeId},
} as ReturnType<Api[keyof Api]>;
}
Expand Down
2 changes: 1 addition & 1 deletion packages/api/test/unit/beacon/testData/node.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ export const testData: GenericServerTestCases<Api> = {
res: {data: {headSlot: "1", syncDistance: "2", isSyncing: false, isOptimistic: true, elOffline: false}},
},
getHealth: {
args: [],
args: [{syncingStatus: 206}],
res: undefined,
},
};
17 changes: 14 additions & 3 deletions packages/beacon-node/src/api/impl/node/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -66,18 +66,29 @@ export function getNodeApi(
return {data: sync.getSyncStatus()};
},

async getHealth(_req, res) {
async getHealth(options) {
const syncingStatus = options?.syncingStatus;

if (syncingStatus != null && (syncingStatus < 100 || syncingStatus > 599)) {
throw new ApiError(400, `Invalid syncing status code: ${syncingStatus}`);
}

let healthStatus: number;

if (sync.getSyncStatus().isSyncing) {
// 206: Node is syncing but can serve incomplete data
res?.code(routes.node.NodeHealth.SYNCING);
healthStatus = syncingStatus ?? routes.node.NodeHealth.SYNCING;
} else {
// 200: Node is ready
res?.code(routes.node.NodeHealth.READY);
healthStatus = routes.node.NodeHealth.READY;
}
// else {
// 503: Node not initialized or having issues
// NOTE: Lodestar does not start its API until fully initialized, so this status can never be served
// }

// Health status is returned to allow route handler to set it as HTTP status code
return healthStatus as unknown as void;
},
};
}
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ describe("beacon node api", function () {
chain: {blsVerifyAllMainThread: true},
},
validatorCount,
logger: testLogger("Node-A", {level: LogLevel.info}),
logger: testLogger("Node-Synced", {level: LogLevel.info}),
});
client = getClient({baseUrl: `http://127.0.0.1:${restPort}`}, {config});
});
Expand Down Expand Up @@ -65,9 +65,8 @@ describe("beacon node api", function () {
});

it("should return 'el_offline' as 'true' when EL not available", async () => {
// Close first instance
await bn.close();
bn = await getDevBeaconNode({
const portElOffline = 9597;
const bnElOffline = await getDevBeaconNode({
params: {
...chainConfigDef,
// eslint-disable-next-line @typescript-eslint/naming-convention
Expand All @@ -82,18 +81,19 @@ describe("beacon node api", function () {
api: {
rest: {
enabled: true,
port: restPort,
port: portElOffline,
},
},
chain: {blsVerifyAllMainThread: true},
},
validatorCount: 5,
logger: testLogger("Node-A", {level: LogLevel.info}),
logger: testLogger("Node-EL-Offline", {level: LogLevel.info}),
});
const clientElOffline = getClient({baseUrl: `http://127.0.0.1:${portElOffline}`}, {config});

// To make BN communicate with EL, it needs to produce some blocks and for that need validators
const {validators} = await getAndInitDevValidators({
node: bn,
node: bnElOffline,
validatorClientCount: 1,
validatorsPerClient: validatorCount,
startIndex: 0,
Expand All @@ -102,12 +102,75 @@ describe("beacon node api", function () {
// Give node sometime to communicate with EL
await sleep(chainConfigDef.SECONDS_PER_SLOT * 2 * 1000);

const res = await client.node.getSyncingStatus();
const res = await clientElOffline.node.getSyncingStatus();
ApiError.assert(res);

expect(res.response.data.elOffline).to.eql(true);

await Promise.all(validators.map((v) => v.close()));
await bnElOffline.close();
});
});

describe("getHealth", () => {
const portSyncing = 9598;

let bnSyncing: BeaconNode;
let clientSyncing: Api;

before(async () => {
bnSyncing = await getDevBeaconNode({
params: chainConfigDef,
options: {
// Node won't consider itself synced without peers
sync: {isSingleNode: false},
api: {
rest: {
enabled: true,
port: portSyncing,
},
},
chain: {blsVerifyAllMainThread: true},
},
validatorCount,
logger: testLogger("Node-Syncing", {level: LogLevel.info}),
});
clientSyncing = getClient({baseUrl: `http://127.0.0.1:${portSyncing}`}, {config});
// Must at least wait for one slot else node considers itself synced pre/at genesis
await sleep(chainConfigDef.SECONDS_PER_SLOT * 1000);
});

after(async () => {
await bnSyncing.close();
});

it("should return 200 status code if node is ready", async () => {
const res = await client.node.getHealth();
expect(res.status).to.equal(200);
});

it("should return 206 status code if node is syncing", async () => {
const res = await clientSyncing.node.getHealth();
expect(res.status).to.equal(206);
});

it("should return custom status code from 'syncing_status' query parameter if node is syncing", async () => {
const statusCode = 204;
const res = await clientSyncing.node.getHealth({syncingStatus: statusCode});
expect(res.status).to.equal(statusCode);
});

it("should only use status code from 'syncing_status' query parameter if node is syncing", async () => {
const res = await client.node.getHealth({syncingStatus: 204});
expect(res.status).to.equal(200);
});

it("should return 400 status code if value of 'syncing_status' query parameter is invalid", async () => {
const res = await clientSyncing.node.getHealth({syncingStatus: 99});
expect(res.status).to.equal(400);

const resp = await clientSyncing.node.getHealth({syncingStatus: 600});
expect(resp.status).to.equal(400);
});
});
});
Loading