When graphql-mesh is run in k8s, 504 Gateway Time-out will appear stably.

[Mr-ZhaoRui]

My Mesh application is deployed as follows:

1. It's based on an Express Node.js server.
2. Docker image is built based on Node 18.
3. It's running in Kubernetes with the following resource allocations:

requests:
 memory: "2000Mi"
 cpu: "500m"
limits:
 memory: "3000Mi"
 cpu: "2000m"

After the pod starts successfully, the Mesh service can be accessed normally within ten minutes. However, after ten minutes, "504 Gateway Time-out" errors start to occur gradually. From the monitoring, CPU and memory usage do not spike or max out. The logs seem to get stuck at this point.

D 2024-03-15T08:29:49.899Z | v1.10.2 61 | retrying_call | [3360] Created child call [3361] for attempt 1
D 2024-03-15T08:29:49.899Z | v1.10.2 61 | load_balancing_call | [3361] start called
D 2024-03-15T08:29:49.899Z | v1.10.2 61 | load_balancing_call | [3361] Pick called
D 2024-03-15T08:29:49.899Z | v1.10.2 61 | load_balancing_call | [3361] Pick result: QUEUE subchannel: null status: undefined undefined
D 2024-03-15T08:29:49.899Z | v1.10.2 61 | retrying_call | [3360] startRead called
D 2024-03-15T08:29:49.899Z | v1.10.2 61 | load_balancing_call | [3361] startRead called
D 2024-03-15T08:29:49.899Z | v1.10.2 61 | retrying_call | [3360] write() called with message of length 9
D 2024-03-15T08:29:49.899Z | v1.10.2 61 | load_balancing_call | [3361] write() called with message of length 9
D 2024-03-15T08:29:49.900Z | v1.10.2 61 | retrying_call | [3360] halfClose called
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters
...

I'm unable to reproduce this issue on my development MacBook Pro.

[Mr-ZhaoRui]

Additional information: I found that directly building the image and running the image can reproduce this issue on my development MacBook Pro.

[ardatan]

How do you use Express Node server?
Mesh by default doesn't use Express. Could you give more details about your setup? Maybe share an example code piece?

[Mr-ZhaoRui]

Initially, I encountered the 504 issue when using mesh start. After that, I attempted to use Express.

express-service.ts

import express from "express";
import { createBuiltMeshHTTPHandler } from "./.mesh";

const app = express();

app.use("/graphql", createBuiltMeshHTTPHandler());

app.listen(4000, () => {
  console.log(`GraphQL API located at http://localhost:4000/graphql`);
});

Dockerfile

# use node:18 as the base image
FROM node:18.12.1

# define workdir
WORKDIR /opt/app

# copy file [package.json , yarn.lock] into container
# [package.json , yarn.lock] are the key files to build/run a Node.js project
COPY package.json ./

# download dependencies declared in package.json
RUN yarn config set registry https://registry.npmmirror.com && yarn install

# patch the bugs for @graphql-mesh/grpc
COPY patch-graphql-mesh-grpc.sh /opt/app
COPY index.js /opt/app
RUN yarn patch

# copy all the source code into workdir (should excule .idea .gitignore , ect)
COPY proto/ /opt/app/proto/
COPY sources/ /opt/app/sources/
COPY .meshrc.yml /opt/app
COPY express-server.ts /opt/app
COPY entrypoint.sh /opt/app
RUN chmod +x /opt/app/entrypoint.sh

# expose port for outbound traffic, here must be the same port as node.js project port
# the default node.js project port is 4000
EXPOSE 4000

# statement the command should be executed when docker image run
ENTRYPOINT ["/opt/app/entrypoint.sh"]

entrypoint.sh

#!/bin/bash

echo "Running command yarn mesh build..."
yarn mesh build

echo "Running command yarn mesh start..."
yarn ts-node express-server.ts

[ardatan]

Not sure if it is related but;

• mesh build is something you should run on buildtime not runtime, so you don't need to copy proto files etc. .mesh folder would be enough for mesh start
• Why do you patch grpc handler?

I'd recommend to run mesh build before building the image then mesh start on runtime, and also could you enable DEBUG env var in your container so we can get more verbose logs?

[Mr-ZhaoRui]

My endpoint.sh has been updated, but the 504 issue still exists.

endpoint.sh

#!/bin/bash

echo "Running command yarn mesh start..."
yarn mesh start

The logs seem to get stuck at this point.

D 2024-03-15T08:29:49.899Z | v1.10.2 61 | retrying_call | [3360] write() called with message of length 9
D 2024-03-15T08:29:49.899Z | v1.10.2 61 | load_balancing_call | [3361] write() called with message of length 9
D 2024-03-15T08:29:49.900Z | v1.10.2 61 | retrying_call | [3360] halfClose called

At the same time, the following log will be printed repeatedly

🕸️  Mesh 🐛 Parsing request to extract GraphQL parameters