Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add error msg in the logs when gw req fails #5369

Merged
merged 7 commits into from
Dec 18, 2024
+38 −13
Merged

chore: add error msg in the logs when gw req fails #5369

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d006255
chore: add error msg in the logs when gw req fails
mihir20 Dec 12, 2024
3de51fc
addressed comments
mihir20 Dec 13, 2024
1b5b7ab
addressed comments
mihir20 Dec 16, 2024
9da44bf
update go version and schema version
mihir20 Dec 18, 2024
e540fdc
update go version in docker file
mihir20 Dec 18, 2024
18445f0
downgrade swagger-editor-validate
mihir20 Dec 18, 2024
3d9265b
update action to apidom-validate
mihir20 Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/verify.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
if: ${{ failure() }}
run: echo 'Not formatted files. Ensure you have run `make fmt` and committed the files locally.'
- name: Validate OpenAPI definition
uses: char0n/swagger-editor-validate@v1
uses: char0n/apidom-validate@v1
with:
definition-file: gateway/openapi.yaml

Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
# syntax=docker/dockerfile:1

# GO_VERSION is updated automatically to match go.mod, see Makefile
ARG GO_VERSION=1.23.3
ARG GO_VERSION=1.23.4
ARG ALPINE_VERSION=3.20
FROM golang:${GO_VERSION}-alpine${ALPINE_VERSION} AS builder
ARG VERSION
ARG REVISION
ARG COMMIT_HASH
ARG ENTERPRISE_TOKEN

Check warning on line 11 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-oss (self-hosted, Linux, ARM64, rudderlabs/develop-rudder-server:mihir-add-logs-arm64, org...

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "ENTERPRISE_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 11 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-oss (self-hosted, Linux, ARM64, rudderlabs/develop-rudder-server:mihir-add-logs-arm64, org...

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "ENTERPRISE_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 11 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-ent (ubuntu-latest, rudderstack/develop-rudder-server-enterprise:mihir-add-logs-amd64, org...

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "ENTERPRISE_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 11 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-ent (ubuntu-latest, rudderstack/develop-rudder-server-enterprise:mihir-add-logs-amd64, org...

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "ENTERPRISE_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 11 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-oss (ubuntu-latest, rudderlabs/develop-rudder-server:mihir-add-logs-amd64, org.opencontain...

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "ENTERPRISE_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 11 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-oss (ubuntu-latest, rudderlabs/develop-rudder-server:mihir-add-logs-amd64, org.opencontain...

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "ENTERPRISE_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 11 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-ent (self-hosted, Linux, ARM64, rudderstack/develop-rudder-server-enterprise:mihir-add-log...

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "ENTERPRISE_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 11 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-ent (self-hosted, Linux, ARM64, rudderstack/develop-rudder-server-enterprise:mihir-add-log...

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "ENTERPRISE_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
ARG RACE_ENABLED=false
ARG CGO_ENABLED=0
ARG PKG_NAME=github.com/rudderlabs/release-demo
Expand Down
31 changes: 28 additions & 3 deletions gateway/handle.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -709,11 +709,11 @@
errorMessage = err.Error()
status = response.GetErrorStatusCode(errorMessage)
responseBody = response.GetStatus(errorMessage)
gw.logger.Infon("response",
mihir20 marked this conversation as resolved.
Show resolved Hide resolved
gw.logger.Errorn("response",
logger.NewStringField("ip", kithttputil.GetRequestIP(r)),
logger.NewStringField("path", r.URL.Path),
logger.NewIntField("status", int64(status)),
logger.NewStringField("body", responseBody),
obskit.Error(err),
)
gw.logger.Debugn("response",
logger.NewStringField("ip", kithttputil.GetRequestIP(r)),
Expand Down Expand Up @@ -763,13 +763,16 @@
if err != nil {
stat.RequestFailed(response.InvalidJSON)
stat.Report(gw.stats)
gw.logger.Errorn("invalid json in request",
obskit.Error(err))
return nil, errors.New(response.InvalidJSON)
}
gw.requestSizeStat.Observe(float64(len(body)))

if len(messages) == 0 {
stat.RequestFailed(response.NotRudderEvent)
stat.Report(gw.stats)
gw.logger.Errorn("no messages in request")
return nil, errors.New(response.NotRudderEvent)
}

Expand All @@ -779,7 +782,10 @@
stat := gwstats.SourceStat{ReqType: reqType}
err := gw.streamMsgValidator(&msg)
if err != nil {
gw.logger.Errorn("invalid message in request", logger.NewErrorField(err))
loggerFields := msg.Properties.LoggerFields()
loggerFields = append(loggerFields, obskit.Error(err))
gw.logger.Errorn("invalid message in request",
loggerFields...)

Check warning on line 788 in gateway/handle.go

View check run for this annotation

Codecov / codecov/patch

gateway/handle.go#L785-L788 

Added lines #L785 - L788 were not covered by tests
stat.RequestEventsFailed(1, response.InvalidStreamMessage)
stat.Report(gw.stats)
return nil, errors.New(response.InvalidStreamMessage)
Expand All @@ -793,6 +799,9 @@
if err != nil {
stat.RequestEventsFailed(1, response.NotRudderEvent)
stat.Report(gw.stats)
loggerFields := msg.Properties.LoggerFields()
loggerFields = append(loggerFields, obskit.Error(err))
gw.logger.Errorn("failed to set type in message", loggerFields...)

Check warning on line 804 in gateway/handle.go

View check run for this annotation

Codecov / codecov/patch

gateway/handle.go#L802-L804 

Added lines #L802 - L804 were not covered by tests
return nil, errors.New(response.NotRudderEvent)
}
}
Expand All @@ -806,19 +815,27 @@
if err != nil {
stat.RequestFailed(response.NotRudderEvent)
stat.Report(gw.stats)
gw.logger.Errorn("failed to set messageID in message",
obskit.Error(err))

Check warning on line 819 in gateway/handle.go

View check run for this annotation

Codecov / codecov/patch

gateway/handle.go#L818-L819 

Added lines #L818 - L819 were not covered by tests
return nil, errors.New(response.NotRudderEvent)
}
}
rudderId, err := getRudderId(userIDFromReq, anonIDFromReq)
if err != nil {
stat.RequestFailed(response.NotRudderEvent)
stat.Report(gw.stats)
gw.logger.Errorn("failed to get rudderId",
obskit.Error(err))

Check warning on line 828 in gateway/handle.go

View check run for this annotation

Codecov / codecov/patch

gateway/handle.go#L827-L828 

Added lines #L827 - L828 were not covered by tests
return nil, errors.New(response.NotRudderEvent)
}
msg.Payload, err = sjson.SetBytes(msg.Payload, "rudderId", rudderId.String())
if err != nil {
stat.RequestFailed(response.NotRudderEvent)
stat.Report(gw.stats)
loggerFields := msg.Properties.LoggerFields()
loggerFields = append(loggerFields, obskit.Error(err))
gw.logger.Errorn("failed to set rudderId in message",
loggerFields...)

Check warning on line 838 in gateway/handle.go

View check run for this annotation

Codecov / codecov/patch

gateway/handle.go#L835-L838 

Added lines #L835 - L838 were not covered by tests
return nil, errors.New(response.NotRudderEvent)
}
writeKey, ok := gw.getWriteKeyFromSourceID(msg.Properties.SourceID)
Expand Down Expand Up @@ -877,13 +894,17 @@
err = fmt.Errorf("filling receivedAt: %w", err)
stat.RequestEventsFailed(1, err.Error())
stat.Report(gw.stats)
gw.logger.Errorn("failed to fill receivedAt in message",
obskit.Error(err))

Check warning on line 898 in gateway/handle.go

View check run for this annotation

Codecov / codecov/patch

gateway/handle.go#L897-L898 

Added lines #L897 - L898 were not covered by tests
return nil, fmt.Errorf("filling receivedAt: %w", err)
}
msg.Payload, err = fillRequestIP(msg.Payload, msg.Properties.RequestIP)
if err != nil {
err = fmt.Errorf("filling request_ip: %w", err)
stat.RequestEventsFailed(1, err.Error())
stat.Report(gw.stats)
gw.logger.Errorn("failed to fill request_ip in message",
obskit.Error(err))

Check warning on line 907 in gateway/handle.go

View check run for this annotation

Codecov / codecov/patch

gateway/handle.go#L906-L907 

Added lines #L906 - L907 were not covered by tests
return nil, fmt.Errorf("filling request_ip: %w", err)
}

Expand All @@ -899,6 +920,10 @@
err = fmt.Errorf("marshalling event batch: %w", err)
stat.RequestEventsFailed(1, err.Error())
stat.Report(gw.stats)
loggerFields := msg.Properties.LoggerFields()
loggerFields = append(loggerFields, obskit.Error(err))
gw.logger.Errorn("failed to marshal event batch",
loggerFields...)

Check warning on line 926 in gateway/handle.go

View check run for this annotation

Codecov / codecov/patch

gateway/handle.go#L923-L926 

Added lines #L923 - L926 were not covered by tests
return nil, fmt.Errorf("marshalling event batch: %w", err)
}
jobUUID := uuid.New()
Expand Down
6 changes: 3 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/rudderlabs/rudder-server

go 1.23.3
go 1.23.4

// Addressing snyk vulnerabilities in indirect dependencies
// When upgrading a dependency, please make sure that
Expand Down Expand Up @@ -78,7 +78,7 @@ require (
github.com/rudderlabs/compose-test v0.1.3
github.com/rudderlabs/rudder-go-kit v0.45.0
github.com/rudderlabs/rudder-observability-kit v0.0.3
github.com/rudderlabs/rudder-schemas v0.5.3
github.com/rudderlabs/rudder-schemas v0.5.4
github.com/rudderlabs/rudder-transformer/go v0.0.0-20240910055720-f77d2ab4125a
github.com/rudderlabs/sql-tunnels v0.1.7
github.com/rudderlabs/sqlconnect-go v1.13.0
Expand Down Expand Up @@ -214,7 +214,7 @@ require (
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.22.1 // indirect
github.com/go-playground/validator/v10 v10.23.0 // indirect
github.com/go-sql-driver/mysql v1.8.1 // indirect
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
github.com/go-viper/mapstructure/v2 v2.1.0 // indirect
Expand Down
8 changes: 4 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -587,8 +587,8 @@ github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/Nu
github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.20.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM=
github.com/go-playground/validator/v10 v10.22.1 h1:40JcKH+bBNGFczGuoBYgX4I6m/i27HYW8P9FDk5PbgA=
github.com/go-playground/validator/v10 v10.22.1/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM=
github.com/go-playground/validator/v10 v10.23.0 h1:/PwmTwZhS0dPkav3cdK9kV1FsAmrL8sThn8IHr/sO+o=
github.com/go-playground/validator/v10 v10.23.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM=
github.com/go-redis/redis v6.15.8+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
github.com/go-redis/redis v6.15.9+incompatible h1:K0pv1D7EQUjfyoMql+r/jZqCLizCGKFlFgcHWWmHQjg=
github.com/go-redis/redis v6.15.9+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
Expand Down Expand Up @@ -1175,8 +1175,8 @@ github.com/rudderlabs/rudder-go-kit v0.45.0 h1:y8ModVsl2rAdqnqv/di82cddkaEBT1qZv
github.com/rudderlabs/rudder-go-kit v0.45.0/go.mod h1:NrHCi0KSzHSMFXQu0t2kgJcE4ClAKklVXfb2glADvQ4=
github.com/rudderlabs/rudder-observability-kit v0.0.3 h1:vZtuZRkGX+6rjaeKtxxFE2YYP6QlmAcVcgecTOjvz+Q=
github.com/rudderlabs/rudder-observability-kit v0.0.3/go.mod h1:6UjAh3H6rkE0fFLh7z8ZGQEQbKtUkRfhWOf/OUhfqW8=
github.com/rudderlabs/rudder-schemas v0.5.3 h1:IWWjAo2TzsjwHNhS2EAr1+0MjvA8BoTpJvB2o/GFwNU=
github.com/rudderlabs/rudder-schemas v0.5.3/go.mod h1:iUpjG/Zb+ioZcNLvXNYXSKQ2LpPlsIDBfxfCDH9ue/E=
github.com/rudderlabs/rudder-schemas v0.5.4 h1:QzI6vIC38W0jGJu6E0vdwh4IAtSKx8ziqff+JL0xmIE=
github.com/rudderlabs/rudder-schemas v0.5.4/go.mod h1:oypQtew1d2jBGJdFMT4zf1XOvXgGEcNPtRFy6vRDAv8=
github.com/rudderlabs/rudder-transformer/go v0.0.0-20240910055720-f77d2ab4125a h1:OZcvpApxEYNkB9UNXrKDUBufQ24Lsr2Cs0pw70tzXBw=
github.com/rudderlabs/rudder-transformer/go v0.0.0-20240910055720-f77d2ab4125a/go.mod h1:3NGitPz4pYRRZ6Xt09S+8hb0tHK/9pZcKJ3OgOTaSmE=
github.com/rudderlabs/sql-tunnels v0.1.7 h1:wDCRl6zY4M5gfWazf7XkSTGQS3yjBzUiUgEMBIfHNDA=
Expand Down
2 changes: 1 addition & 1 deletion suppression-backup-service/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# syntax=docker/dockerfile:1

# GO_VERSION is updated automatically to match go.mod, see Makefile
ARG GO_VERSION=1.23.3
ARG GO_VERSION=1.23.4
ARG ALPINE_VERSION=3.20
FROM golang:${GO_VERSION}-alpine${ALPINE_VERSION} AS builder
RUN mkdir /app
Expand Down
Loading