Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VTTablet: Transmit raw MySQL packets for Fields and Rows over wire with grpc buffer #17135

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
+2,231 −1,099
Draft

VTTablet: Transmit raw MySQL packets for Fields and Rows over wire with grpc buffer #17135

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
548a8b6
proto changes
harshit-gangal Nov 4, 2024
ed6e267
store fields and rows as raw mysql packets in mem.Buffer slice
harshit-gangal Nov 4, 2024
6313ead
use mysql execution options
harshit-gangal Nov 5, 2024
ab80841
store ok packets as raw and add parsing of raw packets for VTGate to …
harshit-gangal Nov 8, 2024
5c7b422
fix data to strip 5 bytes of proto header to extract info
harshit-gangal Nov 8, 2024
f28cfe7
wip: attempt to fix encoding
harshit-gangal Nov 8, 2024
40e8771
wip
harshit-gangal Nov 11, 2024
3991d28
wip: fix slice pointer
harshit-gangal Nov 12, 2024
846022c
only request raw packets for Execute API
harshit-gangal Nov 12, 2024
13af807
Empty-Commit
harshit-gangal Nov 12, 2024
0ffc6a1
removed all log lines
harshit-gangal Nov 12, 2024
c0fd742
Merge remote-tracking branch 'upstream/main' into raw-mysqlpackets
harshit-gangal Nov 12, 2024
f331825
benchmark vttablet query execution
harshit-gangal Nov 13, 2024
21ebb1d
upgrade go modules for grpc to v1.68 for a fix
harshit-gangal Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 93 additions & 75 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module vitess.io/vitess
go 1.23.3

require (
cloud.google.com/go/storage v1.43.0
cloud.google.com/go/storage v1.46.0
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6
github.com/Azure/azure-pipeline-go v0.2.3
github.com/Azure/azure-storage-blob-go v0.15.0
Expand All @@ -15,9 +15,9 @@ require (
github.com/corpix/uarand v0.1.1 // indirect
github.com/dave/jennifer v1.7.1
github.com/evanphx/json-patch v5.9.0+incompatible
github.com/fsnotify/fsnotify v1.7.0
github.com/fsnotify/fsnotify v1.8.0
github.com/go-sql-driver/mysql v1.7.1
github.com/golang/glog v1.2.2
github.com/golang/glog v1.2.3
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v0.0.4
github.com/google/go-cmp v0.6.0
Expand All @@ -27,74 +27,74 @@ require (
github.com/gorilla/mux v1.8.1
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/hashicorp/consul/api v1.29.4
github.com/hashicorp/consul/api v1.30.0
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/serf v0.10.1 // indirect
github.com/icrowley/fake v0.0.0-20180203215853-4178557ae428
github.com/klauspost/compress v1.17.9
github.com/klauspost/compress v1.17.11
github.com/klauspost/pgzip v1.2.6
github.com/krishicks/yaml-patch v0.0.10
github.com/magiconair/properties v1.8.7 // indirect
github.com/minio/minio-go v0.0.0-20190131015406-c8a261de75c1
github.com/montanaflynn/stats v0.7.1
github.com/olekukonko/tablewriter v0.0.5
github.com/opentracing-contrib/go-grpc v0.0.0-20240724223109-9dec25a38fa8
github.com/opentracing-contrib/go-grpc v0.1.0
github.com/opentracing/opentracing-go v1.2.0
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/philhofer/fwd v1.1.3-0.20240612014219-fbbf4953d986 // indirect
github.com/philhofer/fwd v1.1.3-0.20240916144458-20a13a1f6b7c // indirect
github.com/pierrec/lz4 v2.6.1+incompatible
github.com/pires/go-proxyproto v0.7.0
github.com/pires/go-proxyproto v0.8.0
github.com/pkg/errors v0.9.1
github.com/planetscale/pargzip v0.0.0-20201116224723-90c7fc03ea8a
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10
github.com/prometheus/client_golang v1.20.3
github.com/prometheus/common v0.59.1
github.com/planetscale/vtprotobuf v0.6.1-0.20241011083415-71c992bc3c87
github.com/prometheus/client_golang v1.20.5
github.com/prometheus/common v0.60.1
github.com/sjmudd/stopwatch v0.1.1
github.com/soheilhy/cmux v0.1.5
github.com/spf13/cobra v1.8.1
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.19.0
github.com/stretchr/testify v1.9.0
github.com/tchap/go-patricia v2.3.0+incompatible
github.com/tidwall/gjson v1.17.3
github.com/tinylib/msgp v1.2.1 // indirect
github.com/tidwall/gjson v1.18.0
github.com/tinylib/msgp v1.2.4 // indirect
github.com/uber/jaeger-client-go v2.30.0+incompatible
github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82
github.com/z-division/go-zookeeper v1.0.0
go.etcd.io/etcd/api/v3 v3.5.16
go.etcd.io/etcd/client/pkg/v3 v3.5.16
go.etcd.io/etcd/client/v3 v3.5.16
go.etcd.io/etcd/api/v3 v3.5.17
go.etcd.io/etcd/client/pkg/v3 v3.5.17
go.etcd.io/etcd/client/v3 v3.5.17
go.uber.org/mock v0.2.0
golang.org/x/crypto v0.27.0 // indirect
golang.org/x/mod v0.21.0 // indirect
golang.org/x/net v0.29.0
golang.org/x/oauth2 v0.23.0
golang.org/x/sys v0.25.0
golang.org/x/term v0.24.0
golang.org/x/text v0.18.0 // indirect
golang.org/x/time v0.6.0
golang.org/x/tools v0.25.0
google.golang.org/api v0.197.0
google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/grpc v1.66.2
golang.org/x/crypto v0.29.0 // indirect
golang.org/x/mod v0.22.0 // indirect
golang.org/x/net v0.31.0
golang.org/x/oauth2 v0.24.0
golang.org/x/sys v0.27.0
golang.org/x/term v0.26.0
golang.org/x/text v0.20.0 // indirect
golang.org/x/time v0.8.0
golang.org/x/tools v0.27.0
google.golang.org/api v0.205.0
google.golang.org/genproto v0.0.0-20241104194629-dd2ea8efbc28 // indirect
google.golang.org/grpc v1.68.0
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.3.0
google.golang.org/grpc/examples v0.0.0-20210430044426-28078834f35b
google.golang.org/protobuf v1.34.2
gopkg.in/DataDog/dd-trace-go.v1 v1.67.1
google.golang.org/protobuf v1.35.1
gopkg.in/DataDog/dd-trace-go.v1 v1.69.1
gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d // indirect
gopkg.in/ldap.v2 v2.5.1
sigs.k8s.io/yaml v1.4.0
)

require (
github.com/DataDog/datadog-go/v5 v5.5.0
github.com/Shopify/toxiproxy/v2 v2.9.0
github.com/aws/aws-sdk-go-v2 v1.30.4
github.com/aws/aws-sdk-go-v2/config v1.27.31
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.15
github.com/aws/aws-sdk-go-v2/service/s3 v1.60.1
github.com/aws/smithy-go v1.20.4
github.com/Shopify/toxiproxy/v2 v2.11.0
github.com/aws/aws-sdk-go-v2 v1.32.4
github.com/aws/aws-sdk-go-v2/config v1.28.3
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.37
github.com/aws/aws-sdk-go-v2/service/s3 v1.66.3
github.com/aws/smithy-go v1.22.0
github.com/bndr/gotabulate v1.1.2
github.com/gammazero/deque v0.2.1
github.com/google/safehtml v0.1.0
Expand All @@ -107,48 +107,58 @@ require (
github.com/spf13/jwalterweatherman v1.1.0
github.com/xlab/treeprint v1.2.0
go.uber.org/goleak v1.3.0
golang.org/x/exp v0.0.0-20240909161429-701f63a606c0
golang.org/x/sync v0.8.0
golang.org/x/exp v0.0.0-20241108190413-2d47ceb2692f
golang.org/x/sync v0.9.0
gonum.org/v1/gonum v0.14.0
modernc.org/sqlite v1.33.1
)

require (
cloud.google.com/go v0.115.1 // indirect
cloud.google.com/go/auth v0.9.4 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect
cloud.google.com/go/compute/metadata v0.5.1 // indirect
cloud.google.com/go/iam v1.2.1 // indirect
github.com/DataDog/appsec-internal-go v1.7.0 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.57.0 // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.57.0 // indirect
github.com/DataDog/go-libddwaf/v3 v3.4.0 // indirect
github.com/DataDog/go-sqllexer v0.0.14 // indirect
cel.dev/expr v0.18.0 // indirect
cloud.google.com/go v0.116.0 // indirect
cloud.google.com/go/auth v0.10.2 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.5 // indirect
cloud.google.com/go/compute/metadata v0.5.2 // indirect
cloud.google.com/go/iam v1.2.2 // indirect
cloud.google.com/go/monitoring v1.21.2 // indirect
github.com/DataDog/appsec-internal-go v1.9.0 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.59.0 // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.59.0 // indirect
github.com/DataDog/go-libddwaf/v3 v3.5.1 // indirect
github.com/DataDog/go-sqllexer v0.0.16 // indirect
github.com/DataDog/go-tuf v1.1.0-0.5.2 // indirect
github.com/DataDog/sketches-go v1.4.6 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.25.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.49.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.49.0 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.4 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.30 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.12 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.16 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.6 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.44 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.19 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.16 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.18 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.18 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.16 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.22.5 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.30.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.0 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.4.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.5 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.32.4 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
github.com/cncf/xds/go v0.0.0-20240905190251-b4127c9b8d78 // indirect
github.com/coreos/go-semver v0.3.1 // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.4 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.5 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/ebitengine/purego v0.7.1 // indirect
github.com/fatih/color v1.17.0 // indirect
github.com/eapache/queue/v2 v2.0.0-20230407133247-75960ed334e4 // indirect
github.com/ebitengine/purego v0.8.1 // indirect
github.com/envoyproxy/go-control-plane v0.13.1 // indirect
github.com/envoyproxy/protoc-gen-validate v1.1.0 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
Expand All @@ -162,9 +172,12 @@ require (
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-rootcerts v1.0.2 // indirect
github.com/hashicorp/go-secure-stdlib/parseutil v0.1.8 // indirect
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
github.com/hashicorp/go-sockaddr v1.0.7 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/hcl v1.0.1-vault-5 // indirect
github.com/hashicorp/hcl v1.0.1-vault-7 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-ieproxy v0.0.12 // indirect
Expand All @@ -182,8 +195,9 @@ require (
github.com/prometheus/procfs v0.15.1 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/rogpeppe/go-internal v1.12.0 // indirect
github.com/rogpeppe/go-internal v1.13.1 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/sagikazarmark/locafero v0.6.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/secure-systems-lab/go-securesystemslib v0.8.0 // indirect
Expand All @@ -193,22 +207,26 @@ require (
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.55.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.55.0 // indirect
go.opentelemetry.io/otel v1.30.0 // indirect
go.opentelemetry.io/otel/metric v1.30.0 // indirect
go.opentelemetry.io/otel/trace v1.30.0 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.32.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.57.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.57.0 // indirect
go.opentelemetry.io/otel v1.32.0 // indirect
go.opentelemetry.io/otel/metric v1.32.0 // indirect
go.opentelemetry.io/otel/sdk v1.32.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.32.0 // indirect
go.opentelemetry.io/otel/trace v1.32.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20241104194629-dd2ea8efbc28 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241104194629-dd2ea8efbc28 // indirect
google.golang.org/grpc/stats/opentelemetry v0.0.0-20241028142157-ada6787961b3 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/gc/v3 v3.0.0-20240801135723-a856999a2e4a // indirect
modernc.org/libc v1.60.1 // indirect
modernc.org/gc/v3 v3.0.0-20241004144649-1aea3fae8852 // indirect
modernc.org/libc v1.61.0 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
modernc.org/strutil v1.2.0 // indirect
Expand Down
Loading
Loading