Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

endpointRef data race #7910

Open
GiedriusS opened this issue Nov 15, 2024 · 0 comments
Open

endpointRef data race #7910

GiedriusS opened this issue Nov 15, 2024 · 0 comments
Labels
bug component: query

Comments

@GiedriusS
Copy link
Member

From one of the tests:

2024-11-15T08:41:57.5441863Z 08:41:57 querier-1: ==================
2024-11-15T08:41:57.5442722Z 08:41:57 querier-1: WARNING: DATA RACE
2024-11-15T08:41:57.5443668Z 08:41:57 querier-1: Write at 0x00c00091cab0 by goroutine 197:
2024-11-15T08:41:57.5445694Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*endpointRef).updateStatus()
2024-11-15T08:41:57.5447200Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/endpointset.go:610 +0x430
2024-11-15T08:41:57.5448714Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*endpointRef).update()
2024-11-15T08:41:57.5450211Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/endpointset.go:598 +0x364
2024-11-15T08:41:57.5451755Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*EndpointSet).updateEndpoint()
2024-11-15T08:41:57.5453336Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/endpointset.go:387 +0x544
2024-11-15T08:41:57.5454788Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*EndpointSet).Update.func1()
2024-11-15T08:41:57.5456353Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/endpointset.go:300 +0x176
2024-11-15T08:41:57.5457923Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*EndpointSet).Update.gowrap2()
2024-11-15T08:41:57.5459565Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/endpointset.go:305 +0x41
2024-11-15T08:41:57.5460979Z 08:41:57 querier-1: Previous read at 0x00c00091cab0 by goroutine 194:
2024-11-15T08:41:57.5462322Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*endpointRef).LabelSets()
2024-11-15T08:41:57.5463848Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/endpointset.go:704 +0xe4
2024-11-15T08:41:57.5465301Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/store.newAsyncRespSet()
2024-11-15T08:41:57.5466774Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/store/proxy_merge.go:515 +0xd89
2024-11-15T08:41:57.5468235Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series()
2024-11-15T08:41:57.5469688Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/store/proxy.go:302 +0x1918
2024-11-15T08:41:57.5471525Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/store.(*instrumentedStoreServer).Series()
2024-11-15T08:41:57.5473147Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/store/telemetry.go:169 +0xeb
2024-11-15T08:41:57.5474654Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/store.(*limitedStoreServer).Series()
2024-11-15T08:41:57.5476572Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/store/limiter.go:145 +0x301
2024-11-15T08:41:57.5478072Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*querier).selectFn()
2024-11-15T08:41:57.5479495Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/querier.go:347 +0x598
2024-11-15T08:41:57.5480804Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*querier).Select.func1()
2024-11-15T08:41:57.5482233Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/querier.go:293 +0x36b
2024-11-15T08:41:57.5483399Z 08:41:57 querier-1: Goroutine 197 (running) created at:
2024-11-15T08:41:57.5484530Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*EndpointSet).Update()
2024-11-15T08:41:57.5486038Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/endpointset.go:296 +0x924
2024-11-15T08:41:57.5487291Z 08:41:57 querier-1: main.prepareEndpointSet.func2.1()
2024-11-15T08:41:57.5488631Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/cmd/thanos/query.go:934 +0x46
2024-11-15T08:41:57.5489995Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/runutil.Repeat()
2024-11-15T08:41:57.5491402Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:91 +0x109
2024-11-15T08:41:57.5492635Z 08:41:57 querier-1: main.prepareEndpointSet.func2()
2024-11-15T08:41:57.5494146Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/cmd/thanos/query.go:933 +0xbb
2024-11-15T08:41:57.5495421Z 08:41:57 querier-1: github.com/oklog/run.(*Group).Run.func1()
2024-11-15T08:41:57.5496620Z 08:41:57 querier-1: /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x39
2024-11-15T08:41:57.5497765Z 08:41:57 querier-1: github.com/oklog/run.(*Group).Run.gowrap1()
2024-11-15T08:41:57.5499097Z 08:41:57 querier-1: /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x4f
2024-11-15T08:41:57.5500167Z 08:41:57 querier-1: Goroutine 194 (running) created at:
2024-11-15T08:41:57.5501269Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/query.(*querier).Select()
2024-11-15T08:41:57.5502604Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/query/querier.go:277 +0x8fa
2024-11-15T08:41:57.5504105Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql.(*Engine).populateSeries.func1()
2024-11-15T08:41:57.5506050Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/engine.go:964 +0x533
2024-11-15T08:41:57.5507828Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql/parser.inspector.Visit()
2024-11-15T08:41:57.5509745Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/parser/ast.go:348 +0x62
2024-11-15T08:41:57.5513077Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql/parser.Walk()
2024-11-15T08:41:57.5514958Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/parser/ast.go:318 +0xa1
2024-11-15T08:41:57.5516694Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql/parser.Walk()
2024-11-15T08:41:57.5518490Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/parser/ast.go:324 +0x21e
2024-11-15T08:41:57.5520165Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql/parser.Walk()
2024-11-15T08:41:57.5521988Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/parser/ast.go:324 +0x21e
2024-11-15T08:41:57.5523750Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql/parser.Inspect()
2024-11-15T08:41:57.5525603Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/parser/ast.go:359 +0x24c
2024-11-15T08:41:57.5527420Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql.(*Engine).populateSeries()
2024-11-15T08:41:57.5529537Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/engine.go:947 +0x12
2024-11-15T08:41:57.5531295Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt()
2024-11-15T08:41:57.5549334Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/engine.go:717 +0x2be
2024-11-15T08:41:57.5551645Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql.(*Engine).exec()
2024-11-15T08:41:57.5553908Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/engine.go:678 +0x59c
2024-11-15T08:41:57.5555868Z 08:41:57 querier-1: github.com/prometheus/prometheus/promql.(*query).Exec()
2024-11-15T08:41:57.5558096Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/prometheus@v0.55.1-0.20241102120812-a6fd22b9d2c8/promql/engine.go:245 +0x213
2024-11-15T08:41:57.5560503Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/api/query.(*QueryAPI).query.func15()
2024-11-15T08:41:57.5561999Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/api/query/v1.go:714 +0x63
2024-11-15T08:41:57.5563290Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/tracing.DoInSpan()
2024-11-15T08:41:57.5564616Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/tracing/tracing.go:95 +0x153
2024-11-15T08:41:57.5566007Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/api/query.(*QueryAPI).query()
2024-11-15T08:41:57.5567604Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/api/query/v1.go:713 +0x1466
2024-11-15T08:41:57.5569011Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/api/query.(*QueryAPI).query-fm()
2024-11-15T08:41:57.5570060Z 08:41:57 querier-1: <autogenerated>:1 +0x45
2024-11-15T08:41:57.5571331Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/api/query.(*QueryAPI).Register.GetInstr.func1.1()
2024-11-15T08:41:57.5572960Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/api/api.go:223 +0x8a
2024-11-15T08:41:57.5574050Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5575060Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x47
2024-11-15T08:41:57.5576543Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/logging.(*HTTPServerMiddleware).HTTPMiddleware.func1()
2024-11-15T08:41:57.5578146Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/logging/http.go:88 +0x4b7
2024-11-15T08:41:57.5579269Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5580265Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x47
2024-11-15T08:41:57.5581459Z 08:41:57 querier-1: github.com/klauspost/compress/gzhttp.NewWrapper.func1.1()
2024-11-15T08:41:57.5582969Z 08:41:57 querier-1: /go/pkg/mod/github.com/klauspost/compress@v1.17.9/gzhttp/compress.go:497 +0x7cf
2024-11-15T08:41:57.5584219Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5585214Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x47
2024-11-15T08:41:57.5586641Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.func1()
2024-11-15T08:41:57.5588427Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/extprom/http/instrument_server.go:75 +0x162
2024-11-15T08:41:57.5589728Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5590708Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x47
2024-11-15T08:41:57.5592944Z 08:41:57 querier-1: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1()
2024-11-15T08:41:57.5594311Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/client_golang@v1.20.5/prometheus/promhttp/instrument_server.go:296 +0xee
2024-11-15T08:41:57.5595180Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5595789Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x47
2024-11-15T08:41:57.5596653Z 08:41:57 querier-1: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1()
2024-11-15T08:41:57.5598017Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/client_golang@v1.20.5/prometheus/promhttp/instrument_server.go:147 +0xe5
2024-11-15T08:41:57.5598871Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5599453Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x47
2024-11-15T08:41:57.5600442Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.instrumentHandlerInFlight.func2()
2024-11-15T08:41:57.5601599Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/extprom/http/instrument_server.go:164 +0x1cb
2024-11-15T08:41:57.5602351Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5602930Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x47
2024-11-15T08:41:57.5603811Z 08:41:57 querier-1: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1()
2024-11-15T08:41:57.5604978Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/client_golang@v1.20.5/prometheus/promhttp/instrument_server.go:243 +0xee
2024-11-15T08:41:57.5605805Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5606434Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x1f3
2024-11-15T08:41:57.5607350Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/extprom/http.(*tenantInstrumentationMiddleware).NewHandler.func1()
2024-11-15T08:41:57.5608555Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/extprom/http/instrument_tenant_server.go:43 +0x1d3
2024-11-15T08:41:57.5609314Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5609882Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x47
2024-11-15T08:41:57.5610583Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/tracing.HTTPMiddleware.func1()
2024-11-15T08:41:57.5611453Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/tracing/http.go:67 +0xf61
2024-11-15T08:41:57.5612086Z 08:41:57 querier-1: net/http.HandlerFunc.ServeHTTP()
2024-11-15T08:41:57.5612652Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2220 +0x47
2024-11-15T08:41:57.5613410Z 08:41:57 querier-1: github.com/thanos-io/thanos/pkg/server/http/middleware.RequestID.func1()
2024-11-15T08:41:57.5614375Z 08:41:57 querier-1: /go/src/github.com/thanos-io/thanos/pkg/server/http/middleware/request_id.go:40 +0x3b8
2024-11-15T08:41:57.5615317Z 08:41:57 querier-1: github.com/prometheus/common/route.(*Router).handle.func1()
2024-11-15T08:41:57.5616809Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/common@v0.60.0/route/route.go:83 +0x2f0
2024-11-15T08:41:57.5618078Z 08:41:57 querier-1: github.com/julienschmidt/httprouter.(*Router).ServeHTTP()
2024-11-15T08:41:57.5619050Z 08:41:57 querier-1: /go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0xf0b
2024-11-15T08:41:57.5619971Z 08:41:57 querier-1: github.com/prometheus/common/route.(*Router).ServeHTTP()
2024-11-15T08:41:57.5620970Z 08:41:57 querier-1: /go/pkg/mod/github.com/prometheus/common@v0.60.0/route/route.go:126 +0x53
2024-11-15T08:41:57.5621734Z 08:41:57 querier-1: net/http.(*ServeMux).ServeHTTP()
2024-11-15T08:41:57.5622426Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2747 +0x255
2024-11-15T08:41:57.5623123Z 08:41:57 querier-1: net/http.serverHandler.ServeHTTP()
2024-11-15T08:41:57.5623820Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:3210 +0x2a1
2024-11-15T08:41:57.5624444Z 08:41:57 querier-1: net/http.(*conn).serve()
2024-11-15T08:41:57.5625009Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:2092 +0x12a4
2024-11-15T08:41:57.5625677Z 08:41:57 querier-1: net/http.(*Server).Serve.gowrap3()
2024-11-15T08:41:57.5626356Z 08:41:57 querier-1: /usr/local/go/src/net/http/server.go:3360 +0x4f
2024-11-15T08:41:57.5626952Z 08:41:57 querier-1: ==================
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug component: query
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@GiedriusS