milvus输出failed to load some segments

[walker1024]

请教下，在日志文件中看到大量加载segments失败的日志，应该是加载失败一直在尝试加载，关键日志：

{"log":"I20241201 16:18:17.518265 108 Utils.cpp:822] [SERVER][LoadFieldDatasFromRemote][milvus] failed to load data from remote: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.518347519Z"}

是对象存储服务返回的数据有问题吗？

集群信息

• Milvus version:2.4.11
• Deployment mode(standalone or cluster):cluster
• MQ type(rocksmq, pulsar or kafka): kafka
• SDK version(e.g. pymilvus v2.0.0rc2): jdk8
• OS(Ubuntu or CentOS): CentOS
• CPU/Memory:
• GPU: -
• Others:

详细日志如下：

{"log":"[2024/12/01 16:18:17.485 +00:00] [INFO] [segments/segment.go:979] ["start loading field data for field"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [partitionID=454243019795063564] [segmentID=454243019795289930] [fieldID=100] [rowCount=3]\n","stream":"stdout","time":"2024-12-01T16:18:17.485415863Z"}
{"log":"[2024/12/01 16:18:17.485 +00:00] [INFO] [segments/segment.go:1013] ["submitted loadFieldData task to load pool"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [partitionID=454243019795063564] [segmentID=454243019795289930] [fieldID=100] [rowCount=3]\n","stream":"stdout","time":"2024-12-01T16:18:17.485606982Z"}
{"log":"I20241201 16:18:17.485694 105 SegmentSealedImpl.cpp:254] [SERVER][LoadFieldData][milvus] segment 454243019795289930 loads field 100 with num_rows 3\n","stream":"stdout","time":"2024-12-01T16:18:17.485769324Z"}
{"log":"I20241201 16:18:17.485800 105 SegmentSealedImpl.cpp:267] [SERVER][LoadFieldData][milvus] segment 454243019795289930 submits load field 100 task to thread pool\n","stream":"stdout","time":"2024-12-01T16:18:17.485854399Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.517898828Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.517912388Z"}
{"log":"I20241201 16:18:17.518045 85 Utils.cpp:822] [SERVER][LoadFieldDatasFromRemote][milvus] failed to load data from remote: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.518115398Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.518220596Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.518225102Z"}
{"log":"[2024/12/01 16:18:17.518 +00:00] [WARN] [segments/segment.go:1028] ["CStatus returns err"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [partitionID=454243019795063564] [segmentID=454243019795289930] [fieldID=101] [error=" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n"] [extra="LoadFieldData failed"]\n","stream":"stdout","time":"2024-12-01T16:18:17.518263125Z"}
{"log":"I20241201 16:18:17.518265 108 Utils.cpp:822] [SERVER][LoadFieldDatasFromRemote][milvus] failed to load data from remote: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.518347519Z"}
{"log":"[2024/12/01 16:18:17.518 +00:00] [WARN] [segments/segment.go:1028] ["CStatus returns err"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [partitionID=454243019795063564] [segmentID=454243019795289930] [fieldID=100] [error=" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n"] [extra="LoadFieldData failed"]\n","stream":"stdout","time":"2024-12-01T16:18:17.518481754Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.521481944Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.521487905Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.527888827Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.527895815Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.528933281Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.52893886Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.530068849Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.530074284Z"}
{"log":"I20241201 16:18:17.530094 86 Utils.cpp:822] [SERVER][LoadFieldDatasFromRemote][milvus] failed to load data from remote: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.530181711Z"}
{"log":"[2024/12/01 16:18:17.530 +00:00] [WARN] [segments/segment.go:1028] ["CStatus returns err"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [partitionID=454243019795063564] [segmentID=454243019795289930] [fieldID=0] [error=" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n"] [extra="LoadFieldData failed"]\n","stream":"stdout","time":"2024-12-01T16:18:17.530274493Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.533286927Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.533301641Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.533784794Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.533791221Z"}
{"log":"I20241201 16:18:17.533805 87 Utils.cpp:822] [SERVER][LoadFieldDatasFromRemote][milvus] failed to load data from remote: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.533853333Z"}
{"log":"[2024/12/01 16:18:17.533 +00:00] [WARN] [segments/segment.go:1028] ["CStatus returns err"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [partitionID=454243019795063564] [segmentID=454243019795289930] [fieldID=1] [error=" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n"] [extra="LoadFieldData failed"]\n","stream":"stdout","time":"2024-12-01T16:18:17.533942807Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.535658107Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.535665001Z"}
{"log":"I20241201 16:18:17.535683 88 Utils.cpp:822] [SERVER][LoadFieldDatasFromRemote][milvus] failed to load data from remote: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.535742237Z"}
{"log":"[2024/12/01 16:18:17.535 +00:00] [WARN] [segments/segment.go:1028] ["CStatus returns err"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [partitionID=454243019795063564] [segmentID=454243019795289930] [fieldID=102] [error=" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n"] [extra="LoadFieldData failed"]\n","stream":"stdout","time":"2024-12-01T16:18:17.535886686Z"}
{"log":" =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n","stream":"stdout","time":"2024-12-01T16:18:17.541269574Z"}
{"log":"\n","stream":"stdout","time":"2024-12-01T16:18:17.541283467Z"}
{"log":"[2024/12/01 16:18:17.547 +00:00] [WARN] [segments/segment_loader.go:702] ["load segment failed when load data into memory"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [segmentType=Sealed] [requestSegments="[454243019795289930]"] [preparedSegments="[454243019795289930]"] [partitionID=454243019795063564] [segmentID=454243019795289930] [segmentType=L1] [error="At LoadSegment: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n"] [errorVerbose="At LoadSegment: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:715\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:86\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) At LoadSegment\nWraps: (3) =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]\n","stream":"stdout","time":"2024-12-01T16:18:17.547416477Z"}
{"log":"[2024/12/01 16:18:17.547 +00:00] [INFO] [segments/segment_loader.go:704] ["load segment done"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [segmentType=Sealed] [requestSegments="[454243019795289930]"] [preparedSegments="[454243019795289930]"] [partitionID=454243019795063564] [segmentID=454243019795289930] [segmentType=L1]\n","stream":"stdout","time":"2024-12-01T16:18:17.54743729Z"}
{"log":"[2024/12/01 16:18:17.547 +00:00] [ERROR] [funcutil/parallel.go:88] [loadSegmentFunc] [error="At LoadSegment: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n"] [errorVerbose="At LoadSegment: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:715\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:86\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) At LoadSegment\nWraps: (3) =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:88"]\n","stream":"stdout","time":"2024-12-01T16:18:17.547470657Z"}
{"log":"[2024/12/01 16:18:17.547 +00:00] [WARN] [segments/segment_loader.go:741] ["failed to load some segments"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [segmentType=Sealed] [requestSegments="[454243019795289930]"] [preparedSegments="[454243019795289930]"] [error="At LoadSegment: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n"] [errorVerbose="At LoadSegment: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:715\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:86\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) At LoadSegment\nWraps: (3) =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]\n","stream":"stdout","time":"2024-12-01T16:18:17.547533043Z"}
{"log":"[2024/12/01 16:18:17.547 +00:00] [WARN] [segments/segment_loader.go:650] ["release new segment created due to load failure"] [traceID=0453f849a04039a4c651a610aa9c298f] [collectionID=454243019795063563] [segmentType=Sealed] [requestSegments="[454243019795289930]"] [preparedSegments="[454243019795289930]"] [segmentID=454243019795289930] [error="At LoadSegment: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n"] [errorVerbose="At LoadSegment: =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:715\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:86\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) At LoadSegment\nWraps: (3) =\u003e not supported at /workspace/source/internal/core/src/storage/DataCodec.cpp:98\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]\n","stream":"stdout","time":"2024-12-01T16:18:17.54756822Z"}
{"log":"[2024/12/01 16:18:17.547 +00:00] [INFO] [segments/segment.go:1544] ["delete segment from memory"] [collectionID=454243019795063563] [partitionID=454243019795063564] [segmentID=454243019795289930] [segmentType=Sealed] [insertCount=3]\n","stream":"stdout","time":"2024-12-01T16:18:17.547716645Z"}
{"log":"[2024/12/01 16:18:17.553 +00:00] [INFO] [segments/collection.go:119] ["release collection due to ref count to 0"] [collectionID=454243019795063563]\n","stream":"stdout","time":"2024-12-01T16:18:17.553419539Z"}

[xiaofan-luan]

@tedxu
please help on checking it

[sunby]

Milvus first checks a magic number to validate the binlog file, but it seems that the magic number for the binlog file is incorrect. Could you please provide the binlog file for segment 454243019795289930 so I can assist you in troubleshooting more efficiently?

[walker1024]

@walker1024 I guess you probablly need to check your storage.

It's very likely the storage itself has some issue.

object storage? I am using the company's object storage system instead of minio directly. From the log phenomenon, the data can be queried normally, but the log will always prompt "failed to load data from remote"

[xiaofan-luan]

you probably need to check with the storage team. The data returned from object storage seems to be wrong and what we see is that many of the the object storage is not fully compatible with S3

[walker1024]

you probably need to check with the storage team. The data returned from object storage seems to be wrong and what we see is that many of the the object storage is not fully compatible with S3

ok，I will discuss this issue with the company’s engineers.
Why after the first write, the data can be loaded normally in attu. After clicking release, the reload will prompt a loading failure. What is the data flow here?

always showing loading.

[walker1024]

Can you help confirm that the following configuration is correct? Are there any other configurations?

externalS3:
enabled: true
host: "a.com"
port: "80"
accessKey: "bbbb"
secretKey: "ccc"
useSSL: false
bucketName: "ad-milvus"
rootPath: ""
useIAM: false
cloudProvider: "aws"
iamEndpoint: ""
region: "BJ"
useVirtualHost: false

[xiaofan-luan]

same problem here. after flush and index build, the data need to be loaded from S3.

insert and flush won't check any data from S3