Regarding Bulk insert

[navneetrajput06]

Hi Team,
Could you please help me to answer below queries.

1. When we start bulk insertion job without indexing enabled compaction for those files will start automatically or we beed to do compaction manually?.
2. I am using 100 worker thread and 8 shards as we are running with 8 data nodes. Will 100 files start copying together or it depends on number of shared?
3. if we enable collection loading at the time of collection creation will it start loading growing segments in the collection or it will be done after bulk insertions completed.

Thanks,
Navneet

[yhmo]

1. No need to do manual compaction, just let the Milvus compact by itself. Note: there are some bugs for bulkinsert compaction in the earlier versions of 2.4, you'd better use the latest 2.4 to test.

2. What do you mean "100 worker threads"?
   You can input multiple files in one call of bulkinsert so that the files can be processed together in one job.
   Like this: utility.do_bulk_insert(files=[file1, file2, ... fileN])]). The data of N files will be hashed to 8 shards and generate at least 8 segments.

If you input one file for each call, like this:

for file in files:
  utility.do_bulk_insert(files=[file])

The data of each file will be hashed to 8 shards, and generate 8 segments for each file.
So, multiple files for each call is recommended. Note: the max file number for each call is 1024

3. There are no "growing segment" for bulkinsert. It always generates sealed-segment. Once a sealed-segment is generated by bulkinsert job, the query node will load it after it is indexed.

[navneetrajput06]

@yhmo currently my bulk insertion is running with 8 shards and having 8 data nodes in cluster. Should increase datanodes to speedup as i cant change the number of sharda during insertion in progress.

[yhmo]

The data files are parsed by different data nodes parallelly, so, increasing data node number can reduce the time to parse files.
In these data nodes, there are 8 data nodes to persist segments for the 8 shards. Increasing data node number doesn't help the performance of persist segments.

[navneetrajput06]

@yhmo in that case increasing shards will help ?

[yhmo]

Two major works of bulkinsert job:
A. parsing data files
B. persist data into segments

If A is faster, B is the bottleneck. Else if B is faster, then A is the bottleneck. There might be a "best ratio of data nodes and shards", but I don't know how to estimate the ratio.
Anyway, it is not a good idea to increase shards to a large number. Each shard is like a virtual data channel between pulsar and querynode/datanode. If the collection has N partitions and M shards, there will be N * M virtual channels for this collection. Each channel will waste a bit resources to maintain. With a large number of shards, the milvus server might be overburdened.