v0.4.0

Added support:

• S3 compatible storage
• Different filters for each search way
• Highlight/Min should match/keywords as filters for full text search, Japanese/Korean language support for full text search.
• Int8/F16/BF16, single document multi-vector for dense embedding data.
• IVF dense vector index.
• IN operator,adding/deleting columns, parquet format importing and exporting

v0.4.0-dev4

Release v0.4.0-dev4 created from 1e0ac82 at 2024-10-14 15:26:28+00:00

v0.4.0-dev3

Release v0.4.0-dev3 created from e554c06 at 2024-10-09 12:24:17+00:00

slow-test

Release slow-test created from 3f2a0f5 at 2024-10-09 06:40:35+00:00

v0.4.0-dev2

Release v0.4.0-dev2 created from 14962eb at 2024-09-23 02:33:08+00:00

v0.4.0-dev1

Release v0.4.0-dev1 created from 81d7c7f at 2024-09-12 01:21:10+00:00

v0.3.0

What's Changed

• Support simd functions dispatch at runtime by @yangzq50 in #1473
• Add Qdrant benchmark on sparse embedding of SPLADE by @Ami11111 in #1478
• Update simd support for aarch64 by @yangzq50 in #1481
• Fix document error by @JinHai-CN in #1483
• Update performance image by @JinHai-CN in #1482
• Update performance information by @JinHai-CN in #1487
• Refactor document by @JinHai-CN in #1488
• VFS first phrase by @yuzhichang in #1485
• Refactor examples by @JinHai-CN in #1489
• Memidx refactor by @small-turtle-1 in #1484
• Add Milvus benchmark on sparse embedding of SPLADE by @Ami11111 in #1491
• Add: const column vector. by @small-turtle-1 in #1486
• Add Milvus benchmark on dataset sift by @Ami11111 in #1494
• Add symbolic links to large files in vfs by @Ma-cat in #1495
• Initial updates to tensor_search.py by @writinwaters in #1499
• Add asking user beforing downloading dataset and add support for precision calculation for Qdrant by @Ami11111 in #1498
• Support float16 type and bfloat16 type for column data type by @yangzq50 in #1503
• Add Qdrant sparse vector support for benchmark framework by @Ami11111 in #1504
• Initial updates to filter_data.py by @writinwaters in #1505
• Initial updates to hybrid_search.py by @writinwaters in #1507
• Add infinity sparse vector support for benchmark framework by @Ami11111 in #1506
• Fix: support tensor insert for local infinity in python sdk by @yangzq50 in #1502
• Support basic import/export for parquet format. by @Ognimalf in #1446
• Fix: python api error detect. by @small-turtle-1 in #1508
• Update version info and document by @JinHai-CN in #1511
• Updare benchmark framework README by @Ami11111 in #1513
• Update the identifier to lower case when the request is from API but not SQL by @JinHai-CN in #1514
• Add test case for upper and lower cases when creating a database, table or index from python api. by @Ami11111 in #1517
• Updated export_data API reference by @writinwaters in #1518
• Fix bug: upper case column name, type and parameter are not cast to lower case. by @JinHai-CN in #1519
• Add SNAPPY as the third_party by @JinHai-CN in #1520
• Added zstd, bzip2 and brotli to builder image by @yuzhichang in #1523
• Parquet import/export by @small-turtle-1 in #1521
• Refactored Python API reference. by @writinwaters in #1522
• Add insert row limit and change chunk size to 64K again by @JinHai-CN in #1526
• Supports cleaning up vfs by @Ma-cat in #1529
• Simplified python code example by @writinwaters in #1531
• Fix mem leak by @small-turtle-1 in #1532
• Support int8 embedding and uint8 embedding by @yangzq50 in #1527
• Add test cases for tensor in test_knn.py by @Ami11111 in #1528
• Fix compact by @small-turtle-1 in #1537
• Updated setuptools by @yuzhichang in #1524
• Fix corruption if no outputs from analyzer by @yingfeng in #1539
• Add knn test cases for tensor and sparse vector by @Ami11111 in #1536
• Editorial updates to python API by @writinwaters in #1538
• Fix bool column default value for python sdk by @yangzq50 in #1543
• Fix: delta ckp commit ts. by @small-turtle-1 in #1548
• Limit the embedding dimension to 65536 by @JinHai-CN in #1547
• Standard analyzer could output unicode tokens by @yingfeng in #1541
• Fix creating fulltext index oom in debug mode by @Ma-cat in #1550
• Fix: cleanup block when import/compact fail. by @small-turtle-1 in #1551
• Refactor conflictcheck by @small-turtle-1 in #1552
• Refactor WAL Manager by @JinHai-CN in #1553
• Support VFS for fulltext index. by @Ognimalf in #1546
• Merge programs under cases folder and internal folder together by @Ami11111 in #1540
• Editorial updates to Python API reference by @writinwaters in #1557
• Fix: compact with delete bug. by @small-turtle-1 in #1558
• Refactor the txn commit and rollback by @JinHai-CN in #1555
• Editorial updates to the API by @writinwaters in #1560
• Fix bug by @small-turtle-1 in #1559
• Updated return values by @writinwaters in #1564
• Minor editorial updates. by @writinwaters in #1566
• Fix: get cleanup ts by @small-turtle-1 in #1567
• Add http api test cases support for the test framework by @Ami11111 in #1562
• add diskann create_index api by @YReddice in #1556
• Obtains the path to the environment variable in the documentation by @YReddice in #1568
• Add bvecs file import support and fix its bug by @Ami11111 in #1572
• Updated create_index by @writinwaters in #1571
• Fix memory index multiple bugs when using vfs by @Ma-cat in #1561
• Enable wal and ckp unit tests for VFS by @Ognimalf in #1570
• Support float16 and bfloat16 embedding data type by @yangzq50 in #1579
• Fix http api bug that create and drop table with invalid conflict type will not raise exception by @Ami11111 in #1576
• Enable http api delete with empty filter by @Ami11111 in #1577
• Show SIMD support in welcome info by @yangzq50 in #1585
• Update version info of 0.3.0.dev3 by @JinHai-CN in #1584
• Updated example codes by @writinwaters in #1581
• Add debug/admin mode by @JinHai-CN in #1573
• Memidx quota1 by @small-turtle-1 in #1580
• Remove redundant test cases by @JinHai-CN in #1586
• Updated index-related apis by @writinwaters in #1590
• Fix bvecs import use int8 rather than uint8 by @Ami11111 in #1574
• Fix bug and add python test for float16 and bfloat16 embedding type by @yangzq50 in #1588
• Minor editorial udpates. by @writinwaters in #1594
• Update http_adapter to create table with non-empty default field by @Ami11111 in #1593
• Update python SDK by @JinHai-CN in #1592
• delete and update api updates by @writinwaters in #1597
• Fix http adapter and http api bug and add fvecs file export support for http api by @Ami11111 in #1595
• User can set background operation interval by @JinHai-CN in #1596
• minor by @writinwaters in #1600
• Fix error reporting in CMake by @Smertig in #1599
• Fix http adapter bug and update some test cases by @Ami11111 in #1598
• Add list_indexes for embedded python SDK by @JinHai-CN in #1602...

v0.3.0-dev8

Release v0.3.0-dev8 created from 82b5e57 at 2024-09-02 10:01:19+00:00

infinity-v0.2.1

What's Changed

• Fix build script for nanobind by @yangzq50 in #1390
• Fix a docusaurus display issue by @writinwaters in #1393
• Fix: use static link for nanobind by @yangzq50 in #1392
• Remove redundant code by @JinHai-CN in #1396
• Rerank hnsw lvq by @small-turtle-1 in #1395
• Fix convert between embeddings by @Ognimalf in #1398
• Add export offset and limit by @JinHai-CN in #1399
• Opt bmp by @small-turtle-1 in #1400
• Add insert script for mldr benchmark by @yangzq50 in #1401
• Optimize memory by @Ma-cat in #1407
• Fix: data missing when flush block version file by @yangzq50 in #1406
• Install multiple python into builder image. by @yuzhichang in #1404
• Fix import error by @JinHai-CN in #1403
• Removed resource from rpm, deb and image by @yuzhichang in #1411
• Exact phrase optimization by @yuzhichang in #1412
• Export data into multiple files. by @JinHai-CN in #1409
• Accelerate release CI process by @JinHai-CN in #1413
• Create fulltext index optimization by @Ma-cat in #1410
• Update version information before release 0.2.1 by @JinHai-CN in #1414
• Update version by @JinHai-CN in #1420
• Fix path by @small-turtle-1 in #1421
• Fix: sort sparse expr in import and match statement by @small-turtle-1 in #1419
• python modules shall dynamicly link to libstdc++ by @yuzhichang in #1423
• Update version info to 0.2.1.dev3 by @JinHai-CN in #1424
• Add bpreorder by @small-turtle-1 in #1426
• Update MLDR benchmark script by @yangzq50 in #1425
• Fixed a docusaurus display issue by @writinwaters in #1430
• Update fusion weighted sum support and MLDR weighted sum benchmark by @yangzq50 in #1428
• build from source using clang-18 by @Ma-cat in #1429
• Fix parser by @small-turtle-1 in #1431
• Parallel execution of python test in ci by @Ma-cat in #1422
• Link to llvm libc++ instead of gcc libstdc++ by @JinHai-CN in #1435
• Clang18 by @yuzhichang in #1433
• Remove unused dockerfile and update benchmark record by @JinHai-CN in #1437
• Add script for generating colbert embeddings for MLDR benchmark by @yangzq50 in #1434
• Fix optimize index by @small-turtle-1 in #1436
• Name builder container with a random number to allow concurrent workflow by @yuzhichang in #1441
• Modify ci display by @Ma-cat in #1440
• support build infinity in ubuntu 24.04 by @Ma-cat in #1445
• Update version info by @JinHai-CN in #1439
• Add script for insert colbert embeddings for MLDR benchmark by @yangzq50 in #1442
• Optimize buffer mgr by @small-turtle-1 in #1444
• Limit cpu of the builder container and ninja by @yuzhichang in #1447
• Export CMAKE_BUILD_PARALLEL_LEVEL by @yuzhichang in #1450
• Fix txn manager bug by @small-turtle-1 in #1451
• Update MLDR benchmark script by @yangzq50 in #1449
• Add 'show logs' and 'show delta logs' by @JinHai-CN in #1453
• Add 'show catalogs' by @JinHai-CN in #1455
• fix create fulltext index by @Ma-cat in #1454
• Keep order of match expression by @yuzhichang in #1452
• Refactor meta data by @JinHai-CN in #1459
• Refactor entry_list by @JinHai-CN in #1443
• Fix formatting errors by @zjbpaul1317 in #1460
• Fix executor bug by @small-turtle-1 in #1458
• Optimize bp. by @small-turtle-1 in #1461
• Update mblas by @JinHai-CN in #1457
• Fix document errors by @zjbpaul1317 in #1464
• Update version info by @JinHai-CN in #1463
• Adapt slow_test for new runner by @yuzhichang in #1465
• Refactor meta_map by @JinHai-CN in #1462
• Update benchmark script by @JinHai-CN in #1467
• local infinity modify by @Ma-cat in #1468
• Fix slow_test and timeout_kill.sh for new runner by @yuzhichang in #1469
• Fix idxreplay by @small-turtle-1 in #1470
• Support local infinity to check the initialization path by @Ma-cat in #1471
• Fix fullckp by @small-turtle-1 in #1466
• Fix wrong meta directory by @JinHai-CN in #1472
• Fix SHOW and SET bugs by @JinHai-CN in #1474
• Update version and increase the fix length chunk size by @JinHai-CN in #1475
• Add: sparse benchmark. by @small-turtle-1 in #1476
• Fix update version info to 0.2.1 by @JinHai-CN in #1477

New Contributors

• @zjbpaul1317 made their first contribution in #1460

Full Changelog: v0.2.0...v0.2.1

v0.2.0

What's Changed

• Let thrift client reconnect on insert failure by @yuzhichang in #1156
• Declare fma requirement at CMakeLists.txt by @yuzhichang in #1162
• Update blockmax_maxscore_iterator.cppm by @eltociear in #1166
• Added python venv into CentOS 7 builder image by @yuzhichang in #1169
• Update infinity configs and variables. by @JinHai-CN in #1163
• Added quickstart guide. by @writinwaters in #1173
• Fix write conflict by @small-turtle-1 in #1170
• Fix path to build_from_source.md by @YReddice in #1186
• support phrase query with block max by @Ma-cat in #1189
• Support real time secondary index by @yangzq50 in #1171
• Rearranged docs by @writinwaters in #1190
• Fix unittest stuck by @small-turtle-1 in #1192
• Fix parallel readwrite by @small-turtle-1 in #1191
• optimize full-text index creation time by @Ma-cat in #1199
• correct python dependency installation command and add comments to gitignore by @sakura-mac in #1202
• Implement show configs, show session and global variables. by @JinHai-CN in #1203
• Initial support for tensor data type by @yangzq50 in #1205
• Refactor set command by @JinHai-CN in #1206
• Implement HTTP API: SHOW CONFIGS/VARIABLES/CONFIG/VARIABLE, SET CONFIG/VARIABLE by @JinHai-CN in #1207
• Fix python client connecting to server slowly. by @JinHai-CN in #1209
• Refactor tokenizer loading by @JinHai-CN in #1210
• Unified compaction. by @small-turtle-1 in #1208
• Fix MergeAggregate load meta by @yangzq50 in #1212
• Refactor benchmark script by @yuzhichang in #1213
• Updated benchmark.md by @yuzhichang in #1216
• Fix: compact parallel bug. by @small-turtle-1 in #1214
• Support Japanese morphological analyzer by @yingfeng in #1215
• Update build image for opencloudos and docs by @yingfeng in #1218
• Add MatchTensorScan operator by @yangzq50 in #1219
• LOG_ERROR message before raise recoverable error by @JinHai-CN in #1217
• Change log level by @JinHai-CN in #1222
• Add license of mecab by @yingfeng in #1223
• Refactor code issues by @JinHai-CN in #1224
• Refactor code by @JinHai-CN in #1225
• Refactor more codes by @JinHai-CN in #1226
• Refactor: update container and remove unused codes. by @JinHai-CN in #1227
• Introduced blockmax wand by @yuzhichang in #1229
• Merge duplicated TermQueryNode children by @yuzhichang in #1231
• Fix: compact bug. by @small-turtle-1 in #1230
• Fulltext preformance improvement by @yuzhichang in #1234
• Add support for TensorArray data type by @yangzq50 in #1233
• Fix: restart conflict by @yangzq50 in #1237
• Support default value and json format import for Tensor and TensorArray data type by @yangzq50 in #1241
• Simple sparse idx by @small-turtle-1 in #1238
• Optimize index creation time by @Ma-cat in #1240
• Refactor benchmark scripts by @yuzhichang in #1243
• Provide Tensor MaxSim reranker for Fusion operator by @yangzq50 in #1244
• Added back multiprocessing to benchmark scripts by @yuzhichang in #1246
• [Feature]: Exports table to JSONL or CSV file. by @JinHai-CN in #1245
• Removed MemoryPool and RecyclePool by @yuzhichang in #1248
• Check python SDK version when client connected with server. by @JinHai-CN in #1247
• Simple Sparse vector support. by @small-turtle-1 in #1249
• Minor improvement of fulltext query by @yuzhichang in #1250
• Add more support to sparse vector. by @small-turtle-1 in #1251
• Add more support to sparse vector. by @small-turtle-1 in #1254
• Refactor: Error logging by @JinHai-CN in #1256
• Fix: sparse info deserialize bug. by @small-turtle-1 in #1259
• Enable Phrase only for STANDARD by @yuzhichang in #1261
• Update python and http api for match tensor and fusion by @yangzq50 in #1255
• Fix sparse parser. by @small-turtle-1 in #1262
• Add distance and similarity metric as output in KNN search by @JinHai-CN in #1260
• Update client version by @JinHai-CN in #1264
• Fulltext benchmark improvement by @yuzhichang in #1263
• Support MatchTensor Scan and Reranker on TensorArray column by @yangzq50 in #1267
• Update Documents by @JinHai-CN in #1268
• fulltext Share PostingFormat among PostingWriters by @yuzhichang in #1270
• Update links by @writinwaters in #1271
• Add new command: show index segment info by @JinHai-CN in #1273
• Add new command: show index chunk info by @JinHai-CN in #1275
• Fix redundant code by @JinHai-CN in #1277
• Fix missing first_delete_ts_ by @yangzq50 in #1278
• Refactor code by @JinHai-CN in #1280
• Fixed query_batch type mismatch by @yuzhichang in #1282
• Sparse knn1 by @small-turtle-1 in #1283
• Add python example for ColBERT reranker by @yangzq50 in #1284
• Added rrf window_size by @yuzhichang in #1288
• Fix python scripts by @yangzq50 in #1287
• Multiway fusion by @yuzhichang in #1289
• Add three global variables by @JinHai-CN in #1290
• Refactor physical match by @JinHai-CN in #1285
• Fix read file function by @JinHai-CN in #1291
• Fix export embedding to wrong json type by @JinHai-CN in #1293
• [Bug] Fix import jsonl and json file, row count zero by @JinHai-CN in #1294
• Purge bad entries after latest checkpoint by @yuzhichang in #1296
• Stream import by @JinHai-CN in #1298
• Add show buffer command by @JinHai-CN in #1299
• Added weighted_sum fusion by @yuzhichang in #1301
• Changed weighted_sum normalization method by @yuzhichang in #1302
• Sparse knn2 by @small-turtle-1 in #1292
• Escape reserved character in query string by @yuzhichang in #1303
• Fix: mem leak. by @small-turtle-1 in #1304
• Fix garbage outputs of Chinese analyzer by @yingfeng in #1306
• API: export table to file by @JinHai-CN in #1300
• Add EMVB search: Part 1 by @yangzq50 in #1305
• Secondary index support string by @wuxiaobai24 in #1307
• Introduced CutGrain for Chinese analyzer by @yuzhichang in #1309
• Improve Chinese analyzer by @yingfeng in https://github.com/infiniflow/infinity/p...