v1.20.1

Dragonfly v1.20.1

This is a patch release.

fix: S3 access for loading/saving snapshots (#3296)

Signed-off-by: kostas kostas@dragonflydb.io

v1.20.0

Dragonfly v1.20.0

Some prominent changes include:

• Improvements around client side caching (#3136 #3158)
  Specifically, rueidis is now supported (#2454)
• Improvements around search and vector search APIs, FT.ALTER is now supported (#3144, #3148, #3178, #3186)
• JSON.MSET is added (#3167)
• Expiry notifications are now supported (#3154)

What's Changed

• test(cluster_mgr): Add tests for cluster_mgr.py by @chakaz in #3129
• chore: get rid of kv_args and replace it with slices to full_args by @romange in #3101
• fix: Fix live-lock in connection test by @chakaz in #3135
• chore: call breaker_cb_ on shutdown by @romange in #3128
• chore: Add option to disable aws dependency by @dranikpg in #3077
• feat(cluster): Add important logs by @chakaz in #3138
• chore: a small cleanup by @romange in #3140
• test: skip test_cluster_migration_cancel, it is broken by @BorysTheDev in #3146
• chore: update helio by @kostasrim in #3150
• fix: fix bug in cluster/slot_set by @BorysTheDev in #3143
• chore: recommit client tracking by @kostasrim in #3136
• feat(search): Tag field options (separator, case sensitivity) by @dranikpg in #3144
• feat(search): basic FT.ALTER by @dranikpg in #3148
• chore: Introduce pipeline back-pressure by @romange in #3152
• fix: lua and client tracking by @kostasrim in #3163
• feat: client tracking optout by @kostasrim in #3158
• feat: add noloop subcommand in client tracking by @kostasrim in #3164
• fix(ci): docker not using iouring by @kostasrim in #3169
• fix(ci): add missing docker option on reg tests workflow by @kostasrim in #3172
• fix: acl compatibility by @kostasrim in #3147
• chore: Streamer is rewritten with async interface by @romange in #3108
• feat(json): MSET by @dranikpg in #3167
• fix(cluster): Wait for flow creation before cancelling it by @chakaz in #3159
• chore: improve ft.create for vector search by @romange in #3178
• feat: add incoming migration error processing by @BorysTheDev in #3175
• chore: add parsing support for the rest of VECTOR parameters by @romange in #3179
• fix(cluster): Wait before all access to slot migrations by @chakaz in #3180
• fix(server): fix bug in replication on cached mode by @adiholden in #3156
• chore: small acl compat changes by @kostasrim in #3177
• chore: allow calling Context::ReportError from I/O dispatch fiber by @romange in #3182
• chore: add a lexer test covering KNN query by @romange in #3185
• chore(search): Add __vector_score alias by @dranikpg in #3186
• chore: Refactor string span management by @dranikpg in #3165
• chore(ci): run replication tests on arm by @kostasrim in #3168
• chore(search): improve parser error reporting by @romange in #3184
• fix(reply_builder): remove virtual modifier in SendError(ErrorReply) method by @BagritsevichStepan in #3191
• fix(cluster): Support FLUSHALL while slot migration is in progress by @chakaz in #3173
• fix: fix RegisterOnChange methods for journal and db_slice by @BorysTheDev in #3171
• fix(generic_family): fix RenameGeneric command for non-string data types by @BagritsevichStepan in #3181
• fix(server): Rename confusing flag replica_reconnect_on_master_restart by @chakaz in #3193
• fix(unit tests): fix generic family info test by @adiholden in #3187
• test: improve cluster_fuzzy_migration test by @BorysTheDev in #3197
• chore(core): Remove DfImpl inside ScoredMap by @dranikpg in #3199
• chore(tiering): Remove IoMgr by @dranikpg in #3198
• test(cluster): Make sure migration maintains TTL by @chakaz in #3188
• feat(cluster): Support STICK bit in slot migration by @chakaz in #3200
• chore: Re-enable previously flaky test by @chakaz in #3196
• feat(acl): add support of multiple passwords by @kostasrim in #3189
• chore(tiering): More advanced tiering tests by @dranikpg in #3201
• chore: add replica-priority flag by @kostasrim in #3204
• chore: fix tiering regtest test by @dranikpg in #3208
• feat(generic_family): Assemble RESTORE and STICK commands into one commmand during replication of the RENAME command by @BagritsevichStepan in #3209
• core(search): Add EF_RUNTIME parameter by @dranikpg in #3206
• fix(sanitizers): failing json test by @kostasrim in #3176
• feat(server): expiry notifications by @dranikpg in #3154
• fix: replicaof_reject_on_load flake by @kostasrim in #3203
• fix(cluster): Don't miss keys when migrating slots by @chakaz in #3218
• fix(tiering): Fix pending leak during immediate stash error by @dranikpg in #3219
• chore: provide basic logging to catch possible command errors by @romange in #3213
• chore: disable tiering test by @dranikpg in #3236
• fix(transaction): Don't transactionalize empty EVAL inside EXEC by @dranikpg in #3231
• fix: fix deadlock in DFLYCLUSTER CONFIG command and outgoing migration finaliztion by @BorysTheDev in #3239
• chore(tiered): minor fixes + expose buffer allocation type stats by @romange in #3234
• fix(pytest): timed ticker for simpler conditions by @dranikpg in #3242
• fix(server): Fix SCAN deadlock by @dranikpg in #3235
• chore: introduce back-pressure to tiered storage by @romange in #3243
• chore: improve dfly_bench by @romange in #3244
• chore(tiering): External alloc free page chaining by @dranikpg in #3237
• fix: total_stash_overflows statistic by @romange in #3248
• chore: update action versions by @romange in #3250
• chore(lua): Return which undeclared key was accessed by @chakaz in #3245
• chore: replace session wide fixtures with scope by @kostasrim in #3251
• chore: pull helio and adjust mimalloc for 32MiB segments by @kostasrim in #3174
• chore: add more logs around the duplicate value case in small_bins by @romange in #3254
• fix: fix move error during migration finalization by @BorysTheDev in #3253
• chore: upload all the logs from /tmp by @romange in #3263
• fix(tiering): Throttle snapshot load by @dranikpg in #3249
• chore: initiate grow preemptively by @romange in #3267
• chore(monitoring): add more dashboards + memcached by @romange in #3268
• fix(json_family): fix JSON.GET commmand for JSON legacy mode by @BagritsevichStepan in #3261
• chore: dfly_bench can send traffic to memcached by @romange in #3271
• chore: print effective QPS of the server. by @romange in #3274
• chore: debugging fixes by @romange in https...

v1.19.2

Dragonfly v1.19.2

This is a patch release.

What's Changed

chore(lua): Return which undeclared key was accessed (#3245)

Urgency to update

Low

v1.19.1

Dragonfly v1.19.1

This is a patch release.

What's Changed

fix: allow SCANs running within lua scripts #3232
fix: avoid crashing when calling empty EVAL inside MULTI transaction #3230

v1.19.0

Dragonfly v1.19.0

Some prominent changes include:

• SPOP and SRANDMEMBER return truly randomized results (#3022)
• Fix memory blowup in some cases during replication (#3084 #3103)
• Snapshotting works with data tiering (#3073)
• Fix incompatibility issue of RDB snapshot with Redis 6.x (#3121)
• Memory utilization improvements + better introspection of memory usage via /metrics

What's Changed

• chore: update versions and remove caching code by @romange in #3030
• chore: fix macos tiering by @dranikpg in #3032
• feat: support subrange expressions in jsonpathv2 by @romange in #3036
• fix: reduce delay when stop replica #3020 by @BorysTheDev in #3028
• fix(set): fix random in SRANDMEMBER and SPOP commands by @BagritsevichStepan in #3022
• chore: prevent updating gcc on macos by @romange in #3037
• chore: add half-range indices by @romange in #3041
• chore: update helio together with new mimalloc version by @romange in #3040
• fix: header for oom_errors_total by @romange in #3043
• chore: reset enable_direct_fd to avoid socket leakage in kernel by @romange in #3051
• chore: update deprecated actions by @romange in #3052
• chore: remove unneeded check-fails by @romange in #3054
• chore: Export replication memory stats by @romange in #3062
• !: Update grafana panel by @MaoMaoCake in #3064
• chore: add replication memory stats to the dashboard by @romange in #3065
• chore: bpop prints by @dranikpg in #3076
• fix(tiering): wait for IO before test teardown by @dranikpg in #3079
• chore: small replayer fixes by @dranikpg in #3081
• fix: increase lua stack limit to 8KB by @romange in #3080
• chore: minor fixes by @romange in #3082
• fix: fix cluster incorrect keys status by @BorysTheDev in #3083
• fix(acl): return -NOPERM instead of response error by @kostasrim in #3049
• feat(server): Expose serialization bytes via INFO and /metrics by @chakaz in #3087
• chore(streams): Some refactoring by @dranikpg in #3088
• feat(server): remove multi shard sync from replication by @adiholden in #3085
• fix(replication): fullsync phase write to sync on noop by @adiholden in #3084
• fix(tiering): Async delete for small bins by @dranikpg in #3068
• fix: prevent crashing if error happened during snapshot moving by @BorysTheDev in #3092
• fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by @BorysTheDev in #3093
• chore: improve Migration() by @kostasrim in #3033
• feat(streams): Stream optimizations by @dranikpg in #3091
• feat(tiering): Defragmentation by @dranikpg in #3021
• fix: bit shift of kTagMask by @haodongnj in #3099
• feat(cluster_mgr): Allow attaching replicas by @chakaz in #3105
• bug(server): fix replication stuck in full sync by @adiholden in #3103
• fix(server): Sync FLUSH with tiering by @dranikpg in #3098
• chore: export pipeline related metrics by @romange in #3104
• chore: zset error log (from 1.18 branch) by @dranikpg in #3109
• New test for cluster migration: connection issue by @BorysTheDev in #3102
• chore: minor fixes + remove redundant DCHECK by @romange in #3115
• fix: fix cluster_fuzzy_migration test by @BorysTheDev in #3116
• feat(server): Pipeline and dispatch on /metrics by @chakaz in #3117
• chore: pull helio add test for tls deadlock by @kostasrim in #3111
• feat(cluster_mgr): Improvements to cluster_mgr.py by @chakaz in #3118
• refactor client tracking, fix atomicity, squashing and multi/exec by @kostasrim in #2970
• fix(server): fix compatibility with rdb snapshot by @adiholden in #3121
• Revert "refactor client tracking, fix atomicity, squashing and multi/… by @adiholden in #3122
• feat(cluster_mgr): Take over command by @chakaz in #3120
• fix: fix RestoreStreamer to prevent buckets skipping #2830 by @BorysTheDev in #3119
• feat(cluster_mgr): Fix migration action by @chakaz in #3124
• chore: prevent Dispatch fiber to be launched during migration by @romange in #3123
• feat(tiering): Simple snapshotting by @dranikpg in #3073
• fix: upload action in version release by @romange in #3127

New Contributors

• @MaoMaoCake made their first contribution in #3064
• @haodongnj made their first contribution in #3099

Huge thanks to all the contributors! ❤️

Full Changelog: v1.18.0...v1.19.0

v1.18.1

Dragonfly v1.18.1

This is a patch release.

What's Changed

fix: prevent socket leakage in kernel with io_uring mode (#3051)

v1.18.0

Dragonfly v1.18.0

Some prominent changes include:

• ACL improvements: #2945 #2943 #2920 #2982 #2995
• Implementation of json.merge #2960
• Replication - memory improvements
• Very much alpha support for data tiering. Try it out with --tiered_prefix=/pathto/ssd/base and see how your memory usage goes down (STRING type only). Do not use it in prod! 😸

What's Changed

• feat: retry ACK if the configs are different #2833 by @BorysTheDev in #2906
• chore(tiering): Update Get, Set, Del by @dranikpg in #2897
• chore: preparation step for lock fingerprints by @romange in #2899
• fix(transaction): Use FinishHop in schedule by @dranikpg in #2911
• chore(tiering): Fix MacOs build by @dranikpg in #2913
• feat(cluster): Migration cancellation support by @chakaz in #2869
• feat: process migration data after FIN opcode #2864 by @BorysTheDev in #2918
• chore(string_family): Refactor SetCmd by @dranikpg in #2919
• fix: Improve reply latency of HELLO by @romange in #2925
• chore: improve reply latency of SendScoredArray by @romange in #2929
• Namespace support in prometheus rule by @Pothulapati in #2931
• fix: socket closed when RegisterOnErrorCb is called in HandleRequests by @kostasrim in #2932
• chore: bring more clarity when replayer fails by @romange in #2933
• Slot migration cancel crash fix by @BorysTheDev in #2934
• feat: add ability reaply config with migration #2924 by @BorysTheDev in #2926
• fix(test): Unflake fuzzy cluster migration test by @chakaz in #2927
• chore: Remove Schedule() call by @romange in #2938
• chore: get rid of lock keys by @romange in #2894
• fix: introduce info_replication_valkey_compatible flag by @romange in #2936
• feat(metrics): adding max_clients to metrics and info output (#2912) by @racamirko in #2940
• chore: adjust transaction code to keystep/3 commands by @romange in #2941
• feat(tiering): Get, GetSet, Set test by @dranikpg in #2921
• chore(acl): adjust some ACL command responses by @Niennienzz in #2943
• chore: Pull helio with new future by @dranikpg in #2944
• refactor: add cluster namespace by @BorysTheDev in #2948
• chore: Introduce ShardArgs as a distinct type by @romange in #2952
• chore: Log db_index in traffic logger by @dranikpg in #2951
• fixes for v1.18.0 by @adiholden in #2956
• feat(tiering): Support append (and modifications in general) by @dranikpg in #2949
• feat: extended bracket index in jsonpath by @romange in #2954
• chore: Remove TieringV1 by @dranikpg in #2962
• fix(pytests): replace proc.wait() with proc.communicate() to avoid deadlocks by @kostasrim in #2964
• feat(tiering): Registered buffers by @dranikpg in #2967
• feat: add slot migration error processing by @BorysTheDev in #2957
• chore(acl): allow multiple users in acl deluser by @kostasrim in #2945
• feat: implement json.merge by @romange in #2960
• fix: fix deadlock and slot flush for migration cancel #2968 by @BorysTheDev in #2972
• chore(tiering): Lots of metrics by @dranikpg in #2977
• fix: crash during migration when connection is closing by @kostasrim in #2973
• fix: remove acl-check and cancel instead when REPLCONF ACK fails to validate by @kostasrim in #2920
• fix: check return code of process after communicate by @kostasrim in #2976
• fix: allow non hashed passwords when loading users from acl file by @kostasrim in #2982
• chore: update our container distributions versions by @romange in #2983
• chore: remove version checks when running our regtests by @romange in #2988
• chore(acl): add vlog and check on deluser flow by @kostasrim in #2995
• fix(memcached): Register memcached listener to handle --maxclients by @chakaz in #2985
• chore: another preparation commit to get rid of kv_args in transaction by @romange in #2996
• chore: improve performance of Scan operation by @romange in #2990
• fix(server): small string allocations only under 256 bytes str by @adiholden in #2991
• fix(cluster-migration): Support cancelling migration right after starting it by @chakaz in #2992
• chore: fix double header issue by @romange in #3002
• chore: small tiering fixes by @dranikpg in #2966
• feat(benchmark-tests): run in K8s by @zacharya19 in #2965
• Benchmark fixes by @zacharya19 in #3005
• fix(tiering): rename v2 + max_file_size by @dranikpg in #3004
• chore: fix tiering macos stub by @dranikpg in #3006
• chore: export listener stats by @romange in #3007
• chore: pull latest helio by @romange in #3009
• fix(server): lag is 0 when server not in stable state by @adiholden in #3010
• chore: get rid of kv_args and replace it with slices to full_args by @romange in #2942
• fix(server): non auto journal write after callback finish by @adiholden in #3016
• fix(server): shrink replication steaming buf by @adiholden in #3012
• fix(zset): fix random in ZRANDMEMBER command by @BagritsevichStepan in #2994
• Fix benchmark by @adiholden in #3017
• chore: Remove tiering test skip by @dranikpg in #3011
• feat(tiering): simple offload loop by @dranikpg in #2987
• feat(tiering): MGET support by @dranikpg in #3013
• Revert "chore: get rid of kv_args and replace it with slices to full_… by @romange in #3024
• chore(dash): Replace comparator with predicate by @dranikpg in #3025
• feat: add defragment command by @BorysTheDev in #3003

Huge thanks to all the contributors! ❤️

New Contributors

• @racamirko made their first contribution in #2940
• @BagritsevichStepan made their first contribution in #2994

Full Changelog: v1.17.0...v1.18.0

v1.17.1

Dragonfly v1.17.1

This is a patch release.

What's Changed

fix: potential data race in the transactional code (#2911)

v1.17.0

Dragonfly v1.17.0

Some prominent changes include:

• Improved performance for MGET operations (#2453)
• Fix argument parsing in json.objkeys (#2872)
• Fix ipv6 support for replication (#2889)
• Support serialisation of bloom filters - saving to and loading from snapshots (#2846)
• Support of HLL PFADD (#2761)
• Support bullmq workloads that do not have {} hashtags in their queue names (#2890)

What's Changed

• fix: #2745 don't start migration process again after apply the same the same config is applied by @BorysTheDev in #2822
• feat(transaction): Idempotent callbacks (immediate runs) by @dranikpg in #2453
• refactor(cluster): replace sync_id with node_id for slot migration #2835 by @BorysTheDev in #2838
• feat(tiering): Simple OpManager by @dranikpg in #2781
• chore: implement path mutation for JsonFlat by @romange in #2805
• feat(cluster): add migration removing by config #2835 by @BorysTheDev in #2844
• chore: expose direct API on Bloom objects by @romange in #2845
• chore: generalize CompactObject::AllocateMR by @kostasrim in #2847
• feat(tiering): Simplest small bins by @dranikpg in #2810
• refactor: clean cluster slot migration code by @BorysTheDev in #2848
• fix(tests): Fix numsub test by @dranikpg in #2852
• fix: healthcheck for docker containers by @romange in #2853
• fix: possible crash in tls code by @romange in #2854
• fix(server): Do not block admin-port commands by @chakaz in #2842
• fix(pytest): make pytests fail if server crash on shutdown by @adiholden in #2827
• feat(server): add prints on takeover timeout by @adiholden in #2856
• fix(pytest): dont check process return code on kill by @adiholden in #2862
• fix: authorize the http connection to call commands by @romange in #2863
• feat(cluster): Send number of keys for incoming and outgoing migrations. by @chakaz in #2858
• feat(tiering): TieredStorageV2 by @dranikpg in #2849
• bug(server): set connection flags block/pause flag on all blocking commands by @adiholden in #2816
• chore: serialize SBF by @romange in #2846
• fix: test_replicaof_reject_on_load crash on stop by @kostasrim in #2818
• feat(dbslice): Add self-laundering iterator in DbSlice by @chakaz in #2815
• chore: License update by @romange in #2767
• fix(acl): incompatibilities with acl load by @kostasrim in #2867
• fix(json): make path optional in json.objkeys by @kostasrim in #2872
• fix: return wrong type errors for SET...GET command by @romange in #2874
• fix(redis replication): remove partial sync flow ,not supported yet by @adiholden in #2865
• chore: limit traffic logger only to the main interface by @romange in #2877
• chore: relax repltakeover constraints to only exclude write commands by @kostasrim in #2873
• chore(replayer): Roll back to go1.18 by @dranikpg in #2881
• fix: brpoplpush single shard to wake up blocked transactions by @kostasrim in #2875
• chore: LockTable tracks fingerprints of keys by @romange in #2839
• chore: reject TLS handshake when our listener is plain TCP by @romange in #2882
• Add support for Sparse HLL PFADD by @azuredream in #2761
• feat server: bring visibility to script errors by @adiholden in #2879
• chore: clean up REPLTAKEOVER flow by @romange in #2887
• chore(tiering): Move files and move kb literal to common by @dranikpg in #2868
• chore(interpreter): Support object replies by @dranikpg in #2885
• fix(ci/helm): Stick to v0.73.0 version of prom operator by @Pothulapati in #2893
• fix(acl): authentication with UDS socket by @kostasrim in #2895
• feat(cluster): add repeated ACK if an error is happened by @BorysTheDev in #2892
• chore(blocking): Remove faulty DCHECK by @dranikpg in #2898
• chore: add a clear link on how to build dragonfly from source by @romange in #2884
• feat(server): Allow configuration of hashtag extraction by @chakaz in #2890
• fix: fix build under macos by @BorysTheDev in #2901
• fix(cluster_replication): replicate redis cluster node bug fix by @adiholden in #2876
• fix(acl): skip http and add check on connection traversals by @kostasrim in #2883
• fix(zset): Better memory consumption calculation by @chakaz in #2900
• fix: fix ld for num converting by @BorysTheDev in #2902
• chore: add help string for memory_fiberstack_vms_bytes by @romange in #2903
• fix(sanitizers): false positive fail on multi_test::Eval by @kostasrim in #2896
• chore: pull helio and add ipv6 replication test by @dranikpg in #2889
• chore: add ipv6 support for native linux release by @romange in #2908

Huge thanks to all the contributors! ❤️

Full Changelog: v1.16.0...v1.17.0

v1.16.1

Dragonfly v1.16.1

This is a patch release.

What's Changed

• fix: authorize the http connection to call DF commands (#2863)
• possible crash in tls code (#2854 )
• healthcheck for docker containers (#2853)

Full Changelog: v1.16.0...v1.16.1