Random data loss problem

[actcwlf]

Hi, I found random data loss problem in my cluster and failed to figure out the reason.

The ddl is

create table test_trade_calendar (
    trade_dt date not null
)
UNIQUE KEY(`trade_dt`)
DISTRIBUTED BY HASH(`trade_dt`) BUCKETS AUTO
PROPERTIES (
    "replication_allocation" = "tag.location.default: 1"
);

The problem can be reproduced by the following steps:

1. insert continue date series from 20180101 to the newest date
2. insert newest date everyday using scheduling tools like crontab
3. check data integrity every hour. More specifically we check whether the most recent 30 dates are in the table.

After a few days (about 1~7 days) the problem will occur, as shown in the following picture

Since the sqls are generated by other framework, we export the relevant audit_log

audit_log.csv

It can be found that at line 316 doris returns only 29 rows while the result should same as line 301, which returns 30 rows.

We reproduce the problem on both version 2.0.3 and 2.1.1.

THANKS！

[maruyue]

中文问题，建议去ask.selectdb.com去问问题

[actcwlf]

好的，我在那里贴一下问题