Input/output error

[ltagliamonte-dd]

Hello Folks!
I'm recording this error in my app:

{
  "message":        "time=2024-08-23T12:55:29.868Z level=ERROR msg=\"Error running HSET command during batch load\" error=\"response returned from Conn: ERR IO error: While pread offset 691867648 len 16319: /data/kvrocks/db/027707.sst: Input/output error\""
}

the Kvrock instance seems healthy I don't see any sign of stress, would appreciate if anyone has any insight on how to troubleshoot this.

[git-hulk]

Hi @ltagliamonte-dd, this seems a hardware issue. Could you please have a check at your system message via dmesg?

[ltagliamonte-dd]

Hello @git-hulk , thanks but unfortunately this was running in a docker container in k8s and the node has been reclaimed.
Will make sure to add an alert for this and investigate further next time this happens.

[git-hulk]

Sure, you can check the Kubernetes node message if you know it's running on which one.