run on k8s,Added keberos authentication, the first startup was successful, but after restarting,it cannot be started

[seawenc]

base info

ozone version: 1.4.1
orchestration files : kubernetes/examples/ozone-ha/

description

when add keberos authentication, the first startup was successful, but after restarting,it cannot be started

When I deleted the /data directory and restarted it, it was successful

I found that the scm-ui found that all the statuses of ratis were: FOLLOWER

{ HostName : scm-2.scm.ozone.svc.cluster.local, Ratis Port : 9894, Role : FOLLOWER } { HostName : scm-0.scm.ozone.svc.cluster.local, Ratis Port : 9894, Role : FOLLOWER } { HostName : scm-1.scm.ozone.svc.cluster.local, Ratis Port : 9894, Role : FOLLOWER }

SCM-0 error logs:

2024-12-26 09:09:12 INFO  UserGroupInformation:1132 - Login successful for user ozone/ozone@SKY.COM using keytab file ozone.keytab. Keytab auto renewal enabled : false
2024-12-26 09:09:12 INFO  StorageContainerManager:1074 - SCM login successful.
.....
2024-12-26T06:32:29.097315580Z 2024-12-26 06:32:29 INFO  Server:3109 - IPC Server handler 1 on default port 9961, call Call#0 Retry#35832 org.apache.hadoop.hdds.protocol.SCMSecurityProtocol.submitRequest from 10-42-4-248.ozone-debug.ozone.svc.cluster.local:51958 / 10.42.4.248:51958
org.apache.hadoop.hdds.ratis.ServerNotLeaderException: Server:23f9f48d-ba3c-4288-9b82-2dbc96521f18 is not the leader. Could not determine the leader node.
	at org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:110)
2024-12-26T06:32:29.097349366Z 	at org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:248)
	at org.apache.hadoop.hdds.scm.protocol.SCMSecurityProtocolServerSideTranslatorPB.submitRequest(SCMSecurityProtocolServerSideTranslatorPB.java:90)
	at org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos$SCMSecurityProtocolService$2.callBlockingMethod(SCMSecurityProtocolProtos.java:18732)
2024-12-26T06:32:29.097354887Z 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:484)
2024-12-26T06:32:29.097356629Z 	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:595)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
2024-12-26T06:32:29.097364999Z 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
2024-12-26T06:32:29.097366746Z 	at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
2024-12-26T06:32:29.097368384Z 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)

[ChenSammi]

It looks like Kerberos authentication is passed, "Login successful for user ozone/ozone@SKY.COM using keytab file ozone.keytab."

Is it able to share the full log of three SCMs?