-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MRConnector doesn't refresh tikv hosts when a node is rotated in a cluster #148
Comments
Steps to reproduce -
|
Does it cause tasks to fail? We can't tell who is the caller of resolver from stack. |
Do we have more logs about the failed task? For example the entire log of failed process after redaction. |
This stack trace was from hadoop logs. The job does not fail due to it but due to tons of noise, this caused debugging of another issue very hard. Is it possible to fix it ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It tries to connect to old tikv nodes even after tidb makes them "tombstone". When running TiDBMRDemo , I see following errors
Feb 01, 2022 12:53:06 AM org.tikv.shade.io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<21>: (XXX-YYY-ZZZ.com:20160)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host XXX-YYY-ZZZ..com, cause=java.lang.RuntimeException: java.net.UnknownHostException: XXX-YYY-ZZZ..com: Name or service not known
at org.tikv.shade.io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
at org.tikv.shade.io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
at org.tikv.shade.io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: XXX-YYY-ZZZ.com: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at org.tikv.shade.io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
at org.tikv.shade.io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
... 5 more
}
Feb 01, 2022 12:53:06 AM org.tikv.shade.io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<31>: (XXX-YYY-ZZZ.com:20160)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host XXX-YYY-ZZZ.com, cause=java.lang.RuntimeException: java.net.UnknownHostException: XXX-YYY-ZZZ.com: Name or service not known
at org.tikv.shade.io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
at org.tikv.shade.io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
at org.tikv.shade.io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: XXX-YYY-ZZZ.com: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at org.tikv.shade.io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
at org.tikv.shade.io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
... 5 more
}
The text was updated successfully, but these errors were encountered: