Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [CDCSOURCE] MySQL full database synchronization fails in dinky standalone mode #3452

Closed
3 tasks done
Hello-Linux opened this issue May 6, 2024 · 3 comments
Closed
3 tasks done
Labels
Bug Something isn't working Waiting for reply Waiting for reply

Comments

@Hello-Linux
Copy link

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

使用dinky 作业模式选择Local的话mysql 全库同步成功,但是使用standalone模式报错如下

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot instantiate user function.
	at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:405)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperator(OperatorChain.java:859)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:826)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:722)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:815)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:722)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:815)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:722)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:202)
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.<init>(RegularOperatorChain.java:60)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:707)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:693)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.InvalidClassException: org.dinky.cdc.sql.SQLSinkBuilder; unable to create instance
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
	at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.base/java.io.ObjectInputStream.readArray(Unknown Source)
	at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source)
	at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
	at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source)
	at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
	at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source)
	at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
	at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
	at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
	at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:481)
	at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:389)
	... 16 more
Caused by: java.lang.reflect.InvocationTargetException
	at jdk.internal.reflect.GeneratedSerializationConstructorAccessor299.newInstance(Unknown Source)
	at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	at java.base/java.io.ObjectStreamClass.newInstance(Unknown Source)
	... 38 more
Caused by: java.lang.NoClassDefFoundError: com/google/common/collect/Lists
	at org.dinky.cdc.sql.SQLSinkBuilder.initTypeConverterList(SQLSinkBuilder.java:61)
	at org.dinky.cdc.AbstractSinkBuilder.<init>(AbstractSinkBuilder.java:93)
	... 41 more

What you expected to happen

standalone 模式下全库同步成功

How to reproduce

flink version: 1.18.1 dinky: 1.0.0-rc3
默认各种依赖包都安装好了

Anything else

No response

Version

1.0.0

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@Hello-Linux Hello-Linux added Bug Something isn't working Waiting for reply Waiting for reply labels May 6, 2024
@github-actions github-actions bot changed the title dinky standalone 模式下mysql 全库同步失败 MySQL full database synchronization fails in dinky standalone mode May 6, 2024
@Hello-Linux
Copy link
Author

Hello-Linux commented May 6, 2024

/[flink-home]/lib jar文件列表
001
/[dinky-home]/extends/flink1.18/dinky/目录下的jar文件
002

@HejiaHo
Copy link

HejiaHo commented May 6, 2024

我遇到同样的问题,参考 #3299 的方法得到解决。
总结:

  1. 将 dinky-app 文件拷贝到两个目录下:/[flink-home]/lib/[dinky-home]/extends/flink1.18/dinky/ ,第二个注意是dinky下。
  2. 重启flink、dinky 重试后成功。

/[flink-home]/lib 下文件

dinky-app-1.18-1.0.1-jar-with-dependencies.jar
dinky-client-1.18-1.0.1.jar
dinky-client-base-1.0.1.jar
dinky-common-1.0.1.jar
flink-cep-1.18.0.jar
flink-connector-files-1.18.0.jar
flink-connector-jdbc-3.1.2-1.18.jar
flink-csv-1.18.0.jar
flink-dist-1.18.0.jar
flink-json-1.18.0.jar
flink-scala_2.12-1.18.0.jar
flink-sql-connector-mysql-cdc-3.0.0.jar
flink-table-api-java-uber-1.18.0.jar
flink-table-planner-loader-1.18.0.jar
flink-table-runtime-1.18.0.jar
log4j-1.2-api-2.17.1.jar
log4j-api-2.17.1.jar
log4j-core-2.17.1.jar
log4j-slf4j-impl-2.17.1.jar
mysql-connector-j-8.1.0.jar

/[dinky-home]/extends/flink1.18/dinky/ 下文件

dinky-app-1.18-1.0.1-jar-with-dependencies.jar
dinky-catalog-mysql-1.18-1.0.1.jar
dinky-client-1.18-1.0.1.jar

@Hello-Linux
Copy link
Author

Hello-Linux commented May 7, 2024

@HejiaHo 部署的时候我就已经把dinky-app*.jar拷贝到对应的目录下了,还是不行的,报错跟原来#3299 缺少dinky-app的报错还不是完全一样的,不知道问题出在哪里。。。。。

@Hello-Linux Hello-Linux changed the title MySQL full database synchronization fails in dinky standalone mode [Bug] [CDCSOURCE] MySQL full database synchronization fails in dinky standalone mode May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Waiting for reply Waiting for reply
Projects
None yet
Development

No branches or pull requests

2 participants