QR IMAGE GENERATOR NOT RUNNING

[bhojwanisankalp]

Hello everyone I am trying to run dial-service and knowledge-platform-jobs/qrcode-image-generator/ .
My goal is to implement dial-service and generate the qrcode image for published dial codes. I have created some sample dial-codes following the documents and postman collection but currently I am stuck at the qrcode generating part. Have followed the steps like

1. Downloaded flink-1.12.0-scala_2.12
2. Downloaded hadoop dependencies for flink and pasted it in lib folder of flink
3. export HADOOP_CLASSPATH
4. A running yarn cluster using
   ./bin/flink run -m yarn-cluster -p 2 -yjm 1024m -ytm 1024m /qrcode-image-generator/target/qrcode-image-generator-0.0.1.jar
   (changed -> path to the root directory)
   I have run the qrcode-image-generator but it gets failed after some time PFB for the error logs.
   I am not familiar with scala, any help will be appreciated to troubleshoot this problem. Thanks

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/sankalp/Music/kreedo/knowledge-platform-jobs/qrcode-image-generator/flink-1.12.0/lib/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-07-13 20:37:10,374 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/home/sankalp/Music/kreedo/knowledge-platform-jobs/qrcode-image-generator/flink-1.12.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2023-07-13 20:37:10,470 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at /0.0.0.0:8032
2023-07-13 20:37:10,612 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2023-07-13 20:37:10,925 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, slotsPerTaskManager=1}
2023-07-13 20:37:23,718 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1689253877358_0002
2023-07-13 20:37:23,742 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1689253877358_0002
2023-07-13 20:37:23,742 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated
2023-07-13 20:37:23,744 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying cluster, current state ACCEPTED
2023-07-13 20:37:32,831 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - YARN application has been deployed successfully.
2023-07-13 20:37:32,832 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface pop-os.localdomain:42221 of application 'application_1689253877358_0002'.
Job has been submitted with JobID 29623b94a093861dbd923c612887805f

The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: 29623b94a093861dbd923c612887805f)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:971)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1047)
Caused by: java.util.concurrent.ExecutionException: org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: 29623b94a093861dbd923c612887805f)
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at org.apache.flink.client.program.StreamContextEnvironment.getJobExecutionResult(StreamContextEnvironment.java:116)
at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:80)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1822)
at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:746)
at org.sunbird.job.qrimagegenerator.task.QRCodeImageGeneratorTask.process(QRCodeImageGeneratorTask.scala:37)
at org.sunbird.job.qrimagegenerator.task.QRCodeImageGeneratorTask$.main(QRCodeImageGeneratorTask.scala:52)
at org.sunbird.job.qrimagegenerator.task.QRCodeImageGeneratorTask.main(QRCodeImageGeneratorTask.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:316)
... 11 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: 29623b94a093861dbd923c612887805f)
at org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:119)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$22(RestClusterClient.java:602)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:379)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1085)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
at org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:117)
... 18 more
Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=30000)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78)
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:224)
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:217)
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:208)
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:610)
at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89)
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:419)
at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:286)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:201)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:154)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Container name not configured.
at org.sunbird.job.util.CloudStorageUtil.getContainerName(CloudStorageUtil.scala:31)
at org.sunbird.job.util.CloudStorageUtil.(CloudStorageUtil.scala:14)
at org.sunbird.job.qrimagegenerator.functions.QRCodeImageGeneratorFunction.open(QRCodeImageGeneratorFunction.scala:29)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at org.apache.flink.streaming.api.operators.ProcessOperator.open(ProcessOperator.java:56)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:401)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:507)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:531)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
at java.base/java.lang.Thread.run(Thread.java:829)

[kameshbhr]

@pallakartheekreddy, @Jayaprakash8887 Can you please check and respond to this?
cc; @Krishnaj20

[Jayaprakash8887]

@bhojwanisankalp Was there any code changes done on QRCodeImageGeneratorTask?

Caused by: java.lang.Exception: Container name not configured.
at org.sunbird.job.util.CloudStorageUtil.getContainerName(CloudStorageUtil.scala:31)
at org.sunbird.job.util.CloudStorageUtil.(CloudStorageUtil.scala:14)
at org.sunbird.job.qrimagegenerator.functions.QRCodeImageGeneratorFunction.open(QRCodeImageGeneratorFunction.scala:29)

Also, please configure below attributes as per your cloud container:

cloud_storage_type=""
cloud_storage_key=""
cloud_storage_secret=""
cloud_storage_container=""
cloud_storage_endpoint=""

[bhojwanisankalp]

Hi @Jayaprakash8887 I have given the required values for
cloud_storage_type=""
cloud_storage_key=""
cloud_storage_secret=""
cloud_storage_container=""
cloud_storage_endpoint=""
The qr-code-generator process is running. But when I try to publish the dial code the process is not getting any input to generate the qr code. Process like cassandra db, kafka and zookeeper are also running in docker container.
Can't find any steps to create the qr-code against dial codes in the documentation.

[bhojwanisankalp]

@Jayaprakash8887 the job is not working as expected. It should create a qr-code image for the published dial code right. The dial code service is showing dial code to be successfully published but the qr-imag-generator is not getting any request to create a image. It should be triggered through a Kafka topic event I believe according to the docs.

[Jayaprakash8887]

@bhojwanisankalp Yes. Event will be inserted by knowledge middleware. Please check what is happening in knowldge-mw

[pallakartheekreddy]

@bhojwanisankalp is your issue resolved?

[bhojwanisankalp]

Hi @Jayaprakash8887 / @pallakartheekreddy I have tried publishing dial code with knowledge middleware but the qr image generator is still not getting any request to create a qr image. Not sure if I am missing any thing.

[pallakartheekreddy]

@bhojwanisankalp Is your issue resolved?

[pallakartheekreddy]

@bhojwanisankalp As we haven't received a response from you, we will proceed to close this thread. If you have any further questions or require assistance in the future, please don't hesitate to start a new thread. Thank you