CustomOCIStorageService implementation throwing error at runtime.

[Hari-stackroute]

@sowmya-dixit Lern exhaust jobs throwing error when we are calling the below code in OCI environment
FrameworkContext.getStorageService method
The error trace :
java.util.NoSuchElementException: key [aws-s3] not in the list of providers or apis: {} at org.jclouds.ContextBuilder.newBuilder(ContextBuilder.java:175) at org.ekstep.analytics.framework.storage.CustomOCIStorageService.<init>(CustomOCIStorageService.scala:22) at org.ekstep.analytics.framework.FrameworkContext.newStorageService(FrameworkContext.scala:63) at org.ekstep.analytics.framework.FrameworkContext.getStorageService(FrameworkContext.scala:81) at org.sunbird.core.util.DataSecurityUtil$.zipAndPasswordProtect(DataSecurityUtil.scala:151) at org.sunbird.core.exhaust.OnDemandExhaustJob.$anonfun$processRequestEncryption$3(OnDemandExhaustJob.scala:142) at scala.collection.immutable.List.map(List.scala:286) at org.sunbird.core.exhaust.OnDemandExhaustJob.$anonfun$processRequestEncryption$1(OnDemandExhaustJob.scala:139) at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:516) at org.sunbird.core.exhaust.OnDemandExhaustJob.processRequestEncryption(OnDemandExhaustJob.scala:139) at org.sunbird.core.exhaust.OnDemandExhaustJob.processRequestEncryption$(OnDemandExhaustJob.scala:138) at org.sunbird.lms.exhaust.collection.UserInfoExhaustJob$.processRequestEncryption(UserInfoExhaustJob.scala:8) at org.sunbird.core.exhaust.OnDemandExhaustJob.saveRequest(OnDemandExhaustJob.scala:135) at org.sunbird.core.exhaust.OnDemandExhaustJob.saveRequest$(OnDemandExhaustJob.scala:134) at org.sunbird.lms.exhaust.collection.UserInfoExhaustJob$.saveRequest(UserInfoExhaustJob.scala:8) at org.sunbird.core.exhaust.OnDemandExhaustJob$$anon$1.$anonfun$get$1(OnDemandExhaustJob.scala:126) at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:516) at org.sunbird.core.exhaust.OnDemandExhaustJob$$anon$1.get(OnDemandExhaustJob.scala:126) at org.sunbird.core.exhaust.OnDemandExhaustJob$$anon$1.get(OnDemandExhaustJob.scala:124) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) java.util.NoSuchElementException: key [aws-s3] not in the list of providers or apis: {} at org.jclouds.ContextBuilder.newBuilder(ContextBuilder.java:175) at org.ekstep.analytics.framework.storage.CustomOCIStorageService.<init>(CustomOCIStorageService.scala:22) at org.ekstep.analytics.framework.FrameworkContext.newStorageService(FrameworkContext.scala:63) at org.ekstep.analytics.framework.FrameworkContext.getStorageService(FrameworkContext.scala:81) at org.sunbird.core.util.DataSecurityUtil$.zipAndPasswordProtect(DataSecurityUtil.scala:151) at org.sunbird.core.exhaust.OnDemandExhaustJob.$anonfun$processRequestEncryption$3(OnDemandExhaustJob.scala:142) at scala.collection.immutable.List.map(List.scala:286) at org.sunbird.core.exhaust.OnDemandExhaustJob.$anonfun$processRequestEncryption$1(OnDemandExhaustJob.scala:139) at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:516) at org.sunbird.core.exhaust.OnDemandExhaustJob.processRequestEncryption(OnDemandExhaustJob.scala:139) at org.sunbird.core.exhaust.OnDemandExhaustJob.processRequestEncryption$(OnDemandExhaustJob.scala:138) at org.sunbird.lms.exhaust.collection.UserInfoExhaustJob$.processRequestEncryption(UserInfoExhaustJob.scala:8) at org.sunbird.core.exhaust.OnDemandExhaustJob.saveRequest(OnDemandExhaustJob.scala:135) at org.sunbird.core.exhaust.OnDemandExhaustJob.saveRequest$(OnDemandExhaustJob.scala:134) at org.sunbird.lms.exhaust.collection.UserInfoExhaustJob$.saveRequest(UserInfoExhaustJob.scala:8) at org.sunbird.core.exhaust.OnDemandExhaustJob$$anon$1.$anonfun$get$1(OnDemandExhaustJob.scala:126) at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:516) at org.sunbird.core.exhaust.OnDemandExhaustJob$$anon$1.get(OnDemandExhaustJob.scala:126) at org.sunbird.core.exhaust.OnDemandExhaustJob$$anon$1.get(OnDemandExhaustJob.scala:124) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)

We are unable to find why providers of apis is empty, when i created a snippet in local and tested i found the following are values should get load.
The list of providers or apis: {providers=[aws-cloudwatch, rackspace-cloudblockstorage-us, rackspace-cloudblockstorage-uk, rackspace-clouddns-us, rackspace-clouddns-uk, rackspace-clouddatabases-us, rackspace-clouddatabases-uk, aws-sqs, aws-sts, aws-route53, ultradns-ws, dynect, rackspace-cloudloadbalancers-uk, rackspace-cloudloadbalancers-us, aws-ec2, gogrid, elastichosts-lon-p, elastichosts-sat-p, elastichosts-lon-b, openhosting-east1, serverlove-z1-man, skalicloud-sdg-my, go2cloud-jhb1, softlayer, rackspace-cloudservers-us, rackspace-cloudservers-uk, google-compute-engine, digitalocean2, profitbricks, packet, azurecompute-arm, aws-s3, azureblob, b2, google-cloud-storage, rackspace-cloudfiles-us, rackspace-cloudfiles-uk], apis=[rackspace-cloudidentity, openstack-keystone, openstack-keystone-3, openstack-cinder, rackspace-clouddns, cloudwatch, openstack-trove, sqs, sts, route53, chef, rackspace-cloudloadbalancers, openstack-nova, openstack-neutron, stub, openstack-nova-ec2, byon, ec2, elasticstack, cloudstack, docker, s3, transient, atmos, openstack-swift, rackspace-cloudfiles, filesystem]}

we need obsrv team help in resolving the issue.

cc: @rhwarrier @AmiableAnil

[sowmya-dixit]

@Hari-stackroute Can you please post the details of debugger logs you have added as part of Analytics Core?

[Hari-stackroute]

@sowmya-dixit i shared the logs in slack, please check.

[sowmya-dixit]

@Hari-stackroute As suggested please test the job after a clean provision, build & deployment since we are not able to reproduce in local and update us back.

[SanthoshVasabhaktula]

@Hari-stackroute - this is open since two months. I presume this is resolved. Locking the conversation