Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark can't connect to Google Cloud Storage #17

Open
WaterKnight1998 opened this issue Jun 26, 2020 · 2 comments
Open

Spark can't connect to Google Cloud Storage #17

WaterKnight1998 opened this issue Jun 26, 2020 · 2 comments

Comments

@WaterKnight1998
Copy link

I have created a notebook using zeppelint. Inside it I am trying to acces a file in GC Storage.

It is getting this error:

Py4JJavaError: An error occurred while calling o72.csv.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found
@WaterKnight1998 WaterKnight1998 changed the title Spark can't connect to Google Clou Storage Spark can't connect to Google Cloud Storage Jun 27, 2020
@WaterKnight1998
Copy link
Author

@dbanda I saw that this chart came from your personal repo. I would like to be able to use google cloud connector and install prophet on spark executors. How can I do it? I tried using my own spark image which contains the connector jar.

However, after adding spark.jars property in zeppelin interpreter. It doesn't get loaded!

@dbanda
Copy link
Contributor

dbanda commented Jul 1, 2020

What spark interpreter are you using on zeppelin? Judging by your error message, Im assuming you are using %spark.pyspark. You have to add your dependencies as the first line in your notebook file. You will also have to make sure you update the path. There is an bug in zeppelin where the paths to python aren't properly updated. Could you share with me your notebook so that I can investigate?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants