Skip to content
This repository has been archived by the owner on Apr 14, 2023. It is now read-only.

Error in getting data while running with databricks connect #64

Open
ashutosh-dwivedi-e3502 opened this issue Aug 19, 2022 · 0 comments

Comments

@ashutosh-dwivedi-e3502
Copy link

While running the template with a local profile and a spark SQL as input for ingest step, following error occurs because there is no active spark session while running with databricks-connect

/Users/asdwivedi/.pyenv/versions/3.8.10/lib/python3.8/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.8) or chardet (5.0.0)/charset_normalizer (2.0.11) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
2022/08/19 15:54:51 INFO mlflow.pipelines.pipeline: Creating MLflow Pipeline 'mlp-regression-template' with profile: 'databricks'
/Users/asdwivedi/.pyenv/versions/3.8.10/lib/python3.8/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.8) or chardet (5.0.0)/charset_normalizer (2.0.11) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/asdwivedi/.pyenv/versions/3.8.10/lib/python3.8/site-packages/mlflow/pipelines/step.py", line 117, in run
    self.step_card = self._run(output_directory=output_directory)
  File "/Users/asdwivedi/.pyenv/versions/3.8.10/lib/python3.8/site-packages/mlflow/pipelines/steps/ingest/__init__.py", line 70, in _run
    self.dataset.resolve_to_parquet(
  File "/Users/asdwivedi/.pyenv/versions/3.8.10/lib/python3.8/site-packages/mlflow/pipelines/steps/ingest/datasets.py", line 588, in resolve_to_parquet
    spark_df = spark_session.sql(self.sql)
AttributeError: 'NoneType' object has no attribute 'sql'
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant