Replies: 1 comment
-
what is your |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am receiving an error when using pyarrows ParquetDataset() and S3FSFileSystem()
Some Background:
AWS Lambda Function - Runtime 3.7
Below is list of packages and versions being used within the lambda function.
requirements-prod.txt
The Code:
s3_uri = 's3://' + bucket + '/' + key df = pq.ParquetDataset(s3_uri, filesystem=fs) \ .read_pandas() \ .to_pandas()
The Error :
AttributeError: 'AioClientCreator' object has no attribute '_register_lazy_block_unknown_fips_pseudo_regions'
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 86, in lambda_handler
df = utils.load_exceptions_dataframe(bucket, key, fs)
File "/var/task/utils.py", line 74, in load_exceptions_dataframe
df = pq.ParquetDataset(s3_uri, filesystem=fs)
File "/opt/python/pyarrow/parquet.py", line 1272, in init
open_file_func=partial(_open_dataset_file, self._metadata)
File "/opt/python/pyarrow/parquet.py", line 1446, in _make_manifest
if _is_path_like(path_or_paths) and fs.isdir(path_or_paths):
File "/opt/python/fsspec/asyn.py", line 88, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/opt/python/fsspec/asyn.py", line 69, in sync
raise result[0]
File "/opt/python/fsspec/asyn.py", line 25, in _runner
result[0] = await coro
File "/opt/python/s3fs/core.py", line 1161, in _isdir
return bool(await self._lsdir(path))
File "/opt/python/s3fs/core.py", line 559, in _lsdir
await self.set_session()
File "/opt/python/s3fs/core.py", line 403, in set_session
self._s3 = await s3creator.aenter()
File "/opt/python/aiobotocore/session.py", line 37, in aenter
self._client = await self._coro
File "/opt/python/aiobotocore/session.py", line 125, in _create_client
client_config=config, api_version=api_version)
File "/opt/python/aiobotocore/client.py", line 46, in create_client
self._register_lazy_block_unknown_fips_pseudo_regions(service_client)
Any idea on a possible solution?
Beta Was this translation helpful? Give feedback.
All reactions