Skip to content

[Bug] shuffle_data_loader seems we have some dependencies issue #22162

@rkooo567

Description

@rkooo567

Search before asking

  • I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

https://buildkite.com/ray-project/periodic-ci/builds/2666#ae2d15dc-84e8-4754-acf7-61993c44533b

File "/home/ray/anaconda3/lib/python3.7/site-packages/pandas/io/parquet.py", line 423, in to_parquet
--
  | **kwargs,
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/pandas/io/parquet.py", line 195, in write
  | table, path_or_handle, compression=compression, **kwargs
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/pyarrow/parquet.py", line 2092, in write_table
  | writer.write_table(table, row_group_size=row_group_size)
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/pyarrow/parquet.py", line 754, in write_table
  | self.writer.write_table(table, row_group_size=row_group_size)
  | File "pyarrow/_parquet.pyx", line 1506, in pyarrow._parquet.ParquetWriter.write_table
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/fsspec/spec.py", line 1460, in write
  | self.flush()
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/fsspec/spec.py", line 1496, in flush
  | self._initiate_upload()
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/s3fs/core.py", line 1841, in _initiate_upload
  | ACL=self.acl,
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/s3fs/core.py", line 1829, in _call_s3
  | return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/fsspec/asyn.py", line 91, in wrapper
  | return sync(self.loop, func, *args, **kwargs)
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/fsspec/asyn.py", line 71, in sync
  | raise return_result
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/fsspec/asyn.py", line 25, in _runner
  | result[0] = await coro
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/s3fs/core.py", line 281, in _call_s3
  | raise err
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/s3fs/core.py", line 261, in _call_s3
  | out = await method(**additional_kwargs)
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/aiobotocore/client.py", line 174, in _make_api_call
  | operation_model, request_dict, request_context)
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/aiobotocore/client.py", line 193, in _make_request
  | return await self._endpoint.make_request(operation_model, request_dict)
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 83, in _send_request
  | exception):
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 216, in _needs_retry
  | caught_exception=caught_exception, request_dict=request_dict)
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/aiobotocore/hooks.py", line 29, in _emit
  | response = handler(**kwargs)
  | File "/home/ray/anaconda3/lib/python3.7/site-packages/botocore/retryhandler.py", line 189, in __call__
  | retries_context = kwargs['request_dict']['context']['retries']
  | KeyError: 'retries'

Versions / Dependencies

N/A

Reproduction script

Run shuffle_data_loader

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Metadata

Metadata

Assignees

Labels

P0Issues that should be fixed in short orderbugSomething that is supposed to be working; but isn't

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions