Skip to content

Commit

Permalink
Closes #2177: Update to ak.client.maxTransferBytes (#2180)
Browse files Browse the repository at this point in the history
This PR (closes #2177) attempts to track down all places that needed updating to `ak.client.maxTransferBytes`

Co-authored-by: Pierce Hayes <pierce314159@users.noreply.github.com>
  • Loading branch information
stress-tess and Pierce Hayes authored Feb 28, 2023
1 parent 76d896c commit 8ae978b
Show file tree
Hide file tree
Showing 8 changed files with 18 additions and 18 deletions.
8 changes: 4 additions & 4 deletions arkouda/categorical.py
Original file line number Diff line number Diff line change
Expand Up @@ -344,12 +344,12 @@ def to_ndarray(self) -> np.ndarray:
Notes
-----
The number of bytes in the array cannot exceed ``arkouda.maxTransferBytes``,
The number of bytes in the array cannot exceed ``ak.client.maxTransferBytes``,
otherwise a ``RuntimeError`` will be raised. This is to protect the user
from overflowing the memory of the system on which the Python client
is running, under the assumption that the server is running on a
distributed system with much more memory than the client. The user
may override this limit by setting ak.maxTransferBytes to a larger
may override this limit by setting ak.client.maxTransferBytes to a larger
value, but proceed with caution.
"""
if self.categories.size > self.codes.size:
Expand All @@ -376,12 +376,12 @@ def to_list(self) -> List:
Notes
-----
The number of bytes in the Categorical cannot exceed ``arkouda.maxTransferBytes``,
The number of bytes in the Categorical cannot exceed ``ak.client.maxTransferBytes``,
otherwise a ``RuntimeError`` will be raised. This is to protect the user
from overflowing the memory of the system on which the Python client
is running, under the assumption that the server is running on a
distributed system with much more memory than the client. The user
may override this limit by setting ak.maxTransferBytes to a larger
may override this limit by setting ak.client.maxTransferBytes to a larger
value, but proceed with caution.
"""
return self.to_ndarray().tolist()
Expand Down
8 changes: 4 additions & 4 deletions arkouda/pdarraycreation.py
Original file line number Diff line number Diff line change
Expand Up @@ -179,11 +179,11 @@ def array(
Notes
-----
The number of bytes in the input array cannot exceed `arkouda.maxTransferBytes`,
The number of bytes in the input array cannot exceed `ak.client.maxTransferBytes`,
otherwise a RuntimeError will be raised. This is to protect the user
from overwhelming the connection between the Python client and the arkouda
server, under the assumption that it is a low-bandwidth connection. The user
may override this limit by setting ak.maxTransferBytes to a larger value,
may override this limit by setting ak.client.maxTransferBytes to a larger value,
but should proceed with caution.
If the pdrray or ndarray is of type U, this method is called twice recursively
Expand Down Expand Up @@ -245,7 +245,7 @@ def array(
if nbytes > maxTransferBytes:
raise RuntimeError(
f"Creating pdarray would require transferring {nbytes} bytes, which exceeds "
f"allowed transfer size. Increase ak.maxTransferBytes to force."
f"allowed transfer size. Increase ak.client.maxTransferBytes to force."
)
encoded_np = np.array(encoded, dtype=np.uint8)
rep_msg = generic_msg(
Expand Down Expand Up @@ -280,7 +280,7 @@ def array(
size = a.size
if (size * a.itemsize) > maxTransferBytes:
raise RuntimeError(
"Array exceeds allowed transfer size. Increase ak.maxTransferBytes to allow"
"Array exceeds allowed transfer size. Increase ak.client.maxTransferBytes to allow"
)
# Pack binary array data into a bytes object with a command header
# including the dtype and size. If the server has a different byteorder
Expand Down
10 changes: 5 additions & 5 deletions arkouda/strings.py
Original file line number Diff line number Diff line change
Expand Up @@ -1730,12 +1730,12 @@ def to_ndarray(self) -> np.ndarray:
Notes
-----
The number of bytes in the array cannot exceed ``arkouda.maxTransferBytes``,
The number of bytes in the array cannot exceed ``ak.client.maxTransferBytes``,
otherwise a ``RuntimeError`` will be raised. This is to protect the user
from overflowing the memory of the system on which the Python client
is running, under the assumption that the server is running on a
distributed system with much more memory than the client. The user
may override this limit by setting ak.maxTransferBytes to a larger
may override this limit by setting ak.client.maxTransferBytes to a larger
value, but proceed with caution.
See Also
Expand Down Expand Up @@ -1778,12 +1778,12 @@ def to_list(self) -> list:
Notes
-----
The number of bytes in the array cannot exceed ``arkouda.maxTransferBytes``,
The number of bytes in the array cannot exceed ``ak.client.maxTransferBytes``,
otherwise a ``RuntimeError`` will be raised. This is to protect the user
from overflowing the memory of the system on which the Python client
is running, under the assumption that the server is running on a
distributed system with much more memory than the client. The user
may override this limit by setting ak.maxTransferBytes to a larger
may override this limit by setting ak.client.maxTransferBytes to a larger
value, but proceed with caution.
See Also
Expand Down Expand Up @@ -1843,7 +1843,7 @@ def _comp_to_ndarray(self, comp: str) -> np.ndarray:
# Guard against overflowing client memory
if array_bytes > maxTransferBytes:
raise RuntimeError(
"Array exceeds allowed size for transfer. Increase client.maxTransferBytes to allow"
"Array exceeds allowed size for transfer. Increase ak.client.maxTransferBytes to allow"
)
# The reply from the server will be a bytes object
rep_msg = generic_msg(
Expand Down
2 changes: 1 addition & 1 deletion pydoc/usage/arrayview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,6 @@ so ``n[:, :]`` gets indices ``[0,0], [0,1], [1,0], [1,1]`` whereas ``n[[0,1],[0,

Iteration
===========
Iterating directly over an ``ArrayView`` with ``for x in array_view`` is not supported to discourage transferring all array data from the arkouda server to the Python client. To force this transfer, use the ``to_ndarray`` function to return the ``ArrayView`` as a ``numpy.ndarray``. This transfer will raise an error if it exceeds the byte limit defined in ``arkouda.maxTransferBytes``.
Iterating directly over an ``ArrayView`` with ``for x in array_view`` is not supported to discourage transferring all array data from the arkouda server to the Python client. To force this transfer, use the ``to_ndarray`` function to return the ``ArrayView`` as a ``numpy.ndarray``. This transfer will raise an error if it exceeds the byte limit defined in ``ak.client.maxTransferBytes``.

.. autofunction:: arkouda.ArrayView.to_ndarray
2 changes: 1 addition & 1 deletion pydoc/usage/categorical.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,6 @@ Arkouda ``Categorical`` objects support all operations that ``Strings`` support,
Iteration
=========

Iterating directly over a ``Categorical`` with ``for x in categorical`` is not supported to discourage transferring all the ``Categorical`` object's data from the arkouda server to the Python client since there is almost always a more array-oriented way to express an iterator-based computation. To force this transfer, use the ``to_ndarray`` function to return the ``categorical`` as a ``numpy.ndarray``. This transfer will raise an error if it exceeds the byte limit defined in ``arkouda.maxTransferBytes``.
Iterating directly over a ``Categorical`` with ``for x in categorical`` is not supported to discourage transferring all the ``Categorical`` object's data from the arkouda server to the Python client since there is almost always a more array-oriented way to express an iterator-based computation. To force this transfer, use the ``to_ndarray`` function to return the ``categorical`` as a ``numpy.ndarray``. This transfer will raise an error if it exceeds the byte limit defined in ``ak.client.maxTransferBytes``.

.. autofunction:: arkouda.Categorical.to_ndarray
2 changes: 1 addition & 1 deletion pydoc/usage/dataframe.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Data within the above objects can be of the types below. Please Note - Not all l
Iteration
=========

Iterating directly over a ``DataFrame`` with ``for x in df`` is not recommended. Doing so is discouraged because it requires transferring all array data from the arkouda server to the Python client since there is almost always a more array-oriented way to express an iterator-based computation. To force this transfer, use the ``to_pandas`` function to return the ``DataFrame`` as a ``pandas.DataFrame``. This transfer will raise an error if it exceeds the byte limit defined in ``arkouda.maxTransferBytes``.
Iterating directly over a ``DataFrame`` with ``for x in df`` is not recommended. Doing so is discouraged because it requires transferring all array data from the arkouda server to the Python client since there is almost always a more array-oriented way to express an iterator-based computation. To force this transfer, use the ``to_pandas`` function to return the ``DataFrame`` as a ``pandas.DataFrame``. This transfer will raise an error if it exceeds the byte limit defined in ``ak.client.maxTransferBytes``.

.. autofunction:: arkouda.DataFrame.to_pandas

Expand Down
2 changes: 1 addition & 1 deletion pydoc/usage/pdarray.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ The ``pdarray`` class supports most Python special methods, including arithmetic
Iteration
=========

Iterating directly over a ``pdarray`` with ``for x in array`` is not supported to discourage transferring all array data from the arkouda server to the Python client since there is almost always a more array-oriented way to express an iterator-based computation. To force this transfer, use the ``to_ndarray`` function to return the ``pdarray`` as a ``numpy.ndarray``. This transfer will raise an error if it exceeds the byte limit defined in ``arkouda.maxTransferBytes``.
Iterating directly over a ``pdarray`` with ``for x in array`` is not supported to discourage transferring all array data from the arkouda server to the Python client since there is almost always a more array-oriented way to express an iterator-based computation. To force this transfer, use the ``to_ndarray`` function to return the ``pdarray`` as a ``numpy.ndarray``. This transfer will raise an error if it exceeds the byte limit defined in ``ak.client.maxTransferBytes``.

.. autofunction:: arkouda.pdarray.to_ndarray

Expand Down
2 changes: 1 addition & 1 deletion pydoc/usage/strings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Because strings are a variable-width data type, and because of the way Arkouda r
I/O
===========

Arrays of strings can be transferred between the Arkouda client and server using the ``arkouda.array`` and ``Strings.to_ndarray`` functions (see :ref:`IO-label`). The former converts a Python list or NumPy ``ndarray`` of strings to an Arkouda ``Strings`` object, whereas the latter converts an Arkouda ``Strings`` object to a NumPy ``ndarray``. As with numeric arrays, if the size of the data exceeds the threshold set by ``arkouda.maxTransferBytes``, the client will raise an exception.
Arrays of strings can be transferred between the Arkouda client and server using the ``arkouda.array`` and ``Strings.to_ndarray`` functions (see :ref:`IO-label`). The former converts a Python list or NumPy ``ndarray`` of strings to an Arkouda ``Strings`` object, whereas the latter converts an Arkouda ``Strings`` object to a NumPy ``ndarray``. As with numeric arrays, if the size of the data exceeds the threshold set by ``ak.client.maxTransferBytes``, the client will raise an exception.

Arkouda currently only supports the HDF5 file format for disk-based I/O. In order to read an array of strings from an HDF5 file, the strings must be stored in an HDF5 ``group`` containing two datasets: ``segments`` (an integer array corresponding to ``offsets`` above) and ``values`` (a ``uint8`` array corresponding to ``bytes`` above). See :ref:`data-preprocessing-label` for more information and guidelines.

Expand Down

0 comments on commit 8ae978b

Please sign in to comment.