Easily Speed up Pandas with Modin

[shoukewei]

I just wrote a blog article about speeding up Pandas with Modin, in which it compares the speed of reading multiple data files with Pandas, Modin with Ray and Modin with Dask. I hope this article could helful to some of you guys.

[YarShev]

Hi, @shoukewei. That is a great article. I think it will be definetely useful for plenty of Modin users. Thanks for writing it. I would also propose to you to pay attention on Modin on unidist with MPI backend. Though it may be slower than Ray or Dask on certain cases, exploring the operations which are slower with MPI would help us a lot in order to focus on the concrete cases.

[shoukewei]

Hi, @YarShev, thanks for positive feedback. Actually, I tried unidist with MPI backend, but there is error. I have not found a solution. So I have not included in the article. The error goes as follows:

`UserWarning: unidist execution environment not yet initialized. Initializing...
To remove this warning, run the following python code before doing dataframe operations:

        import unidist
        unidist.init()

---

Exception Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_16340\3810847768.py in
1 start = time.time()
----> 2 ud_df = ud.concat(map(ud.read_csv, files))
3 end = time.time()
4 ud_duration = end - start
5 print("Time to read data with modin and dask: {} seconds".format(round(ud_duration, 3)))

~\anaconda3\lib\site-packages\modin\logging\logger_decorator.py in run_and_log(*args, **kwargs)
126 """
127 if LogMode.get() == "disable":
--> 128 return obj(*args, **kwargs)
129
130 logger = get_logger()

~\anaconda3\lib\site-packages\modin\pandas\general.py in concat(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)
447 list_of_objs = list(objs.values())
448 else:
--> 449 list_of_objs = list(objs)
450 if len(list_of_objs) == 0:
451 raise ValueError("No objects to concatenate")

~\anaconda3\lib\site-packages\modin\logging\logger_decorator.py in run_and_log(*args, **kwargs)
126 """
127 if LogMode.get() == "disable":
--> 128 return obj(*args, **kwargs)
129
130 logger = get_logger()

~\anaconda3\lib\site-packages\modin\pandas\io.py in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, error_bad_lines, warn_bad_lines, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options)
216 f_locals.pop("mangle_dupe_cols", None)
217 kwargs = {k: v for k, v in f_locals.items() if k in _pd_read_csv_signature}
--> 218 return _read(**kwargs)
219
220

~\anaconda3\lib\site-packages\modin\pandas\io.py in _read(**kwargs)
83 modin.pandas.DataFrame
84 """
---> 85 Engine.subscribe(_update_engine)
86 from modin.core.execution.dispatching.factories.dispatcher import FactoryDispatcher
87

~\anaconda3\lib\site-packages\modin\config\pubsub.py in subscribe(cls, callback)
215 """
216 cls._subs.append(callback)
--> 217 callback(cls)
218
219 @classmethod

~\anaconda3\lib\site-packages\modin\pandas_init_.py in _update_engine(publisher)
150 from modin.core.execution.unidist.common import initialize_unidist
151
--> 152 initialize_unidist()
153 elif publisher.get() == "Cloudray":
154 from modin.experimental.cloud import get_connection

~\anaconda3\lib\site-packages\modin\core\execution\unidist\common\utils.py in initialize_unidist()
46 )
47
---> 48 unidist.init()
49
50 num_cpus = sum(v["CPU"] for v in unidist.cluster_resources().values())

~\anaconda3\lib\site-packages\unidist\api.py in init()
90 Ray backend is used by default.
91 """
---> 92 init_backend()
93
94

~\anaconda3\lib\site-packages\unidist\core\base\utils.py in init_backend()
41 from unidist.core.backends.mpi.utils import initialize_mpi
42
---> 43 initialize_mpi()
44 backend_cls = MPIBackend()
45 elif backend_name == BackendName.MP:

~\anaconda3\lib\site-packages\unidist\core\backends\mpi\utils.py in initialize_mpi()
10 from unidist.core.backends.mpi.core import init
11
---> 12 init()

~\anaconda3\lib\site-packages\unidist\core\backends\mpi\core\controller\api.py in init()
104 info.Set("hosts", hosts)
105
--> 106 intercomm = MPI.COMM_SELF.Spawn(
107 sys.executable,
108 args,

mpi4py\MPI\Comm.pyx in mpi4py.MPI.Intracomm.Spawn()

Exception: Internal MPI error!, error stack:
MPI_Comm_spawn(cmd="C:\Users\shouk\anaconda3\python.exe", argv=0x000001DC2B0CB8D0, maxprocs=17, info=0x9c000000, root=0, MPI_COMM_SELF, intercomm=0x000001DC2B0CB980, errors=0x0000000000000000) failed
Internal MPI error! FAILspawn not supported without process manager`

[YarShev]

@shoukewei, most likely, you ran your code just with python script.py, but if you look closer at the instruction, you will notice that the script should be run as follows.

mpiexec -n 1 python script.py

Please let me know if it is not obvious from the instruction.

[shoukewei]

Hi Laroslav, thanks for your solution. It works using `mpiexec -n 1 python script.py` in terminal. It uses only 13.239 seconds, which is faster than Dask and Ray. But originally, I worked in Jupuyter notebook, and the error came from running in the Jupyter notebook. Do you have any idea about the error in Jupyter notebook. Thanks Shouke

…

On Mon, Feb 13, 2023 at 12:34 PM Iaroslav Igoshev ***@***.***> wrote: @shoukewei <https://github.com/shoukewei>, most likely, you ran your code just with python script.py, but if you look closer at the instruction <https://modin.readthedocs.io/en/latest/development/using_pandas_on_unidist.html>, you will notice that the script should be run as follows. mpiexec -n 1 python script.py Please let me know if it is not obvious from the instruction. — Reply to this email directly, view it on GitHub <#5646 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASFUUJ27C3BE7WGDYE7WRM3WXKLFVANCNFSM6AAAAAAUZALX3E> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[YarShev]

Using MPI backend in interactive environment such as jupyter notebook is a bit tricky. We have an instruction on how to run Modin with unidist on MPI backend in jupyter. You likely want to read it through - https://github.com/modin-project/modin/tree/master/examples/tutorial/jupyter/execution/pandas_on_unidist. Please let me know if you are still experiencing an issue after reading the instruction.

[shoukewei]

Thanks for all information.