Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Deprecate load_data_frame #417

Merged
merged 9 commits into from
Dec 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions docs/for_pandas.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,19 +37,17 @@ This is useful when dataframe has nullable columns because pandas auto-conversio
Easy to Load DataFrame
----------------------

The :func:`~gokart.task.TaskOnKart.load_data_frame` method is used to load input ``pandas.DataFrame``.
The :func:`~gokart.task.TaskOnKart.load` method is used to load input ``pandas.DataFrame``.

.. code:: python

def requires(self):
return MakeDataFrameTask()

def run(self):
df = self.load_data_frame(required_columns={'colA', 'colB'}, drop_columns=True)
df = self.load()

This allows us to omit ``reset_index`` and ``drop`` when loading. If there is a missing column in an example above, ``AssertionError`` will be raised. This feature is useful for pipelines based on pandas.

Please refer to :func:`~gokart.task.TaskOnKart.load_data_frame`.
Please refer to :func:`~gokart.task.TaskOnKart.load`.


Fail on empty DataFrame
Expand Down
3 changes: 3 additions & 0 deletions docs/task_on_kart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,9 @@ TaskOnKart.load_data_frame

Please refer to :doc:`for_pandas`.

.. warning::
This function is deprecated. Please use :func:`~gokart.task.TaskOnKart.load` instead.


TaskOnKart.fail_on_empty_dump
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
9 changes: 9 additions & 0 deletions gokart/task.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,17 @@
import inspect
import os
import random
import sys
import types
from importlib import import_module
from logging import getLogger
from typing import Any, Callable, Dict, Generator, Generic, Iterable, List, Optional, Set, TypeVar, Union, overload

if sys.version_info < (3, 13):
from typing_extensions import deprecated
else:
from warning import deprecated

import luigi
import pandas as pd
from luigi.parameter import ParameterVisibility
Expand Down Expand Up @@ -311,6 +317,9 @@ def _load(targets):

return _load(self._get_input_targets(target))

@deprecated("""This function is deprecated. use `load` instead.
If you want to specify `required_columns` and `drop_columns`, please extract the columns after loading. ex: `load()[['colA', 'colB']]`
""")
def load_data_frame(
self, target: Union[None, str, TargetOnKart] = None, required_columns: Optional[Set[str]] = None, drop_columns: bool = False
) -> pd.DataFrame:
Expand Down
4 changes: 2 additions & 2 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ APScheduler = "*"
redis = "*"
dill = "*"
backoff = "^2.2.1"
typing-extensions = [
{version = "^4.11.0", python = "<3.13"}
]

[tool.poetry.group.dev.dependencies]
ruff = "*"
Expand Down
Loading