Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Materialize train test #472

Merged
merged 16 commits into from
Dec 27, 2024
Merged

Conversation

HoustonJ2013
Copy link
Contributor

@HoustonJ2013 HoustonJ2013 commented Dec 22, 2024

Closes #470.

Add the col_stats in the dataset materialize function, allowing the column stats calculated from train dataset to be used in the test data materialization.

train_df = ..
test_df = ..
train_dataset = Dataset(train_df, 
                 col_to_stype=col_to_stype, 
                 target_col="target")
train_dataset.materialize()
test_dataset = Dataset(test_df, 
                 col_to_stype=col_to_stype, 
                 target_col="target")
test_dataset.materialize(col_stats=train_dataset.col_stats)

Copy link
Contributor

@weihua916 weihua916 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! this makes sense. can we do the following?

  • Add basic validation logic and raise error if col stats is not appropriate.
  • Add a unit test.

torch_frame/data/dataset.py Show resolved Hide resolved
@weihua916 weihua916 merged commit 896919e into pyg-team:master Dec 27, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dataset materialize for train and test data separately
2 participants