Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add auto_series_param and custom_series_param #58

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

dkrako
Copy link

@dkrako dkrako commented Feb 28, 2024

The current state of the package only supports 2d image-type input. Theorywise, it shouldn't be a problem to apply the functionality to time series inputs.

This PR adds the class AutoSeriesParam, which is designed according to AutoImageParam, but supports sequences instead of images. This way 1d-support can be integrated neatly without changing any existing code.

It is therefore needed to to setup the dreamer manually this way:

    series_param = AutoSeriesParam(
        length=sequence_length,
        channels=channels,
        standard_deviation=0.01,
        batch_size=batch_size,
        device="cpu",
    )

    series_transforms = transforms.Compose(
        [
            RandomSeriesTranslate(0.1),
            RandomSeriesScale(0.5, 1.2),
        ]
    )

    dreamy_boi = Dreamer(model=model, device='cpu', quiet=False)
    dreamy_boi.set_custom_transforms(series_transforms)

    result = dreamy_boi.render(
        layers=[model.conv1],
        iters=10,
        image_parameter=series_param,
    )

Maybe the series transforms could be setup by default in the dreamer, if the image_parameter is an instance of BaseSeriesParam?

There are still some issues left where I am unsure what to do:

  1. The AutoImageParam class has a normalize() method to normalize the output. It uses imagenet mean and std to apply z-score standardization. There's no such equivalent in time series data due to their heterogeneous nature so we need a way to feed in the mean and std values. The Dreamer class has a set_custom_normalization() method for this use, but the InverseTransform relies on 4d-tensors. I'm not sure how to integrate time series support into this.

  2. The AutoImageParam.postprocess() method applies the function lucid_colorspace_to_rgb(). I don't really understand what it does and I'm clueless on how to treat this in the time series domain.

  3. I left out the implementation of a CustomSeriesParam class for now, but that should come in a follow-up PR.

@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 83.09859% with 36 lines in your changes are missing coverage. Please review.

Project coverage is 83.23%. Comparing base (bdaa9e3) to head (545f28b).
Report is 11 commits behind head on master.

Files Patch % Lines
torch_dreams/series_transforms.py 66.12% 21 Missing ⚠️
torch_dreams/auto_series_param.py 76.19% 15 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #58      +/-   ##
==========================================
- Coverage   91.70%   83.23%   -8.47%     
==========================================
  Files          19       25       +6     
  Lines         747     1038     +291     
==========================================
+ Hits          685      864     +179     
- Misses         62      174     +112     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Mayukhdeb
Copy link
Owner

Hello @dkrako,
Thanks for the PR 🙂 Highly appreciate the detailed notes as well!
I will take a look at this and get back to you ASAP. Thanks again!

@dkrako
Copy link
Author

dkrako commented Mar 13, 2024

I added a CustomSeriesParam class, but I have some issues with the resulting tensor shapes after calling rfft.
They are only half the required length and I'm a bit clueless what to do.

@Mayukhdeb
Copy link
Owner

Hello @dkrako,

Could you please provide more details on the exact use-case for these new components to help me better understand the PR? A Colab notebook or a reproducible snippet would greatly assist in this.

I am eager to merge it as soon as I have a clear understanding of the use-case(s), which will allow me to update the documentation in the README accordingly.

Thank you!

@Mayukhdeb
Copy link
Owner

@dkrako what is the shape of the tensor that your model expects for time series images?
If you can describe the problem in-detail, I can help you debug this on your branch👀

@dkrako
Copy link
Author

dkrako commented Mar 14, 2024

Thanks a lot!

The time series I am referring to are not really 2d-images but multi-channel 1d-series. I myself am working mostly with eye tracking data, where you can have 2 channels (horizontal and vertical component of the eye gaze) but there is a range of different applications you can imagine (EEG data, seismographic data, acceleration sensors, gyroscopes, climate readings for a specific location).

Regardless of the application, the expected data layout in pytorch is NCL (tensorflow expects NLC), where N is the batch dimension, C the channel dimension and L the length (time) dimension. A series with the shape (64, 4, 1000) has a batch size of 64, 4 channels and 1000 time steps.

Analogous to the well known 2d convolution using the pytorch Conv2d class, we use the Conv1d equivalent in the time series domain. For applying the method of feature visualization which torch-dreams is applying, we wouldn't generate any images (NCHW) but time series (NCL).

I'll provide a notebook with a general use-case ASAP.

I will use this single channel toy dataset here for the example: https://github.com/greydanus/mnist1d

@dkrako
Copy link
Author

dkrako commented Mar 14, 2024

here is an example colab notebook I have just created: https://colab.research.google.com/drive/152dNe02_MzSzCxQwAfXY7rtvql9NfPKh?usp=sharing

It loads and plots some data, trains a simple 1d-cnn.
the AutoSeriesParam renders the image successfully, but the CustomSeriesParam has some shape errors.
It think the error is somewhere with using rfft wrong but I don't know how to fix this.

@dkrako
Copy link
Author

dkrako commented Mar 14, 2024

Do you mind giving me permission to run test workflows without further approval, please?
I'd like to increase test coverage but currently you need to approve every single workflow run so I have to run everything locally.

@dkrako
Copy link
Author

dkrako commented Mar 14, 2024

The failing tests are the same error as the one in the colab notebook (CustomSeriesParam).

@dkrako dkrako changed the title add auto_series_param add auto_series_param and custom_series_param Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants