Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a fix for custom code tokenizers in pipelines #32300

Merged
merged 10 commits into from
Aug 27, 2024
Merged

Add a fix for custom code tokenizers in pipelines #32300

merged 10 commits into from
Aug 27, 2024

Commits on Aug 23, 2024

  1. Add a fix for the case when tokenizers are passed as a string

    @Rocketknight1
    Rocketknight1 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    2ed3ed9 View commit details
    Browse the repository at this point in the history

  2. Support image processors and feature extractors as well

    @Rocketknight1
    Rocketknight1 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    20e7c31 View commit details
    Browse the repository at this point in the history

  3. Reverting load_feature_extractor and load_image_processor

    @Rocketknight1
    Rocketknight1 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    26abe89 View commit details
    Browse the repository at this point in the history

  4. Add test

    @Rocketknight1
    Rocketknight1 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    d117685 View commit details
    Browse the repository at this point in the history

  5. Test is torch-only

    @Rocketknight1
    Rocketknight1 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    43d4a50 View commit details
    Browse the repository at this point in the history

  6. Add tests for preprocessors and feature extractors and move test

    @Rocketknight1
    Rocketknight1 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    d8c30c4 View commit details
    Browse the repository at this point in the history

  7. Extremely experimental fix

    @Rocketknight1
    Rocketknight1 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    d0f5eb6 View commit details
    Browse the repository at this point in the history

  8. Revert that change, wrong branch!

    @Rocketknight1
    Rocketknight1 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    5833c98 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Typo!

    @Rocketknight1
    Rocketknight1 committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    d0b2356 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Split tests

    @Rocketknight1
    Rocketknight1 committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    d77b39d View commit details
    Browse the repository at this point in the history