Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hackweek(kafkatasks): add kafkatasks postgres model #76394

Draft
wants to merge 56 commits into
base: master
Choose a base branch
from
Draft

hackweek(kafkatasks): add kafkatasks postgres model #76394

wants to merge 56 commits into from

Commits on Aug 19, 2024

  1. Empty files!

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    5b4fed7 View commit details
    Browse the repository at this point in the history

  2. We'll need migrations probably

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    9bc027a View commit details
    Browse the repository at this point in the history

  3. Start sketching registry/namespace and decorator

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    06ef2f8 View commit details
    Browse the repository at this point in the history

  4. Get task decorator working

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    f828edb View commit details
    Browse the repository at this point in the history

  5. Janky message production

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    c5e8ee7 View commit details
    Browse the repository at this point in the history

  6. add kafkatasks model

    @victoria-yining-huang
    victoria-yining-huang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    6d492df View commit details
    Browse the repository at this point in the history

  7. 🛠️ apply pre-commit fixes

    @getsantry
    getsantry[bot] committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    8b4eb85 View commit details
    Browse the repository at this point in the history

  8. restore server config

    @victoria-yining-huang
    victoria-yining-huang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    1049e7f View commit details
    Browse the repository at this point in the history

  9. remove src/sentry/models/kafkatasks.py

    @victoria-yining-huang
    victoria-yining-huang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    454b2ec View commit details
    Browse the repository at this point in the history

  10. add src/sentry/taskworker/models.py

    @victoria-yining-huang
    victoria-yining-huang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    40263ae View commit details
    Browse the repository at this point in the history

  11. Add module to INSTALLED_APPS

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    fb9e7ae View commit details
    Browse the repository at this point in the history

  12. Adds base service skeleton, sample impl for get and set tasks

    @GabeVillalobos
    GabeVillalobos committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    4880129 View commit details
    Browse the repository at this point in the history

  13. Start sketching out retry and retry state

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    1c0a8e5 View commit details
    Browse the repository at this point in the history

  14. Fix mistakes

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    8f3761d View commit details
    Browse the repository at this point in the history

  15. Adds processing state to mark tasks as inaccessible to other workers,… 

    … adds transaction
    @GabeVillalobos
    GabeVillalobos committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    a586417 View commit details
    Browse the repository at this point in the history

  16. add migration

    @victoria-yining-huang
    victoria-yining-huang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    511630a View commit details
    Browse the repository at this point in the history

  17. Fixes for new schema changes

    @GabeVillalobos
    GabeVillalobos committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    b02c45b View commit details
    Browse the repository at this point in the history

  18. Fix typos

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    15ab2b0 View commit details
    Browse the repository at this point in the history

  19. Sketch in deadline and idempotent task attributes

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    eb2a78e View commit details
    Browse the repository at this point in the history

  20. Sketch out worker command line interface

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    4934bba View commit details
    Browse the repository at this point in the history

  21. Rework initial PendingTask migration for primary key

    @GabeVillalobos
    GabeVillalobos committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    7ff2496 View commit details
    Browse the repository at this point in the history

  22. Adds namespace column to PendingTasks

    @GabeVillalobos
    GabeVillalobos committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    7caf551 View commit details
    Browse the repository at this point in the history

  23. start sketching in the worker

    @markstory
    markstory committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    d871941 View commit details
    Browse the repository at this point in the history

  24. 🛠️ apply pre-commit fixes

    @getsantry
    getsantry[bot] committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    c5db13b View commit details
    Browse the repository at this point in the history

  25. Adds stub command for task processing

    @GabeVillalobos
    GabeVillalobos committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    909985e View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Adds test CLI command to run task worker

    @GabeVillalobos
    GabeVillalobos committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    35d053e View commit details
    Browse the repository at this point in the history

  2. Adds actual connected namespace example

    @GabeVillalobos
    GabeVillalobos committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    4de2859 View commit details
    Browse the repository at this point in the history

  3. add consumer code

    @victoria-yining-huang
    victoria-yining-huang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    84c21ca View commit details
    Browse the repository at this point in the history

  4. Sketch in retry flow a bit more

    @markstory
    markstory committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    4a38086 View commit details
    Browse the repository at this point in the history

  5. Add some logging

    @markstory
    markstory committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    8c09371 View commit details
    Browse the repository at this point in the history

  6. Fix logging

    @markstory
    markstory committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    8958166 View commit details
    Browse the repository at this point in the history

  7. update some PendingTasks fields based on the example message, and add… 

    … first try of flushing to db
    @victoria-yining-huang
    victoria-yining-huang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    177cedc View commit details
    Browse the repository at this point in the history

  8. Allow unsafe migration - hackweek!

    @markstory
    markstory committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    b7118df View commit details
    Browse the repository at this point in the history

  9. remove cli from consumer, and remove old migrations

    @victoria-yining-huang
    victoria-yining-huang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    1d2de31 View commit details
    Browse the repository at this point in the history

  10. add latest migration

    @victoria-yining-huang
    victoria-yining-huang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    6e82cad View commit details
    Browse the repository at this point in the history

  11. Adds retry logic and updating for tasks

    @GabeVillalobos
    GabeVillalobos committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    c19ebf8 View commit details
    Browse the repository at this point in the history

  12. Adds sample task and namespace

    @GabeVillalobos
    GabeVillalobos committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    a217c7b View commit details
    Browse the repository at this point in the history

  13. refactor

    @victoria-yining-huang
    victoria-yining-huang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    f07f8a2 View commit details
    Browse the repository at this point in the history

  14. Flattens retry fields on model

    @GabeVillalobos
    GabeVillalobos committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    6d89742 View commit details
    Browse the repository at this point in the history

  15. Pushing work for live E2E test, with retry support

    @GabeVillalobos
    GabeVillalobos committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    9c3fd22 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. Make sentry run consumer taskworker --consumer-group hackweek work 

    - Fix warnings and TODOs related to datetimes.
- Fix startup error in strategy_factory.
- Simplified logging configuration
    @markstory
    markstory committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    41f15fc View commit details
    Browse the repository at this point in the history

  2. add offset but idk if it's right

    @victoria-yining-huang
    victoria-yining-huang committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    72e7ab6 View commit details
    Browse the repository at this point in the history

  3. Sketch in processing timeout, retries and deadlines

    @markstory
    markstory committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    1d90a9a View commit details
    Browse the repository at this point in the history

  4. Improve logging

    @markstory
    markstory committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    18fded4 View commit details
    Browse the repository at this point in the history

  5. Fix mistakes

    @markstory
    markstory committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    4805c19 View commit details
    Browse the repository at this point in the history

  6. Get retries working

    @markstory
    markstory committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    43bab4d View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Use config tooling better

    @markstory
    markstory committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    4108408 View commit details
    Browse the repository at this point in the history

  2. Cleanup

    @markstory
    markstory committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    b7c762e View commit details
    Browse the repository at this point in the history

  3. Rename cli var

    @GabeVillalobos
    GabeVillalobos committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    52c3691 View commit details
    Browse the repository at this point in the history

  4. baby refactors for retry state conversion

    @GabeVillalobos
    GabeVillalobos committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    e499899 View commit details
    Browse the repository at this point in the history

  5. Updates strategy, config to use protobufs with kafka tasks

    @GabeVillalobos
    GabeVillalobos committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    a2b449d View commit details
    Browse the repository at this point in the history

  6. Fixes some warnings around datetime setting

    @GabeVillalobos
    GabeVillalobos committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    3064fb9 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. Merge branch 'master' into hackweek-kafkatasks

    @markstory
    markstory committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    8273cd0 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. create pending task store

    @enochtangg
    enochtangg committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    8d78757 View commit details
    Browse the repository at this point in the history

  2. fanout cpu intensive work in RunTaskWithMultiprocessing

    @john-z-yang
    john-z-yang committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c4ab4ea View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. fix shutdown and reduce inital value

    @john-z-yang
    john-z-yang committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    51caaea View commit details
    Browse the repository at this point in the history