Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Anomaly Detection: Unit and Integration Tests #1146

Merged
merged 54 commits into from
Sep 13, 2024
Merged

Anomaly Detection: Unit and Integration Tests #1146

merged 54 commits into from
Sep 13, 2024

Conversation

aayush-se
Copy link
Member

  • Generated Synthetic timeseries (cyclical, cyclical with anomaly, noisy, noisy with anomaly) to use for unit and integration tests

    • Timeseries generated indata-analysis GitHub repo

  • Wrote unit tests for the following modules in anomaly_detection:

    • Detectors
      • anomaly_detectors.py
      • mp_config.py
      • mp_scorers.py
      • mp_utils.py
      • normalizers.py
      • window_size_selctors.py
    • Models
      • converters.py
      • timeseries_anomalies.py
      • timeseries.py

Wrote integration tests for the store and detect endpoints for batch, stream, and combo pipelines

NOTE: Wanted to submit this PR to start getting some feedback as there are a lot of changes. As of submitting this PR, the following unit/integration tests do not work and I wanted to get some feedback:

  • test_detect_anomalies_online in test_anomaly_detection.py: Was unsure how to handle database changes during integration test
  • test_detect in test_anomaly_detectors.py: was running into shape errors with the matrix profile
  • test_incorrect_padding in test_mp_utils.py: this one is fixed as of this PR so needs to be merged into main to reflect changes

@aayush-se
Add basic tests for batch detection
2d1f028
@aayush-se
Create files for detector tests
1ba1d60
@aayush-se
Create unit tests for normalizing
e316a05
@aayush-se
Stub out test classes and functions
3c7067e
@aayush-se
Stub out test classes and functions
7735931
@aayush-se
Tests for window size
218d4b2
@aayush-se
Complete test for mp_utils
b151e77
@aayush-se
Tests for batch scoring
eb0cd64
@aayush-se
Create synthetic data for testing
6e5e418
@aayush-se
Test batch scoring with synthetic data
b0d6964
@aayush-se
Include more synthetic data and complete batch scoring test
a3211d3
@aayush-se
Window size selector test
b116110
@aayush-se
Test for incorrect padding
058010c
@aayush-se
Delete old sample time series
a8dd6bf
@aayush-se
Test stream score against synthetic data
be8a866
@aayush-se
Unit tests for batch and streaming
eb36f9d
@aayush-se
Unit test for converter
e896dcc
@aayush-se
Tests for get_anomaly_algo_data
234caf2
@aayush-se
Unit test for get_anomaly_algo_data
054ce50
@aayush-se
pre-commit
a898f71
@aayush-se
Creat util for reading synthetic timeseries json
50e08ab
@aayush-se
Integration tests for store and detect endpoints
ea5d369
@aayush-se
Update synthetic data to include timestamps
fd13206
@aayush-se
Deleted old synthetic files
235b945
@aayush-se
Remove incorrect change
94b95af
@aayush-se
Refactor using test_utils helper for reading time series
55f2aa6
@aayush-se aayush-se requested a review from a team September 10, 2024 06:10
@aayush-se aayush-se changed the title Anomaly detection/unit tests Anomaly Detection: Unit and Integration Tests Sep 10, 2024
corps
corps reviewed Sep 10, 2024
View reviewed changes
tests/seer/anomaly_detection/test_anomaly_detection.py Outdated
)

# Successful
self.assertEqual(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert response == X

sentry-autofix bot and others added 15 commits September 10, 2024 11:44
@sentry-autofix @jennmueng @aayush-se
🤖 Update ThreadDetails Model with Optional ID (#1110)
6765e91 
👋 Hi there! This PR was automatically generated by Autofix 🤖

This fix was triggered by Jenn Mueng

Fixes
[SEER-9R](https://sentry.io/organizations/sentry/issues/5734622145/)

This change updates the `ThreadDetails` model in the
`src/seer/automation/models.py` file. Specifically, it modifies the `id`
field to be optional by setting its default value to `None`.
Additionally, a blank line has been added before the class definition
for better readability.

If you have any questions or feedback for the Sentry team about this
fix, please email [autofix@sentry.io](mailto:autofix@sentry.io) with the
Run ID (see below).

### 🤓 Stats for the nerds:
Run ID: **738**
Prompt tokens: **97008**
Completion tokens: **5084**
Total tokens: **102092**

---------

Co-authored-by: sentry-autofix-experimental[bot] <157164994+sentry-autofix-experimental[bot]@users.noreply.github.com>
Co-authored-by: jenn.muengtaweepongsa <jenn.muengtaweepongsa@sentry.io>
@roaga @aayush-se
fix(issue summary) Make breadcrumb fields optional to prevent validat…
ab97fb3 
…ion errors (#1111)

Some issues don't have all the breadcrumbs fields, so we make them
optional now.
@jennmueng @aayush-se
feat(security): Add API bearer token auth via JWT (#1063)
93c1f1c 
Mainly for the staging deployment in `ml-ai`

Introduces an `ENFORCE_API_AUTH` env variable that will enforce either
the rpc secret signing or the bearer token.
@roaga @aayush-se
(autofix): Pass issue summary to autofix, store raw summary output in…
5c1356f 
…stead of response (#1108)

This PR includes:

1. Changing the issue summary storage logic to store the raw output
`IssueSummary` from GPT, rather than the trimmed
`SummarizeIssueResponse` as before. The extra info, such as
`reason_step_by_step` is useful.
2. Pass the issue summary to Autofix as additional context. (part of
#998) Remove pre-processing step on event details as well. Improvement
in evals: 0.48/0.52 -> 0.51/0.55
3. Allow optional issue summary in `AutofixRequest` to support new eval
dataset
@roaga @aayush-se
fix(autofix): Add cascade delete to Autofix TTL, add Issue Summary TTL (
64605c2 
#1112)

The job to delete old Autofix runs failed when there was a PR mapped to
it. This PR adds a cascade delete so that it works.
https://sentry.sentry.io/issues/5764875740/?project=6178942&query=is:unresolved%20issue.priority:%5Bhigh,%20medium%5D&statsPeriod=14d&stream_index=4

It also adds on to the job to delete old Issue Summaries for the TTL.

Issue #1116
@roaga @aayush-se
Switch coding agent to claude
842272f
@jennmueng @aayush-se
Revert "Switch coding agent to claude"
01c6e96 
This reverts commit 4cb53fd.
@ram-senth @aayush-se
Fix for SEER-A9 (#1121)
6d74264 
Need to clean up existing test data before applying db migration to fix
[SEER-A9](https://sentry.sentry.io/issues/5797165799)
@aayush-se
fix(anomaly detection): handle overflow for SuSS algorithm (#1118)
3203272 
- Set up upper bound to avoid `window_size` overflow when hitting
`_score()` (upper limit is np.integer max size which is 2**63 - 1)
- Write basic unit tests to confirm functionality
@roaga @aayush-se
feat(autofix): Switch coding agent to claude (#1122)
cb77d28 
Switches coding agent to Claude (#951)

Evals: 0.51/0.55 -> 0.54/0.61 (huge bump!)

Note: don't deploy this until Sept 5, 2024
@jennmueng @aayush-se
use gcp project env vars (#1123)
fef0ca7
@jennmueng @aayush-se
fix bad var (#1125)
752ceb9
@jennmueng @aayush-se
fix: revert claude agent + gcp env var for now (#1126)
85a2bd5
@aayush-se
Convert asserts to pytest equivalents
6607715
@aayush-se
Revert "Add values for dummy variables and assert mocks called"
e93cecc 
This reverts commit 365fcd5.
@aayush-se
Copy link
Member Author

Reverted changes -- improperly merged from main. Will updated changes from comments again.

@aayush-se aayush-se merged commit 84ed952 into main Sep 13, 2024
11 checks passed
@aayush-se aayush-se deleted the anomaly-detection/unit-tests branch September 13, 2024 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@corps corps corps left review comments

@ram-senth ram-senth ram-senth approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@aayush-se @corps @ram-senth @roaga @jennmueng