Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(eap-spans): Drop ineffective indices to speed up insertion #6206

Merged
merged 2 commits into from
Aug 14, 2024

Conversation

phacops
Copy link
Contributor

@phacops phacops commented Aug 14, 2024

Inserting data in this cluster is slow and having many indices is one of the reason. These indices we added were not as effective as we thought so it's harmless to remove them and will probably help insertion.

@phacops phacops requested a review from a team as a code owner August 14, 2024 19:42
Copy link

github-actions bot commented Aug 14, 2024

This PR has a migration; here is the generated SQL

-- start migrations

-- forward migration events_analytics_platform : 0005_drop_attribute_key_project_id_indexes
Local op: ALTER TABLE eap_spans_local DROP INDEX IF EXISTS bf_attr_str_0, DROP INDEX IF EXISTS bf_attr_str_1, DROP INDEX IF EXISTS bf_attr_str_2, DROP INDEX IF EXISTS bf_attr_str_3, DROP INDEX IF EXISTS bf_attr_str_4, DROP INDEX IF EXISTS bf_attr_str_5, DROP INDEX IF EXISTS bf_attr_str_6, DROP INDEX IF EXISTS bf_attr_str_7, DROP INDEX IF EXISTS bf_attr_str_8, DROP INDEX IF EXISTS bf_attr_str_9, DROP INDEX IF EXISTS bf_attr_str_10, DROP INDEX IF EXISTS bf_attr_str_11, DROP INDEX IF EXISTS bf_attr_str_12, DROP INDEX IF EXISTS bf_attr_str_13, DROP INDEX IF EXISTS bf_attr_str_14, DROP INDEX IF EXISTS bf_attr_str_15, DROP INDEX IF EXISTS bf_attr_str_16, DROP INDEX IF EXISTS bf_attr_str_17, DROP INDEX IF EXISTS bf_attr_str_18, DROP INDEX IF EXISTS bf_attr_str_19, DROP INDEX IF EXISTS bf_attr_num_0, DROP INDEX IF EXISTS bf_attr_num_1, DROP INDEX IF EXISTS bf_attr_num_2, DROP INDEX IF EXISTS bf_attr_num_3, DROP INDEX IF EXISTS bf_attr_num_4, DROP INDEX IF EXISTS bf_attr_num_5, DROP INDEX IF EXISTS bf_attr_num_6, DROP INDEX IF EXISTS bf_attr_num_7, DROP INDEX IF EXISTS bf_attr_num_8, DROP INDEX IF EXISTS bf_attr_num_9, DROP INDEX IF EXISTS bf_attr_num_10, DROP INDEX IF EXISTS bf_attr_num_11, DROP INDEX IF EXISTS bf_attr_num_12, DROP INDEX IF EXISTS bf_attr_num_13, DROP INDEX IF EXISTS bf_attr_num_14, DROP INDEX IF EXISTS bf_attr_num_15, DROP INDEX IF EXISTS bf_attr_num_16, DROP INDEX IF EXISTS bf_attr_num_17, DROP INDEX IF EXISTS bf_attr_num_18, DROP INDEX IF EXISTS bf_attr_num_19;
Local op: ALTER TABLE eap_spans_local DROP INDEX IF EXISTS bf_project_id;
-- end forward migration events_analytics_platform : 0005_drop_attribute_key_project_id_indexes




-- backward migration events_analytics_platform : 0005_drop_attribute_key_project_id_indexes
Local op: ALTER TABLE eap_spans_local ADD INDEX IF NOT EXISTS bf_attr_str_0 mapKeys(attr_str_0) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_1 mapKeys(attr_str_1) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_2 mapKeys(attr_str_2) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_3 mapKeys(attr_str_3) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_4 mapKeys(attr_str_4) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_5 mapKeys(attr_str_5) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_6 mapKeys(attr_str_6) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_7 mapKeys(attr_str_7) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_8 mapKeys(attr_str_8) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_9 mapKeys(attr_str_9) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_10 mapKeys(attr_str_10) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_11 mapKeys(attr_str_11) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_12 mapKeys(attr_str_12) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_13 mapKeys(attr_str_13) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_14 mapKeys(attr_str_14) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_15 mapKeys(attr_str_15) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_16 mapKeys(attr_str_16) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_17 mapKeys(attr_str_17) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_18 mapKeys(attr_str_18) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_19 mapKeys(attr_str_19) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_0 mapKeys(attr_num_0) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_1 mapKeys(attr_num_1) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_2 mapKeys(attr_num_2) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_3 mapKeys(attr_num_3) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_4 mapKeys(attr_num_4) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_5 mapKeys(attr_num_5) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_6 mapKeys(attr_num_6) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_7 mapKeys(attr_num_7) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_8 mapKeys(attr_num_8) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_9 mapKeys(attr_num_9) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_10 mapKeys(attr_num_10) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_11 mapKeys(attr_num_11) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_12 mapKeys(attr_num_12) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_13 mapKeys(attr_num_13) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_14 mapKeys(attr_num_14) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_15 mapKeys(attr_num_15) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_16 mapKeys(attr_num_16) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_17 mapKeys(attr_num_17) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_18 mapKeys(attr_num_18) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_19 mapKeys(attr_num_19) TYPE bloom_filter GRANULARITY 1;
Local op: ALTER TABLE eap_spans_local ADD INDEX IF NOT EXISTS bf_project_id project_id TYPE bloom_filter GRANULARITY 1;
-- end backward migration events_analytics_platform : 0005_drop_attribute_key_project_id_indexes

@phacops phacops enabled auto-merge (squash) August 14, 2024 20:10
@phacops phacops merged commit e13eb69 into master Aug 14, 2024
30 checks passed
@phacops phacops deleted the pierre/eap-spans-drop-attribute-key-indexes branch August 14, 2024 20:39
@asottile-sentry
Copy link
Member

asottile-sentry commented Aug 14, 2024

this is breaking sentry

ValueError: Duplicate migration number for the following files:
    snuba/snuba_migrations/events_analytics_platform/0005_drop_attribute_key_project_id_indexes.py
    snuba/snuba_migrations/events_analytics_platform/0005_remove_attribute_mv.py

@getsentry-bot
Copy link
Contributor

PR reverted: 6882653

getsentry-bot added a commit that referenced this pull request Aug 14, 2024
#6206)"

This reverts commit e13eb69.

Co-authored-by: asottile-sentry <103459774+asottile-sentry@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants