feat(profiles): Add dataset configurations and processor for profile chunks

[phacops]

Related:

• feat(profiles): Add schema definition for profile chunks sentry-kafka-schemas#272
• feat: Add support for DateTime64 column type #5896
• feat(profiles): Add a dataset to store profile chunks #5895

[github-actions]

This PR has a migration; here is the generated SQL

-- start migrations

-- forward migration profile_chunks : 0001_create_profile_chunks_table
Local op: CREATE TABLE IF NOT EXISTS profile_chunks_local (project_id UInt64, profiler_id UUID, chunk_id UUID, start_timestamp DateTime64(6) CODEC (DoubleDelta), end_timestamp DateTime64(6) CODEC (DoubleDelta), retention_days UInt16, partition UInt16, offset UInt64) ENGINE ReplicatedReplacingMergeTree('/clickhouse/tables/profile_chunks/{shard}/default/profile_chunks_local', '{replica}') ORDER BY (project_id, profiler_id, start_timestamp, cityHash64(chunk_id)) PARTITION BY (retention_days, toStartOfDay(start_timestamp)) SAMPLE BY cityHash64(chunk_id) TTL toDateTime(end_timestamp) + toIntervalDay(retention_days) SETTINGS index_granularity=8192;
Distributed op: CREATE TABLE IF NOT EXISTS profile_chunks_dist (project_id UInt64, profiler_id UUID, chunk_id UUID, start_timestamp DateTime64(6) CODEC (DoubleDelta), end_timestamp DateTime64(6) CODEC (DoubleDelta), retention_days UInt16, partition UInt16, offset UInt64) ENGINE Distributed(`cluster_one_sh`, default, profile_chunks_local, cityHash64(profiler_id));
-- end forward migration profile_chunks : 0001_create_profile_chunks_table




-- backward migration profile_chunks : 0001_create_profile_chunks_table
Distributed op: DROP TABLE IF EXISTS profile_chunks_dist;
Local op: DROP TABLE IF EXISTS profile_chunks_local;
-- end backward migration profile_chunks : 0001_create_profile_chunks_table

[volokluev]

Looks good, just turn on those allocation policies. You see them off by default to to a rollout a while ago. They're all on in prod now

[codecov]

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 876 tests with 1 failed, 873 passed and 2 skipped.

View the full list of failed tests

Test Description

Failure message

Testsuite: pytest Test name: tests.datasets.test_factory::test_dataset_load[profiles] Envs: - default

Traceback (most recent call last): File ".../datasets/entities/factory.py", line 50, in get return self._entity_map[name] ~~~~~~~~~~~~~~~~^^^^^^ KeyError: EntityKey.CHUNKS The above exception was the direct cause of the following exception: Traceback (most recent call last): File ".../tests/datasets/test_factory.py", line 27, in test_dataset_load for entity in dataset.get_all_entities(): ^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../snuba/datasets/dataset.py", line 42, in get_all_entities return [get_entity(entity_key) for entity_key in self.__all_entities] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../snuba/datasets/dataset.py", line 42, in <listcomp> return [get_entity(entity_key) for entity_key in self.__all_entities] ^^^^^^^^^^^^^^^^^^^^^^ File ".../datasets/entities/factory.py", line 86, in get_entity return _ent_factory().get(name) ^^^^^^^^^^^^^^^^^^^^^^^^ File ".../datasets/entities/factory.py", line 52, in get raise InvalidEntityError(f"entity {name!r} does not exist") from error snuba.datasets.entities.factory.InvalidEntityError: entity EntityKey.CHUNKS does not exist