ref: Deprecate TOPIC_PARTITION_COUNTS setting

[lynnagara]

Snuba should fetch the actual value from the broker on startup rather than rely on a hardcoded TOPIC_PARTITION_COUNTS settings. This will ensure that it always has the correct values and remove the manual process and associated drift in ops that is currently involved in keeping this up to date.

Depends on getsentry/sentry#70478

This is a prerequisite to https://github.com/getsentry/ops/pull/10392 and https://github.com/getsentry/ops/pull/10451 which will enable clean up of topicctl configuration in ops.

[fpacifici]

?

[lynnagara]

i'm debugging a test that fails in CI but not locally

[lynnagara]

removed now

[evanh]

Out of curiousity, if the partition check fails for some reason, what would happen

[evanh]

This connection already happens in bootstrap.py. Is there a way we could leverage that code, to avoid connecting twice on startup?

Also, there is a utility function that does this: https://github.com/getsentry/snuba/blob/master/snuba/consumers/utils.py#L17

Could we use that instead?

[lynnagara]

Ugggh I just noticed a pretty bad bug with that other function. It does not properly resolve the topic name if it is overridden to something else and assumes the default. So I guess we can't safely use it right now, but also... this probably needs to be fixed?

[evanh]

Sounds like it should be fixed. Did you want to add it to this PR?

[lynnagara]

Out of curiousity, if the partition check fails for some reason, what would happen

The pod would crash if Kafka or the topic is unavailable for some reason. This applies to the subscription scheduler pod as well as the API pod if the subscription POST endpoint is called.

[enochtangg]

This applies to the subscription scheduler pod as well as the API pod if the subscription POST endpoint is called

Is there any way around this so that the API pod doesn't crash? This kinda reminds me of the old readiness health-check we had before where snuba API would crash if it isn't able to connect to ClickHouse. Ideally, the API still works for all other the other datasets even if one partition check fails for a single dataset.

[lynnagara]

Discussed offline. Looks like exception handling and returning a proper response is already automatically handled by

snuba/snuba/web/views.py

Lines 173 to 194 in 177e23e

	@application.errorhandler(InternalServerError)
	def handle_internal_server_error(exception: InternalServerError) -> Response:
	original = getattr(exception, "original_exception", None)
	if original is None:
	return Response(
	json.dumps(
	{"error": {"type": "internal_server_error", "message": str(exception)}},
	indent=4,
	),
	500,
	{"Content-Type": "application/json"},
	)
	return Response(
	json.dumps(
	{"error": {"type": "internal_server_error", "message": str(original)}},
	indent=4,
	),
	500,
	{"Content-Type": "application/json"},
	)

[lynnagara]

also needs getsentry/sentry#70809

[lynnagara]

sentry test is failing

[getsentry-bot]

PR reverted: ed0426b