TT-12074 Panic trying to convert empty data into AnalyticsRecord

[dcupif]

Description

The pump crashes when trying to convert empty data into an AnalyticsRecord.

panic: interface conversion: interface {} is nil, not analytics.AnalyticsRecord

How to reproduce

Following our previous issue #752, we are facing limitations where we hit the hardware network interface max bandwidth.

When doing so, we observe almost instantly that the pump crashes with the above-mentioned log.

We use the pump to export analytics records as Prometheus metrics.

[tbuchaillot]

Hey @dcupif! What pump version are you using? Also, would you mind attaching your pump config and logs? It will help to identify the issue.

[dcupif]

Hi @tbuchaillot, thanks for the quick reply.

We are using version 1.8.3 of your public Docker image tykio/tyk-pump-docker-pub.

Here's the gateway configuration for the pump:

{
    "storage": {
        "type": "redis",
        "addrs": [ "HOST" ],
        "username": "USERNAME",
        "password": "PASSWORD",
        "use_ssl": true,
        "database": 0,
        "optimisation_max_idle": 2000,
        "optimisation_max_active": 4000
    },
    "enable_analytics": true,
    "analytics_config": {
        "purge_delay": -1,
        "ignored_ips": []
    },
    "optimisations_use_async_session_write": true,
    "enable_non_transactional_rate_limiter": true,
    "enable_sentinel_rate_limiter": false,
    "enable_redis_rolling_limiter": false,
    "allow_master_keys": false,
    "hash_keys": true,
    "close_connections": false,
    "http_server_options": {
        "enable_websockets": true
    },
    "global_session_lifetime": 100,
    "force_global_session_lifetime": true,
    "max_idle_connections_per_host": 500,
    "enable_jsvm": false,
}

Here's the actual pump configuration:

{
    "analytics_storage_type": "redis",
    "analytics_storage_config": {
        "type": "redis",
        "host": "HOST",
        "port": "PORT",
        "username": "USERNAME",
        "password": "PASSWORD",
        "redis_use_ssl": true,
        "database": 0,
        "optimisation_max_idle": 200,
        "optimisation_max_active": 500
    },
    "enable_analytics": true,
    "analytics_config": {
        "purge_delay": -1,
        "ignored_ips": []
    },
    "purge_delay": 4,
    "dont_purge_uptime_data": true,
    "log_format": "json",
    "pumps": {
        "prometheus": {
        "type": "prometheus",
        "meta": {
            "listen_address": "0.0.0.0:9090",
            "path": "/metrics",
            "disabled_metrics": ["tyk_http_status_per_key", "tyk_latency"],
            "custom_metrics": [
            {
                "name": "tyk_http_status_per_api",
                "description": "HTTP status codes per API",
                "metric_type": "counter",
                "labels": [ "response_code","api_id","api_name","method" ]
            },
            {
                "name": "tyk_latency_per_api",
                "description": "Latency buckets per API",
                "metric_type": "histogram",
                "labels": [ "api_id","api_name","method" ],
                "buckets": [1,2,5,10,20,30,50,75,100,200,300,400,500,1000,2000,5000,10000,30000,60000]
            }
            ]
        }
        },
        "stdout": {
        "type": "stdout",
        "meta" : {
            "log_field_name": "tyk-analytics-record",
            "format": "json"
        }
        }
    }
}

And here is an extract of the logs:

panic: interface conversion: interface {} is nil, not analytics.AnalyticsRecord
	/__w/tyk-pump/tyk-pump/main.go:392 +0x76
github.com/TykTechnologies/tyk-pump/pumps.(*PrometheusPump).WriteData(0xc0003ae240, {0x22b73a0, 0xc000896000}, {0xc000780000, 0x194a, 0x0?})
created by main.execPumpWriting
main.execPumpWriting.func2(0xc000522ed0?, {0x22b73a0, 0xc000896000}, {0x22c3be0, 0xc0003ae240}, 0xc0003f2180?)
	/__w/tyk-pump/tyk-pump/main.go:390 +0x432
	/__w/tyk-pump/tyk-pump/pumps/prometheus.go:251 +0xd54

[dcupif]

Hi @tbuchaillot, let me know if you are missing information to pursue investigation, this issue is still relevant to us. Thank you 🙏

[tbuchaillot]

Thanks for the information @dcupif! Agree that this should be related with your network/redis problems. A malformed record shouldn't reach the Prometheus pump.
@mativm02 can you have a look? Discarding the malformed records there should be super low effort.

[caroltyk]

Similar to issue #768

[caroltyk]

Closing this issue as duplicated. Please check for updates on #768. Thanks!