client.filter_log_events paginator returns 1 extra next_token

[ShogunLSG]

I ran into a problem with the pagination api in the back end where it returns an extra next_token that points to an empty response on the next call. (even after all the required log events have been returns it returns 1 more that returns nothing).
I'm not sure if this is a bug with the api or a problem with my code but at the moment i can't see a problem with the code i had written.

the api:

response_iterator = paginator.paginate(
        logGroupName=log_group,
        startTime=int_start_time,
        endTime=int_end_time,
        filterPattern=query,
        PaginationConfig={
            "MaxItems": page_size,
            "PageSize": page_size,
            "StartingToken": start_token,
        },
    )

for the query i used metric filters.

[ShogunLSG]

Please let me know if you need more context or information.

[tim-finnigan]

Hi @ShogunLSG it looks like you're using the DescribeLogGroups paginator, is that correct? I'll also link to the boto3 paginator documentation for more general information. The results returned can vary between service APIs and also depend on things like the max results limit and filters used. If you add boto3.set_stream_logger('') to you script that will provide more insight into the behavior. If you want to share those debug logs (with any sensitive info redacted) then we could also look into it further. Thanks!