Pagination for TimestreamDB

[sutiv]

Queries can take a long time and the AWS API gateway times out.

Is there a possibility to use pagination as offered by boto3? https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/timestream-query.html#TimestreamQuery.Client.get_paginator

[kukushking]

Hi @sutiv pagination is implemented but It appears to be using default config which essentially means no pagination. I'll add a parameter for pagination details.

[sutiv]

thanks a lot!

[sutiv]

Almost perfect, but isn´t there something missing?! Something like next_token = page.get('NextToken').

paginator.paginate() will return a 'NextToken', if a 'StartingToken' was given AND the page isn´t the last page. This NextToken isn´t returned by your solution and thereby I can´t call the next page.

[kukushking]

Actually current implementation lists through all pages and collects the results so you don't have to:

    for page in paginator.paginate(QueryString=sql, PaginationConfig=pagination_config or {}):
        if not schema:
            schema = _process_schema(page=page)
        for row in page["Rows"]:
            rows.append(_process_row(schema=schema, row=row))

Although now that I think about it, it would be useful to be able to iterate through pages in case result set is too big. I'll add this.

[jaidisido]

Available in 2.11.0