Query pagination

When retrieving many entities, you may not want to retrieve all at once, e.g., for performance reasons or to prevent connection timeouts, but rather in a chunked way. For that purpose, there is the page_length parameter in the execute_query() function. If this is set to a non-zero integer, the behavior of the function changes in that it returns a Python generator which can be used, e.g., in loops or in list comprehension. The generator yields a Container containing the next page_length many entities from the query result.

The following example illustrates this on the demo server.

import linkahead as db

# 10 at the time of writing of this example
print(db.execute_query("FIND MusicalInstrument"))

# Retrieve in pages of length 5 and iterate over the pages
for page in db.execute_query("FIND MusicalInstrument", page_length=5):
    # each page is a container
    print(type(page))
    # exactly page_length=5 for the first N-1 pages,
    # and possibly less for the last page
    print(len(page))
    # the items on each page are subclasses of Entity
    print(type(page[0]))
    # The id of the first entity on the page is different for all pages
    print(page[0].id)

# You can use this in a list comprehension to fill a container
container_paginated = db.Container().extend(
    [ent for page in db.execute_query("FIND MusicalInstrument", page_length=5) for ent in page]
)
# The result is the same as in the unpaginated case, but the
# following can cause connection timeouts in case of very large
# retrievals
container_at_once = db.execute_query("FIND MusicalInstrument")
for ent1, ent2 in zip(container_paginated, container_at_once):
   print(ent1.id == ent2.id)  # always true

As you can see, you can iterate over a paginated query and then access the entities on each page during the iteration.

Note

The page_length keyword is ignored for COUNT queries where execute_query() always returns the integer result and in case of unique=True where always exactly one Entity is returned.

Warning

Be careful when combining query pagination with insert, update, or delete operations. If your database changes while iterating over a paginated query, the client will raise a PagingConsistencyError since the server can’t guarantee that the query results haven’t changed in the meantime.