400 Error - {'code': 400, 'message': 'limit + offset must be less than 10000'}

[SamsGitHub1]

Describe the bug
Unable to get device ids using GetDeviceDetails/QueryDevicesByFilterScroll which has more than 10000 devices for any CIDs

To Reproduce
print(falcon.query_devices_by_filter(offset="10000",limit=1)["body"])
{'code': 400, 'message': 'limit + offset must be less than 10000'}]}

Expected behavior
If any Member CID has more than 10000 device ids , we should be able to pull the responses

Environment (please complete the following information):

• OS: windows
• Python: 3.9
  falcon | 3.0.1
  falcon-api | 0.0.1

My Requirement : I need to pull all the device detail information using query_devices_by_filter method if any Member cid has more than 10000 records, I am getting below error
{'code': 400, 'message': 'limit + offset must be less than 10000'}]}

If you need any more information , I can provide . Thanks in advance

Additional context
Add any other context about the problem here.

[jshcodes]

Hi @SamsGitHub1!

QueryDevicesByFilter (query_devices_by_filter) only supports result sets of less than 10,000. So the filter would need to trim that down in order to retrieve the device you're looking for. (offset will not accept a value greater than 9998 in the example above.)

QueryDevicesByFilterScroll (query_devices_by_filter_scroll) does not have this limitation, but paging is a little different. (Note that the offset is a string.) We have a sample talking about this located here.

If you provide a sample of your code, we're happy to check it out and assist. 🧐

Don't forget to sanitize any CID or AID values, and your API keys from any code you post.

[SamsGitHub1]

print(falcon.query_devices_by_filter(offset="",limit=100)["body"])
print(falcon.query_devices_by_filter(offset="0",limit=1000)["body"]["meta"]["pagination"])
print(falcon.query_devices_by_filter(offset="1001",limit=1000)["body"]["meta"]["pagination"])
print(falcon.query_devices_by_filter(offset="2001",limit=1000)["body"]["meta"]["pagination"])
# print(falcon.query_devices_by_filter(offset="3001",limit=1000)["body"]["meta"]["pagination"])
print(falcon.query_devices_by_filter(offset="10000",limit=1)["body"])

[jshcodes]

Without seeing the results of the code above, I'm not entirely sure how to help yet, so I'm going to ask more questions. 😄

For the last statement in the post above:

print(falcon.query_devices_by_filter(offset="10000",limit=1)["body"])

This will fail, as the query_devices_by_filter method cannot produce results that are greater than 10,000. If you are trying to get the details for device 10,001 in a specific query result, you will need to move over to using query_devices_by_filter_scroll and swap to using offset as a string. (Basically a positional token, this will not be an integer.)

[SamsGitHub1]

Yes, I have tried that even, and I am getting Bad request response
code : print(falcon.query_devices_by_filter_scroll(offset="10000",limit=1))

errors': [{'code': 400, 'message': 'Bad request'}]}

[jshcodes]

Offsets are not (nor do they look like) integers when using query_devices_by_filter_scroll, so we have to handle them differently to accomplish the same goal.

Check out this example and let us know if it helps get you started.

def get_query_results(max_rows: int):
    """
    Performs a QueryDevicesByFilterScroll operation.
    Loops thru all results available.
    """
    # Higher than counter so our loop starts
    total = 1
    # Running counter
    counter = 0
    # List to hold all of the IDs returned by calls
    returning = []
    # Since the scrolling offset value is a string,
    # we will need to use a second variable to track
    # it's value when using this method
    offset_value = ""
    # A counter is used to keep track of our position in the
    # entire resultset.
    while counter < total:
        # Scrolling offset token, no maximum record count
        # We use the offset token string, not the offset integer
        result = falcon.query_devices_by_filter_scroll(
            sort="hostname|asc",
            limit=max_rows,
            offset=offset_value,
            # filter="platform_name:'Linux'"   # Uncomment to add filter
        )["body"]

        # Increment our counter by the value of our limit
        counter += max_rows

        # Retrieve the value of the offset. This will be a token string.
        offset_value = result["meta"]["pagination"]["offset"]

        # This will be the same every time, overrides our init value of 1
        total = result["meta"]["pagination"]["total"]

        # Retrieve the list of IDs returned
        id_list = result["resources"]

        # Append this list to our running list of all IDs
        returning.extend(id_list)

    return returning


# Connect to the Hosts API
falcon = Hosts(client_id="API_CLIENT_ID_HERE",
               client_secret="API_CLIENT_SECRET_HERE"
               )

# Number of records to return per call, max is 5000
LIMIT = 5000

all_results_returned = get_query_results(LIMIT)

print(all_results_returned)

[jshcodes]

Hi @SamsGitHub1!

Bumping this thread to see if you are still encountering issues.

[SamsGitHub1]

Hi @jshcodes ,,

Thanks a lot ,above code help to get list of ids.... Thank you so much for your help.