Is there a rate limit? Getting transient empty responses with many requests.

[Aariq]

When performing many requests (~2000) to the /awards/ endpoints in succession I get many of them failed ( ~1000) with nothing returned from the server (no headers, no data). But then if I re-run those failed requests one by one manually, they work fine. I feel like I must be hitting some kind of rate limit, but I don't see one documented anywhere. I'm currently throttling to 100 requests / 30 seconds with a maximum of 5 requests in parallel, but still having this issue.

An example failed request that worked on the second try: https://api.usaspending.gov/api/v2/awards/ASST_NON_2331010_4900/

[aguest-kc]

There is a global rate limit that applies to most endpoints of 1,000 request in 300 seconds (some endpoints are further rate limited). For the ~1,000 request that returned nothing (no headers, no data) do you know what status code you received?

[Aariq]

I'm using httr2 and I just get nothing

Error in `req_perform_parallel()`:
! Failed to perform HTTP request.
Caused by error:
! Server returned nothing (no headers, no data) [api.usaspending.gov]:
Empty reply from server

If I try to ask for the status code I just get an error. I'm not exactly sure what would cause this, but perhaps it isn't waiting long enough to get a response or has something to do with parallel requests. Currently the max number of concurrent requests is 10.

A few give 404 errors, but those seem to be correct, e.g. https://api.usaspending.gov/api/v2/awards/ASST_NON_2502740_4900/

[Aariq]

Even if I use sequential requests, I still get some of these empty reply errors