Documentation Index
Fetch the complete documentation index at: https://docs.scrapellm.com/llms.txt
Use this file to discover all available pages before exploring further.
The ScrapeLLM API uses standard HTTP status codes and consistent error response formats across all endpoints.
HTTP status codes
| Status code | Meaning | Description |
|---|
200 | Success | Request completed successfully |
202 | Accepted | Async job submitted successfully |
400 | Bad Request | Request validation failed |
401 | Unauthorized | Missing or invalid API key |
404 | Not Found | Job ID not found or expired |
408 | Request Timeout | AI provider did not respond within the timeout period |
429 | Too Many Requests | Credit limit reached or rate limit exceeded |
500 | Internal Server Error | Scraper returned an unexpected error — safe to retry |
503 | Service Unavailable | Scraper pool is warming up — retry in a few seconds |
504 | Gateway Timeout | Network timeout reaching the scraper |
detail field:
{
"detail": "Human-readable description of the error."
}
Authentication error (401)
{
"detail": "Missing API key. Pass your key in the X-API-Key header or as ?api_key= in the URL."
}
Credit limit (429)
{
"detail": "Credit limit reached. Upgrade your plan or wait for your billing cycle to reset."
}
Timeout (408)
The AI provider did not respond within the timeout window. Increase timeout (up to 600 seconds) or retry with bypass_cache=true.
Job not found (404)
{
"detail": "Job not found. It may have expired (jobs are retained for 24 hours) or the ID is invalid."
}
Common error types
Authentication errors (401)
| Cause | Fix |
|---|
No X-API-Key header and no ?api_key= param | Add the header or query param |
| Key is invalid or revoked | Regenerate your key in the dashboard |
| Account has no credits | Upgrade plan or wait for billing cycle reset |
Credit / rate limit (429)
| Cause | Fix |
|---|
| Monthly credit allowance exhausted | Upgrade plan or wait for next billing cycle |
| Scraper queue full | Retry after a few seconds with exponential backoff |
Server errors (500 / 503 / 504)
These are transient errors — it is always safe to retry.
| Code | Cause |
|---|
500 | Scraper returned an unexpected error or empty result |
503 | Scraper pool is warming up |
504 | Network timeout to the underlying scraper |
Retry logic
Implement exponential backoff for 408, 429, 500, 503, and 504 errors:
import time, requests
def scrape_with_retry(prompt, max_retries=3):
for attempt in range(max_retries):
resp = requests.get(
"https://api.scrapellm.com/scrapers/chatgpt",
headers={"X-API-Key": "YOUR_API_KEY"},
params={"prompt": prompt, "country": "US"},
)
if resp.status_code == 200:
return resp.json()
if resp.status_code in (408, 429, 500, 503, 504) and attempt < max_retries - 1:
time.sleep(2 ** attempt) # 1s, 2s, 4s ...
continue
resp.raise_for_status()
Async jobs are automatically retried server-side up to 3 times before
being marked failed. Credits are restored if all attempts fail.
Async job errors
When an async job reaches failed status, the error message is in the error field:
{
"job_id": "3f7a2b1c-...",
"status": "failed",
"scraper": "chatgpt",
"error": "execution failed",
"created_at": "2026-05-11T12:00:00+00:00",
"completed_at": "2026-05-11T12:01:45+00:00"
}
failed job can be resubmitted by making a new request. Credits from the failed job are restored automatically.
Troubleshooting
401 Unauthorized
- Verify the
X-API-Key header is present and correctly spelled
- Check the key is valid and hasn’t been revoked in the dashboard
- Confirm your account has remaining credits
429 Too Many Requests
- Check your credit balance in the dashboard
- If the queue is full, implement retry with exponential backoff
- Consider using async job mode to avoid holding connections
500 / 503 / 504
These are almost always transient. Retry immediately or after a brief delay. If the error persists for more than a few minutes, contact [email protected] with your job_id.
Amazon Rufus 500
If Rufus returns no products for a very vague query, the endpoint returns 500 with "Scraper returned an empty result." — retry with a more specific shopping query. Credits are restored automatically.
