Error handling

The ScrapeLLM API uses standard HTTP status codes and consistent error response formats across all endpoints.

HTTP status codes

Status code	Meaning	Description
`200`	Success	Request completed successfully
`202`	Accepted	Async job submitted successfully
`400`	Bad Request	Request validation failed
`401`	Unauthorized	Missing or invalid API key
`404`	Not Found	Job ID not found or expired
`408`	Request Timeout	AI provider did not respond within the timeout period
`429`	Too Many Requests	Credit limit reached or rate limit exceeded
`500`	Internal Server Error	Scraper returned an unexpected error - safe to retry
`503`	Service Unavailable	Scraper pool is warming up - retry in a few seconds
`504`	Gateway Timeout	Network timeout reaching the scraper

Error response format

All errors return a JSON body with a detail field:

{
  "detail": "Human-readable description of the error."
}

Authentication error (401)

{
  "detail": "Missing API key. Pass your key in the X-API-Key header or as ?api_key= in the URL."
}

Credit limit (429)

{
  "detail": "Credit limit reached. Upgrade your plan or wait for your billing cycle to reset."
}

Timeout (408)

The AI provider did not respond within the timeout window. Increase timeout (up to 600 seconds) or retry with bypass_cache=true.

Job not found (404)

{
  "detail": "Job not found. It may have expired (jobs are retained for 24 hours) or the ID is invalid."
}

Common error types

Authentication errors (401)

Cause	Fix
No `X-API-Key` header and no `?api_key=` param	Add the header or query param
Key is invalid or revoked	Regenerate your key in the dashboard
Account has no credits	Upgrade plan or wait for billing cycle reset

Credit / rate limit (429)

Cause	Fix
Monthly credit allowance exhausted	Upgrade plan or wait for next billing cycle
Scraper queue full	Retry after a few seconds with exponential backoff

Server errors (500 / 503 / 504)

These are transient errors - it is always safe to retry.

Code	Cause
`500`	Scraper returned an unexpected error or empty result
`503`	Scraper pool is warming up
`504`	Network timeout to the underlying scraper

Retry logic

Implement exponential backoff for 408, 429, 500, 503, and 504 errors:

import time, requests

def scrape_with_retry(prompt, max_retries=3):
    for attempt in range(max_retries):
        resp = requests.get(
            "https://api.scrapellm.com/scrapers/chatgpt",
            headers={"X-API-Key": "YOUR_API_KEY"},
            params={"prompt": prompt, "country": "US"},
        )
        if resp.status_code == 200:
            return resp.json()
        if resp.status_code in (408, 429, 500, 503, 504) and attempt < max_retries - 1:
            time.sleep(2 ** attempt)  # 1s, 2s, 4s ...
            continue
        resp.raise_for_status()

async function scrapeWithRetry(prompt, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    const resp = await fetch(
      `https://api.scrapellm.com/scrapers/chatgpt?prompt=${encodeURIComponent(prompt)}&country=US`,
      { headers: { "X-API-Key": "YOUR_API_KEY" } }
    );

    if (resp.ok) return resp.json();

    const retryable = [408, 429, 500, 503, 504].includes(resp.status);
    if (retryable && i < maxRetries - 1) {
      await new Promise(r => setTimeout(r, Math.pow(2, i) * 1000));
      continue;
    }
    throw new Error(`HTTP ${resp.status}`);
  }
}

Async jobs are automatically retried server-side up to 3 times before being marked failed. Credits are restored if all attempts fail.

Async job errors

When an async job reaches failed status, the error message is in the error field:

{
  "job_id": "3f7a2b1c-...",
  "status": "failed",
  "scraper": "chatgpt",
  "error": "execution failed",
  "created_at": "2026-05-11T12:00:00+00:00",
  "completed_at": "2026-05-11T12:01:45+00:00"
}

A failed job can be resubmitted by making a new request. Credits from the failed job are restored automatically.

Troubleshooting

401 Unauthorized

Verify the X-API-Key header is present and correctly spelled
Check the key is valid and hasn’t been revoked in the dashboard
Confirm your account has remaining credits

429 Too Many Requests

Check your credit balance in the dashboard
If the queue is full, implement retry with exponential backoff
Consider using async job mode to avoid holding connections

500 / 503 / 504

These are almost always transient. Retry immediately or after a brief delay. If the error persists for more than a few minutes, contact [email protected] with your job_id.

Amazon Rufus 500

If Rufus returns no products for a very vague query, the endpoint returns 500 with "Scraper returned an empty result." - retry with a more specific shopping query. Credits are restored automatically.

Introduction

Guides

HTTP status codes

Error response format

Authentication error (401)

Credit limit (429)

Timeout (408)

Job not found (404)

Common error types

Authentication errors (401)

Credit / rate limit (429)

Server errors (500 / 503 / 504)

Retry logic

Async job errors

Troubleshooting

401 Unauthorized

429 Too Many Requests

500 / 503 / 504

Amazon Rufus 500

​HTTP status codes

​Error response format

​Authentication error (401)

​Credit limit (429)

​Timeout (408)

​Job not found (404)

​Common error types

​Authentication errors (401)

​Credit / rate limit (429)

​Server errors (500 / 503 / 504)

​Retry logic

​Async job errors

​Troubleshooting

​401 Unauthorized

​429 Too Many Requests

​500 / 503 / 504

​Amazon Rufus 500

HTTP status codes

Error response format

Authentication error (401)

Credit limit (429)

Timeout (408)

Job not found (404)

Common error types

Authentication errors (401)

Credit / rate limit (429)

Server errors (500 / 503 / 504)

Retry logic

Async job errors

Troubleshooting

401 Unauthorized

429 Too Many Requests

500 / 503 / 504

Amazon Rufus 500