Work with AI - ScrapeLLM

ScrapeLLM is designed to work inside AI pipelines. This page covers how to provide ScrapeLLM context to AI assistants and how to wire it into agents and LLM workflows.

Documentation index for AI tools

Paste the following URL into any AI assistant (Claude, ChatGPT, Cursor, etc.) to give it full context about the ScrapeLLM API:

https://docs.scrapellm.com/llms.txt

This file lists every documentation page and OpenAPI spec in a format optimised for LLM consumption.

Copy-paste context

If you want to give an AI assistant a quick summary of ScrapeLLM without fetching the docs, copy the block below:

ScrapeLLM is a REST API that extracts real AI responses from ChatGPT, Perplexity, 
Grok, Copilot, Gemini, Google AI Mode, and Amazon Rufus as structured JSON.

Authentication: pass your API key in the X-API-Key header.
Base URL: https://api.scrapellm.com

Sync request (GET):
  GET /scrapers/{scraper}?prompt=...&country=US
  Headers: X-API-Key: YOUR_KEY
  Returns: { scraper, status, result, links/sources/citations, credits_used, ... }

Async request (POST → poll):
  POST /scrapers/{scraper}/jobs?prompt=...&country=US
  Returns: { job_id, status: "pending" }
  Poll: GET /jobs/{job_id} until status is "done" or "failed"

Scrapers: chatgpt, perplexity, grok, copilot, gemini, google_ai_mode, amazon_rufus
Credits: 3 per request. Free tier: 1,500/month.

Using ScrapeLLM as an AI agent tool

ScrapeLLM works naturally as a tool in any LLM agent framework. Here are minimal tool definitions:

from langchain.tools import tool
import requests

@tool
def scrape_chatgpt(prompt: str, country: str = "US") -> dict:
    """Query ChatGPT and return the real web UI response as structured JSON,
    including cited sources and internal search queries."""
    return requests.get(
        "https://api.scrapellm.com/scrapers/chatgpt",
        headers={"X-API-Key": "YOUR_API_KEY"},
        params={"prompt": prompt, "country": country},
    ).json()

@tool
def scrape_perplexity(prompt: str, country: str = "US") -> dict:
    """Query Perplexity AI and return the response with numbered citations and source snippets."""
    return requests.get(
        "https://api.scrapellm.com/scrapers/perplexity",
        headers={"X-API-Key": "YOUR_API_KEY"},
        params={"prompt": prompt, "country": country},
    ).json()

Common AI pipeline patterns

Cross-provider brand monitoring

Query the same prompt across multiple AI providers in parallel to measure your brand’s AI visibility:

import asyncio, aiohttp

async def monitor_brand(prompt: str, scrapers: list[str], country: str = "US"):
    async with aiohttp.ClientSession() as session:
        tasks = [
            session.get(
                f"https://api.scrapellm.com/scrapers/{s}",
                headers={"X-API-Key": "YOUR_API_KEY"},
                params={"prompt": prompt, "country": country},
            )
            for s in scrapers
        ]
        responses = await asyncio.gather(*tasks)
        return {
            s: await r.json()
            for s, r in zip(scrapers, responses)
        }

results = asyncio.run(monitor_brand(
    "What CRM do sales teams recommend?",
    scrapers=["chatgpt", "perplexity", "grok", "gemini"],
))

for scraper, data in results.items():
    print(f"{scraper}: {data['result'][:200]}")

Feed AI responses into another LLM

Pipe ScrapeLLM output directly into an LLM for synthesis or analysis:

import requests
from openai import OpenAI

client = OpenAI()

# 1. Get the real AI response
scrape = requests.get(
    "https://api.scrapellm.com/scrapers/perplexity",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={"prompt": "Best project management tools for remote teams?"},
).json()

# 2. Feed it to GPT-4o for analysis
analysis = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You analyse AI search results for brand mentions."},
        {"role": "user", "content": f"Analyse this Perplexity response:\n\n{scrape['result']}\n\nSources: {scrape['sources']}"},
    ],
)

print(analysis.choices[0].message.content)

Documentation Index

​Documentation index for AI tools

​Copy-paste context

​Using ScrapeLLM as an AI agent tool

​Common AI pipeline patterns

​Cross-provider brand monitoring

​Feed AI responses into another LLM