Complete Guide to Building AI Agents with Search APIs

AI agents are the next evolution beyond chatbots -- autonomous systems that can plan, execute multi-step workflows, use tools, and iterate on their own output. The missing piece for most agents? Access to real-time web data. This guide covers how to build production-grade AI agents using search and scraping APIs for grounding, with a focus on SearchHive's integrated API stack.

Key Takeaways

• Web-grounded AI agents produce more accurate, current responses than knowledge-cutoff models
• SearchHive's three-product stack provides search, scraping, and analysis in one pipeline
• Tool-use patterns (function calling) let agents decide when to search, scrape, or analyze
• Cost control is critical -- agents can burn through API credits fast without guardrails
• Python + httpx + your LLM is the most flexible agent architecture

Background

DataSync, a fintech startup, needed an AI agent that could autonomously research public companies for investment signals. The agent needed to:

• Search for recent news and SEC filings
• Extract financial data from company websites and EDGAR
• Analyze sentiment across multiple sources
• Generate structured research reports with citations

They started with an LLM-only approach but found the model hallucinated financial figures and missed recent events. The agent needed real-time web access.

The Challenge

Building a web-grounded AI agent involves several interconnected problems:

1. Tool selection: The agent needs to decide when to search vs when to scrape a specific URL vs when to analyze collected data
2. Cost management: Unrestricted tool access leads to exponential API calls per query
3. Latency: Each search/scrape/analyze step adds seconds. A 10-step agent workflow can take 60+ seconds
4. Error recovery: Scraped pages fail, search returns no results, APIs rate-limit -- the agent needs to handle all of this gracefully
5. Output quality: The final report needs structured data, citations, and accuracy -- not LLM hallucinations

The Solution with SearchHive

DataSync built their agent on SearchHive's API stack with three tool categories:

import httpx
import json

SEARCHHIVE_API_KEY = "your-key"
BASE = "https://api.searchhive.dev/v1"
HEADERS = {
    "Authorization": f"Bearer {SEARCHHIVE_API_KEY}",
    "Content-Type": "application/json"
}

# Tool 1: SwiftSearch - web search
async def web_search(query, num_results=5):
    resp = await httpx.post(
        f"{BASE}/swift/search",
        headers=HEADERS,
        json={"query": query, "num_results": num_results},
        timeout=30.0
    )
    return resp.json().get("results", [])

# Tool 2: ScrapeForge - page extraction
async def scrape_page(url, selectors=None):
    payload = {"url": url, "format": "markdown"}
    if selectors:
        payload["extract_selectors"] = selectors
    resp = await httpx.post(
        f"{BASE}/scrape/extract",
        headers=HEADERS,
        json=payload,
        timeout=60.0
    )
    return resp.json().get("content", "")

# Tool 3: DeepDive - AI analysis
async def analyze(query, context, fmt="structured"):
    resp = await httpx.post(
        f"{BASE}/deep/analyze",
        headers=HEADERS,
        json={
            "query": query,
            "context": context[:10000],
            "output_format": fmt
        },
        timeout=45.0
    )
    return resp.json()

Implementation

Step 1: Agent Loop with Tool Calling

The core agent loop receives a research query, plans which tools to use, executes them, and iterates until it has enough information to produce a final answer.

import asyncio

MAX_ITERATIONS = 8  # prevent infinite loops
MAX_CREDITS_PER_RUN = 50  # cost guardrail
credits_used = 0

SYSTEM_PROMPT = """You are a financial research agent. You have three tools:
1. web_search(query) - Search the web for information
2. scrape_page(url) - Extract content from a specific URL
3. analyze(query, context) - Analyze collected data with AI

For each step, output a JSON object with:
{"tool": "tool_name", "args": {...}, "reason": "why this tool helps"}

When you have enough information, output:
{"done": true, "report": "your final report here"}

Rules:
- Always cite sources with URLs
- Never fabricate financial figures
- Prefer primary sources (SEC filings, company IR pages)
- Use scrape_page for specific URLs found in search results
"""

async def agent_loop(user_query, llm_client):
    global credits_used, MAX_CREDITS_PER_RUN
    
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_query}
    ]
    
    collected_context = []
    
    for iteration in range(MAX_ITERATIONS):
        if credits_used >= MAX_CREDITS_PER_RUN:
            break
        
        # Get LLM decision
        response = await llm_client.chat(messages)
        decision = json.loads(response)
        
        if decision.get("done"):
            return decision["report"]
        
        # Execute chosen tool
        tool = decision["tool"]
        args = decision["args"]
        
        if tool == "web_search":
            results = await web_search(args["query"])
            credits_used += 1
            summary = format_search_results(results)
            collected_context.append(summary)
            
        elif tool == "scrape_page":
            content = await scrape_page(args["url"])
            credits_used += 1
            collected_context.append(f"Source: {args['url']}\n{content[:3000]}")
            
        elif tool == "analyze":
            analysis = await analyze(args["query"], "\n\n".join(collected_context))
            credits_used += 1
            return analysis.get("analysis", "")
        
        # Feed result back to LLM
        messages.append({"role": "assistant", "content": json.dumps(decision)})
        messages.append({"role": "user", "content": f"Tool result: {summary}"})
    
    return "Agent reached maximum iterations."

def format_search_results(results):
    lines = []
    for r in results[:5]:
        lines.append(f"- {r['title']}: {r['url']}")
        lines.append(f"  {r.get('snippet', '')}")
    return "\n".join(lines)

Step 2: Example Research Workflow

Here's how the agent handles a query like "Summarize Tesla's Q1 2026 earnings and analyst sentiment":

1. web_search("Tesla Q1 2026 earnings results") -- Finds IR page, news articles, SEC filing
2. scrape_page(tesla-ir-page-url) -- Extracts earnings press release content
3. web_search("Tesla Q1 2026 analyst sentiment bullish bearish") -- Finds analyst opinions
4. scrape_page(analyst-article-url) -- Extracts key quotes and ratings
5. analyze("Summarize Tesla Q1 2026 earnings and sentiment", collected_context) -- Generates final report

Five API calls, under 45 seconds, with primary-source citations.

Step 3: Structured Output with Citations

The analyze tool returns structured free JSON formatter that maps directly to the output format:

async def generate_research_report(query):
    # Step 1: Gather sources
    search_results = await web_search(query, num_results=8)
    
    # Step 2: Scrape top 3 results
    scrape_tasks = []
    for r in search_results[:3]:
        scrape_tasks.append(scrape_page(r["url"]))
    scraped = await asyncio.gather(*scrape_tasks)
    
    # Step 3: Analyze with context
    context = "\n\n---\n\n".join(scraped)
    analysis = await analyze(
        "Generate a structured research report with sections: "
        "Summary, Key Findings, Risks, Outlook. Include citations.",
        context
    )
    
    return analysis

Results

DataSync's search-grounded agent produced measurably better output than their previous LLM-only approach:

The 3x cost increase was negligible compared to the massive improvement in accuracy and utility. Investment analysts rated the search-grounded reports as "significantly more useful" in blind testing.

Lessons Learned

1. Limit iterations aggressively. Without a hard cap, agents spiral into infinite search loops. DataSync found 5-8 iterations sufficient for most queries.

2. Prefer scraping over search when you have a URL. Don't re-search for something you've already found. If a search result has the URL you need, scrape it directly.

3. Track credit usage per run. Set a maximum budget per query and log each tool call. Agents sometimes call the same tool twice with slight variations.

4. Cache search results within a session. If the agent searches for "Tesla revenue" twice in one run, return the cached results on the second call.

5. Structure your prompts for tool selection. The system prompt should clearly define when each tool is appropriate, otherwise the LLM over-uses search and under-uses scraping.

Getting Started

Building AI agents with web access is straightforward with SearchHive's unified API. The free tier includes 500 credits -- enough to test full agent workflows before committing to a paid plan.

• SwiftSearch docs -- web search integration
• ScrapeForge docs -- page extraction
• DeepDive docs -- AI analysis

Read more about building AI agents and compare with alternative search APIs to find the right stack for your agent architecture.