MCP (Model Context Protocol) is the standard for connecting AI agents to external tools and data sources. If you want your AI agent to search the web, scrape pages, or extract data — instead of relying on stale training data — MCP integration is the way to do it.
This tutorial shows you how to set up MCP integration for AI agents using SearchHive as the web data provider. You will have an AI agent that can search, scrape, and research the web in real-time.
Key Takeaways
- MCP (Model Context Protocol) lets AI agents access external tools through a standardized interface
- SearchHive provides a native MCP server that gives AI agents access to SwiftSearch, ScrapeForge, and DeepDive
- Setup takes under 5 minutes: install the npm package, add your API key, configure your MCP client
- Works with Claude Desktop, Cline, Cursor, and any MCP-compatible client
- Free tier includes 500 credits — enough to build and test your agent integration
Prerequisites
Before starting, you need:
- Node.js 18+ (for the MCP server)
- An MCP-compatible AI client (Claude Desktop, Cline VS Code extension, or Cursor)
- A SearchHive API key (get one free — 500 credits, no CC)
- npm (usually included with Node.js)
Step 1: Understand What MCP Does for AI Agents
MCP solves a specific problem: AI models are trained on static data, but users need real-time information. Without MCP, an AI agent cannot:
- Look up current prices
- Read a competitor's website
- Find recent news articles
- Extract data from a specific URL
MCP provides a standardized protocol for the AI agent to call external tools. The agent decides when to use a tool, sends a request through MCP, and gets structured data back. SearchHive's MCP server exposes three capabilities:
- swift_search — real-time web search (SwiftSearch)
- scrape — scrape any URL (ScrapeForge)
- deep_dive — extract structured data from pages (DeepDive)
Step 2: Install the SearchHive MCP Server
The MCP server is an npm package that runs locally and connects to SearchHive's API:
npm install @searchhive/mcp-server
You do not need to start it manually — your MCP client (Claude Desktop, Cline, etc.) will launch it automatically based on your configuration.
Step 3: Get Your SearchHive API Key
If you do not already have one:
- Go to searchhive.dev/signup
- Create a free account (no credit card)
- You get 500 free credits immediately
- Copy your API key from the dashboard
Step 4: Configure Claude Desktop
Claude Desktop is the most common MCP client. Edit your Claude Desktop configuration file:
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json
Add the SearchHive MCP server configuration:
{
"mcpServers": {
"searchhive": {
"command": "npx",
"args": ["@searchhive/mcp-server"],
"env": {
"SEARCHHIVE_API_KEY": "your_api_key_here"
}
}
}
}
Replace your_api_key_here with your actual API key. Restart Claude Desktop, and the SearchHive tools will be available.
Step 5: Configure Cline (VS Code Extension)
Cline is a popular AI coding assistant that supports MCP:
- Install the Cline extension in VS Code
- Open Cline settings (gear icon)
- Go to MCP Servers
- Click "Add MCP Server"
- Enter:
- Name:
searchhive - Command:
npx - Args:
@searchhive/mcp-server - Env:
SEARCHHIVE_API_KEY=your_api_key_here
- Name:
- Save and restart Cline
Step 6: Configure Cursor IDE
Cursor also supports MCP integration:
- Open Cursor settings
- Go to the MCP section
- Add a new MCP server with the same configuration as Claude Desktop
- The SearchHive tools will appear in Cursor's tool menu
Step 7: Use the MCP Tools in Your AI Agent
Once configured, your AI agent can use SearchHive tools automatically. Here are example prompts that trigger MCP usage:
Web Search: "Search for the latest Python web scraping libraries and compare their features"
The agent will call swift_search and return real-time results.
Page Scraping: "Go to https://competitor.com/pricing and extract all their plan details"
The agent will call scrape to get the page content, then extract the pricing information.
Deep Research: "Find the top 5 AI agent frameworks and summarize their documentation"
The agent will chain multiple MCP calls: search for frameworks, scrape their documentation pages, and synthesize the results.
Step 8: Programmatic MCP Usage with Python
If you are building your own AI agent with Python, you can also use SearchHive directly (bypassing MCP for lower latency):
from searchhive import SwiftSearch, ScrapeForge, DeepDive
class WebAgent:
def __init__(self, api_key):
self.search = SwiftSearch(api_key=api_key)
self.scrape = ScrapeForge(api_key=api_key)
self.extract = DeepDive(api_key=api_key)
def research_topic(self, topic, num_sources=5):
# Step 1: Search for relevant sources
results = self.search.search(query=topic, num=num_sources)
# Step 2: Scrape top results for full content
articles = []
for result in results.organic[:3]:
page = self.scrape.scrape(result.url)
articles.append({
"title": result.title,
"url": result.url,
"content": page.content[:2000] # First 2000 chars
})
return articles
def extract_structured_data(self, url):
# Extract structured data from any page
data = self.extract.extract(url)
return {
"title": data.title,
"author": data.author,
"content": data.content,
"headings": [h.text for h in data.headings]
}
agent = WebAgent(api_key="your_api_key")
# Research a topic
articles = agent.research_topic("best practices for RAG systems 2025")
for article in articles:
print(f"\n{article['title']}")
print(f" {article['url']}")
print(f" {len(article['content'])} chars extracted")
# Extract structured data from a specific page
data = agent.extract_structured_data("https://docs.example.com/api-reference")
print(f"\nExtracted {len(data['headings'])} headings")
Common Issues
MCP Server Not Starting
If Claude Desktop cannot start the MCP server, check:
- Node.js is installed and accessible in your PATH
- The npm package installed correctly:
npm list @searchhive/mcp-server - Your API key is valid: test it with
curl -H "Authorization: Bearer your_key" https://api.searchhive.dev/v1/search
Agent Not Using the Tools
Some AI clients need explicit permission to use MCP tools. Make sure:
- The MCP server shows as "connected" in your client's status
- You are using a model that supports tool use (Claude 3.5+ Sonnet, GPT-4, etc.)
- Your prompt implies the need for web data — the agent decides when to call tools
Rate Limit Errors
If you hit rate limits, upgrade your SearchHive plan. The free tier has lower limits, and the Builder plan ($49/mo) provides 100,000 credits with higher throughput.
Next Steps
Now that your AI agent can access web data through MCP, here are ways to extend it:
- Build a competitor monitoring system with automated alerts
- Set up SPA scraping for JavaScript-heavy sites
- Learn automation for monitoring best practices
Get started free — grab your SearchHive API key with 500 credits and connect your first AI agent to the web in minutes. No credit card required.
Full MCP documentation is available at searchhive.dev/docs.