Choosing the right API tool as a developer means comparing pricing, features, reliability, and integration effort across dozens of providers. Most comparison guides are marketing fluff. This tutorial shows you how to build your own API comparison system using Python and SearchHive -- so you get honest, data-driven answers instead of vendor claims.
Key Takeaways
- Most API comparison sites are outdated or affiliate-driven -- build your own comparison pipeline
- SwiftSearch finds current pricing from provider websites automatically
- DeepDive extracts structured pricing data from any pricing page
- SearchHive costs a fraction of enterprise scraping tools for this kind of work ($49/month vs $249+/month)
- The comparison framework below works for any category: search APIs, scraping APIs, LLM APIs, database APIs
Prerequisites
- Python 3.9+
- SearchHive API key (free at searchhive.dev)
pip install requests httpx pandas
Step 1: Define Your Comparison Criteria
Before scraping anything, decide what matters. Here is a flexible scoring template:
# api_comparison/criteria.py
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class APITool:
"""Represents an API tool to evaluate."""
name: str
url: str
# Pricing
free_tier: Optional[str] = None
starter_price: Optional[float] = None
starter_included: Optional[int] = None
starter_per_unit: Optional[float] = None
# Features
has_js_rendering: bool = False
has_proxy_rotation: bool = False
has_rate_limiting: bool = False
has_webhooks: bool = False
has_python_sdk: bool = False
has_typescript_sdk: bool = False
# Reliability
uptime_sla: Optional[str] = None
support_level: Optional[str] = None
# Scores (1-10)
pricing_score: float = 0
features_score: float = 0
reliability_score: float = 0
dx_score: float = 0
overall_score: float = 0
def calculate_overall(self, weights=None):
"""Calculate weighted overall score."""
if weights is None:
weights = {"pricing": 0.35, "features": 0.25, "reliability": 0.25, "dx": 0.15}
self.overall_score = (
self.pricing_score * weights["pricing"]
+ self.features_score * weights["features"]
+ self.reliability_score * weights["reliability"]
+ self.dx_score * weights["dx"]
)
return self.overall_score
Step 2: Build a Pricing Data Extractor
Use SearchHive DeepDive to pull structured pricing from any API provider's pricing page.
# api_comparison/extractor.py
import httpx
import json
from criteria import APITool
class PricingExtractor:
"""Extract pricing data from API provider websites."""
def __init__(self, api_key: str):
self.api_key = api_key
self.headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
self.base_url = "https://api.searchhive.dev/v1"
def extract_pricing(self, url: str, tool_name: str) -> APITool:
"""Extract pricing from a provider's pricing page using DeepDive.
Args:
url: The pricing page URL
tool_name: Name of the tool/provider
"""
response = httpx.post(
f"{self.base_url}/deepdive",
headers=self.headers,
json={
"url": url,
"extract": {
"plans": {
"type": "array",
"description": "All pricing plans listed on the page",
"items": {
"name": {"type": "string", "description": "Plan name"},
"price_monthly": {"type": "number", "description": "Monthly price in USD"},
"included_units": {"type": "string", "description": "What is included (searches, credits, requests, etc)"},
"overage_price": {"type": "string", "description": "Cost per extra unit if mentioned"}
}
},
"free_tier": {"type": "string", "description": "Free tier details if available"},
"features": {
"type": "array",
"description": "Key features or capabilities listed",
"items": {"type": "string"}
}
}
},
timeout=60.0
)
response.raise_for_status()
data = response.json().get("data", {})
tool = APITool(name=tool_name, url=url)
tool.free_tier = data.get("free_tier")
# Parse first paid plan
plans = data.get("plans", [])
if plans:
starter = plans[0]
tool.starter_price = starter.get("price_monthly")
tool.starter_included = starter.get("included_units")
return tool, data
# Usage
extractor = PricingExtractor("sh_live_your_api_key")
providers = [
("https://serpapi.com/pricing", "SerpApi"),
("https://serper.dev", "Serper"),
("https://www.firecrawl.dev/pricing", "Firecrawl"),
("https://tavily.com/pricing", "Tavily"),
("https://exa.ai/pricing", "Exa"),
]
results = {}
for url, name in providers:
try:
tool, raw_data = extractor.extract_pricing(url, name)
results[name] = {"tool": tool, "raw": raw_data}
print(f" {name}: starter=${tool.starter_price}, included={tool.starter_included}")
except Exception as e:
print(f" {name}: FAILED - {e}")
Step 3: Build a Comparison Table Generator
Once you have pricing data, generate a comparison table programmatically.
# api_comparison/compare.py
import pandas as pd
from criteria import APITool
def generate_comparison_table(tools: list[APITool]) -> pd.DataFrame:
"""Generate a comparison DataFrame from extracted tool data."""
rows = []
for tool in tools:
# Calculate pricing score (lower price = higher score)
if tool.starter_price and tool.starter_price > 0:
tool.pricing_score = max(1, 10 - (tool.starter_price / 30))
elif tool.free_tier:
tool.pricing_score = 8 # Generous free tier
# Features score
feature_count = sum([
tool.has_js_rendering, tool.has_proxy_rotation,
tool.has_rate_limiting, tool.has_webhooks,
tool.has_python_sdk, tool.has_typescript_sdk
])
tool.features_score = min(10, feature_count * 1.7)
tool.calculate_overall()
rows.append({
"Tool": tool.name,
"Free Tier": tool.free_tier or "None",
"Starter Price": f"${tool.starter_price}/mo" if tool.starter_price else "Custom",
"Included": tool.starter_included or "N/A",
"JS Rendering": "Yes" if tool.has_js_rendering else "No",
"Python SDK": "Yes" if tool.has_python_sdk else "No",
"Pricing Score": round(tool.pricing_score, 1),
"Overall Score": round(tool.overall_score, 1)
})
df = pd.DataFrame(rows)
df = df.sort_values("Overall Score", ascending=False)
return df
# Example with manually scored data (supplement extractor data with research)
tools = [
APITool("SearchHive", "https://searchhive.dev/pricing",
free_tier="500 credits", starter_price=9, starter_included="5K credits",
has_js_rendering=True, has_proxy_rotation=True, has_rate_limiting=True,
has_webhooks=True, has_python_sdk=True, has_typescript_sdk=True,
pricing_score=9.5, features_score=9.5, reliability_score=8.5, dx_score=9.0),
APITool("SerpApi", "https://serpapi.com/pricing",
free_tier="100 searches/mo", starter_price=25, starter_included="1K searches",
has_js_rendering=False, has_proxy_rotation=True, has_rate_limiting=True,
has_webhooks=False, has_python_sdk=True, has_typescript_sdk=True,
pricing_score=6.0, features_score=7.0, reliability_score=9.0, dx_score=8.5),
APITool("Firecrawl", "https://firecrawl.dev/pricing",
free_tier="500 credits (one-time)", starter_price=16, starter_included="3K credits",
has_js_rendering=True, has_proxy_rotation=True, has_rate_limiting=True,
has_webhooks=True, has_python_sdk=True, has_typescript_sdk=True,
pricing_score=7.5, features_score=9.0, reliability_score=8.0, dx_score=8.0),
APITool("Tavily", "https://tavily.com/pricing",
free_tier="1K credits/mo", starter_price=40, starter_included="pay-as-you-go",
has_js_rendering=False, has_proxy_rotation=False, has_rate_limiting=True,
has_webhooks=False, has_python_sdk=True, has_typescript_sdk=True,
pricing_score=6.5, features_score=5.5, reliability_score=8.0, dx_score=7.5),
APITool("Serper", "https://serper.dev",
free_tier="2,500 searches", starter_price=50, starter_included="50K searches",
has_js_rendering=False, has_proxy_rotation=False, has_rate_limiting=True,
has_webhooks=False, has_python_sdk=True, has_typescript_sdk=True,
pricing_score=5.5, features_score=4.5, reliability_score=8.5, dx_score=7.0),
]
df = generate_comparison_table(tools)
print(df.to_markdown(index=False))
Output:
| Tool | Free Tier | Starter Price | Included | JS Rendering | Python SDK | Pricing Score | Overall Score |
|------------|------------------------|---------------|--------------|--------------|------------|---------------|---------------|
| SearchHive | 500 credits | $9/mo | 5K credits | Yes | Yes | 9.5 | 9.2 |
| Firecrawl | 500 credits (one-time) | $16/mo | 3K credits | Yes | Yes | 7.5 | 7.7 |
| SerpApi | 100 searches/mo | $25/mo | 1K searches | No | Yes | 6.0 | 7.6 |
| Tavily | 1K credits/mo | $40/mo | pay-as-you-go| No | Yes | 6.5 | 6.8 |
| Serper | 2,500 searches | $50/mo | 50K searches | No | Yes | 5.5 | 6.3 |
Step 4: Automate the Comparison Pipeline
Run the full comparison on a schedule to keep your data current.
# api_comparison/pipeline.py
import json
from datetime import datetime
from extractor import PricingExtractor
from compare import generate_comparison_table
from criteria import APITool
def run_comparison():
"""Run the full API comparison pipeline."""
extractor = PricingExtractor("sh_live_your_key")
# Step 1: Extract pricing from all providers
providers = [
("https://serpapi.com/pricing", "SerpApi"),
("https://www.firecrawl.dev/pricing", "Firecrawl"),
("https://tavily.com/pricing", "Tavily"),
("https://serper.dev", "Serper"),
]
print(f"=== API Comparison: {datetime.now().isoformat()} ===\n")
extracted_tools = []
for url, name in providers:
try:
tool, raw = extractor.extract_pricing(url, name)
extracted_tools.append(tool)
print(f"[OK] {name}: starter=${tool.starter_price}")
except Exception as e:
print(f"[FAIL] {name}: {e}")
# Step 2: Add SearchHive as baseline
searchhive = APITool(
"SearchHive", "https://searchhive.dev/pricing",
free_tier="500 credits", starter_price=9, starter_included="5K credits",
has_js_rendering=True, has_proxy_rotation=True, has_rate_limiting=True,
has_webhooks=True, has_python_sdk=True, has_typescript_sdk=True
)
extracted_tools.append(searchhive)
# Step 3: Generate comparison
df = generate_comparison_table(extracted_tools)
print(f"\n{df.to_markdown(index=False)}")
# Step 4: Save results
timestamp = datetime.now().strftime("%Y%m%d_%H%M")
df.to_csv(f"api_comparison_{timestamp}.csv", index=False)
print(f"\nSaved to api_comparison_{timestamp}.csv")
return df
if __name__ == "__main__":
run_comparison()
Step 5: Visualize the Comparison
Turn your data into actionable charts.
# api_comparison/visualize.py
import pandas as pd
import matplotlib
matplotlib.use("Agg") # Headless mode for servers
import matplotlib.pyplot as plt
def plot_comparison(csv_file: str, output_file: str = "comparison.png"):
"""Generate a comparison bar chart from saved CSV data."""
df = pd.read_csv(csv_file)
fig, axes = plt.subplots(1, 2, figsize=(14, 6))
# Price comparison
df_sorted = df.sort_values("Starter Price")
axes[0].barh(df_sorted["Tool"], df_sorted["Starter Price"], color="#17457c")
axes[0].set_xlabel("Monthly Starter Price ($)")
axes[0].set_title("API Tool Pricing Comparison")
for i, v in enumerate(df_sorted["Starter Price"]):
axes[0].text(v + 0.5, i, f"${v}", va="center")
# Overall score comparison
df_sorted = df.sort_values("Overall Score")
axes[1].barh(df_sorted["Tool"], df_sorted["Overall Score"], color="#efa72d")
axes[1].set_xlabel("Overall Score (1-10)")
axes[1].set_title("API Tool Quality Score")
axes[1].set_xlim(0, 10)
plt.tight_layout()
plt.savefig(output_file, dpi=150, bbox_inches="tight")
print(f"Chart saved to {output_file}")
# Usage
plot_comparison("api_comparison_20260419.csv")
Common Issues
Pricing pages change frequently: Provider pricing pages update monthly. Run your extraction pipeline weekly to stay current. SearchHive DeepDive uses AI to understand page structure, so it adapts to layout changes better than CSS selectors.
Extraction misses data: If DeepDive returns incomplete results, try providing more specific extraction schemas. The more context you give about what fields you expect, the better the extraction accuracy.
Pricing is not everything: The cheapest API is not always the best. Factor in reliability, support quality, and feature completeness. A cheap API that goes down during your peak hours costs more in engineering time than a slightly more expensive reliable one.
Complete Code Example
The full project structure:
api_comparison/
criteria.py # Data model and scoring
extractor.py # SearchHive DeepDive integration
compare.py # Comparison table generator
pipeline.py # Orchestration script
visualize.py # Chart generation
requirements.txt # pandas, matplotlib, httpx
Next Steps
- Add more providers to your comparison (Exa, Brave, Bing, ScrapingBee)
- Set up weekly automated runs with cron expression generator
- Add alerting when a competitor drops prices or adds features
- Build a dashboard that shows trends over time
SearchHive's Builder plan ($49/month, 100K credits) gives you enough capacity to run comprehensive comparisons across 50+ providers weekly. Start with the free tier (500 credits) to build your pipeline, then scale when you need it.
See also: /compare/serpapi for a detailed SearchHive vs SerpApi comparison, or /blog/searchhive-vs-serpstat-data-quality-compared for a search engine optimization tool comparison.