How to Build a Social Media Monitoring API -- Step-by-Step Tutorial

Social media monitoring is how brands, researchers, and investors track mentions, sentiment, and trends across platforms. This tutorial shows you how to build a social media monitoring system using web search and scraping APIs -- without depending on each platform's unreliable official APIs.

Prerequisites

• Python 3.9+
• A SearchHive API key (free, 500 credits)
• Basic familiarity with Python requests and free JSON formatter

Key Takeaways

• Official social media APIs are increasingly restricted -- web search is a more reliable alternative
• SearchHive's SwiftSearch + ScrapeForge can monitor public social content across platforms
• Sentiment analysis can be added with any LLM or open-source model
• The complete monitoring system runs for under $49/month with SearchHive's Builder plan

Step 1: Set Up Your API Client

Start by creating a reusable client for SearchHive's APIs:

import requests
import time
from datetime import datetime, timedelta

SEARCHHIVE_API_KEY = "your_api_key_here"
BASE_URL = "https://api.searchhive.dev/v1"

class SocialMonitor:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.headers = {"Authorization": f"Bearer {api_key}"}

    def _request(self, endpoint: str, payload: dict, timeout: int = 30) -> dict:
        resp = requests.post(
            f"{BASE_URL}/{endpoint}",
            headers=self.headers,
            json=payload,
            timeout=timeout
        )
        resp.raise_for_status()
        return resp.json()

    def search_mentions(self, query: str, limit: int = 20) -> list[dict]:
        """Search for social media mentions using SwiftSearch."""
        social_query = f"{query} site:twitter.com OR site:x.com OR site:reddit.com OR site:linkedin.com"
        result = self._request("swift-search", {
            "query": social_query,
            "limit": limit,
            "format": "markdown",
            "recency": "week"
        })
        return result.get("results", [])

Step 2: Search for Brand Mentions

The core of social monitoring is finding when people talk about your brand, product, or keywords. SwiftSearch lets you target specific social platforms:

monitor = SocialMonitor(SEARCHHIVE_API_KEY)

# Search for brand mentions across social platforms
mentions = monitor.search_mentions(
    query = '"SearchHive" OR "Search Hive" API scraping',
    limit = 20
)

for m in mentions:
    print(f"[{m.get('source', 'unknown')}] {m.get('title', 'No title')}")
    print(f"  URL: {m.get('url', '')}")
    print(f"  Snippet: {m.get('snippet', '')[:200]}")
    print()

SwiftSearch returns results from Twitter/X, Reddit, LinkedIn, and other platforms in a unified format. The recency: "week" parameter ensures you get recent mentions only.

Step 3: Extract Full Content from Mentions

Snippets give you a preview, but for sentiment analysis or detailed monitoring, you need the full content. Use ScrapeForge to extract complete posts:

    def get_full_mention(self, url: str) -> dict:
        """Extract full content from a social media post URL."""
        result = self._request("scrapeforge", {
            "url": url,
            "format": "markdown",
            "render_js": True,
            "extract": {
                "prompt": "Extract: author name, post text, date posted, number of likes/replies/shares, and any replies visible on the page"
            }
        })
        return result

# Get full content for each mention
for mention in mentions[:5]:  # Process top 5
    url = mention.get("url")
    if url:
        full = monitor.get_full_mention(url)
        print(f"Full post: {full.get('content', '')[:500]}")
        time.sleep(1)  # Be polite between requests

The AI extraction prompt tells ScrapeForge to pull structured data from the page, including engagement metrics (likes, replies, shares).

Step 4: Add Sentiment Analysis

Sentiment analysis tells you whether mentions are positive, negative, or neutral. You can use any LLM for this:

    def analyze_sentiment(self, text: str) -> dict:
        """Analyze sentiment of a social media post."""
        import json
        # Use your preferred LLM -- OpenAI, Claude, local model, etc.
        prompt = f"""Analyze the sentiment of this social media post.
Return JSON with: sentiment (positive/negative/neutral), confidence (0-1), topics (list of strings), urgency (low/medium/high)

POST: {text[:1000]}"""

        # Example using OpenAI (replace with your LLM call)
        resp = requests.post(
            "https://api.openai.com/v1/chat/completions",
            headers={"Authorization": f"Bearer {OPENAI_API_KEY}"},
            json={
                "model": "gpt-4o-mini",
                "messages": [{"role": "user", "content": prompt}],
                "temperature": 0
            },
            timeout=15
        )
        content = resp.json()["choices"][0]["message"]["content"]
        # Parse the JSON from the response
        return json.loads(content)

# Analyze sentiment of mentions
for mention in mentions[:10]:
    text = mention.get("snippet", "")
    sentiment = monitor.analyze_sentiment(text)
    print(f"{sentiment['sentiment']} ({sentiment['confidence']:.0%}) - {mention.get('title', '')[:80]}")

Step 5: Track Mentions Over Time

A monitoring system needs to track mentions over time and alert on spikes. Store results in a simple database or file:

import json
from pathlib import Path

class MentionTracker:
    def __init__(self, storage_path: str = "mentions.json"):
        self.path = Path(storage_path)
        self.data = self._load()

    def _load(self) -> dict:
        if self.path.exists():
            return json.loads(self.path.read_text())
        return {"mentions": [], "daily_counts": {}}

    def add_mentions(self, mentions: list[dict], sentiment_results: list[dict]):
        today = datetime.now().strftime("%Y-%m-%d")
        for mention, sentiment in zip(mentions, sentiment_results):
            entry = {
                **mention,
                "sentiment": sentiment.get("sentiment", "neutral"),
                "scraped_at": datetime.now().isoformat()
            }
            self.data["mentions"].append(entry)

        # Update daily count
        self.data["daily_counts"][today] = len(mentions)

        # Keep last 30 days of data
        cutoff = (datetime.now() - timedelta(days=30)).isoformat()
        self.data["mentions"] = [
            m for m in self.data["mentions"]
            if m["scraped_at"] > cutoff
        ]

        self.path.write_text(json.dumps(self.data, indent=2))

    def get_daily_summary(self) -> dict:
        today = datetime.now().strftime("%Y-%m-%d")
        today_mentions = [
            m for m in self.data["mentions"]
            if m["scraped_at"].startswith(today)
        ]
        positive = sum(1 for m in today_mentions if m["sentiment"] == "positive")
        negative = sum(1 for m in today_mentions if m["sentiment"] == "negative")
        return {
            "date": today,
            "total_mentions": len(today_mentions),
            "positive": positive,
            "negative": negative,
            "neutral": len(today_mentions) - positive - negative
        }

Step 6: Set Up Automated Monitoring

Run your monitor on a schedule using Python's schedulers or cron expression generator:

import schedule

tracker = MentionTracker()

def run_monitoring_cycle():
    print(f"Running monitoring cycle at {datetime.now()}")
    try:
        # Search for new mentions
        mentions = monitor.search_mentions(
            query='"YourBrand" OR "your product"',
            limit=20
        )

        if mentions:
            # Analyze sentiment for each
            sentiments = [
                monitor.analyze_sentiment(m.get("snippet", ""))
                for m in mentions
            ]

            # Track and store
            tracker.add_mentions(mentions, sentiments)

            # Alert on spike
            summary = tracker.get_daily_summary()
            if summary["negative"] > 5:
                send_alert(f"High negative sentiment spike: {summary['negative']} negative mentions today")

            print(f"  Found {len(mentions)} mentions. Daily total: {summary['total_mentions']}")
        else:
            print("  No new mentions found.")
    except Exception as e:
        print(f"  Error: {e}")

# Run every 2 hours
schedule.every(2).hours.do(run_monitoring_cycle)

if __name__ == "__main__":
    run_monitoring_cycle()  # Run once immediately
    while True:
        schedule.run_pending()
        time.sleep(60)

Step 7: Build a Dashboard View

Generate a daily summary that can be displayed in a dashboard or sent via email/Slack:

def generate_daily_report(tracker: MentionTracker) -> str:
    summary = tracker.get_daily_summary()
    recent = tracker.data["mentions"][-10:]  # Last 10 mentions

    report = f"""# Social Media Monitoring Report - {summary['date']}

## Summary
- Total mentions: {summary['total_mentions']}
- Positive: {summary['positive']} | Negative: {summary['negative']} | Neutral: {summary['neutral']}

## Recent Mentions
"""
    for m in reversed(recent):
        emoji = {"positive": "🟢", "negative": "🔴", "neutral": "⚪"}.get(m["sentiment"], "⚪")
        report += f"- {emoji} {m.get('title', 'No title')[:80]}\n"
        report += f"  Source: {m.get('source', 'unknown')} | {m.get('url', '')}\n\n"

    return report

Common Issues

Platform blocking: Social media sites aggressively block scrapers. SearchHive handles proxy rotation and anti-bot measures, but if you're hitting rate limits, reduce your query frequency or increase the delay between requests.

JavaScript-rendered content: Most social platforms render content dynamically. Always set render_js: True in ScrapeForge requests.

Rate limits: SearchHive's free tier has 500 credits. The $49/month Builder plan (100K credits) handles hundreds of monitoring cycles per month comfortably. See /compare/serpapi for a cost comparison.

Sentiment accuracy: LLM-based sentiment analysis is ~85-90% accurate for social media text. For higher accuracy, fine-tune a model on your specific domain.

Next Steps

1. Add more platforms: Expand your queries to include Mastodon, Threads, and niche forums
2. Competitor monitoring: Track competitor mentions alongside your own
3. Image/content monitoring: Use DeepDive for multi-page research on trending topics
4. Real-time alerts: Integrate with Slack, Discord, or email for instant notifications

Summary

You can build a complete social media monitoring system with SearchHive's APIs in under 200 lines of Python. The combination of SwiftSearch for discovery and ScrapeForge for content extraction covers the full monitoring pipeline, from finding mentions to analyzing sentiment.

Start with 500 free credits -- no credit card required. Scale to production with the $49/month Builder plan that includes 100K credits for search, scraping, and research. Check out the docs for API references and code examples.

For more on building data pipelines, see /blog/data-extraction-from-websites-common-questions-answered and /blog/automation-retry-strategies-common-questions-answered.