A growing share of the web renders content dynamically. React, Vue, Svelte, and Angular frameworks ship minimal HTML to the browser and populate the page via JavaScript. Standard HTTP requests — the kind Python's requests library makes — see an empty shell instead of actual content.
JavaScript rendering APIs solve this by running a headless browser on the server side, executing the page's JavaScript, and returning the fully rendered HTML or converted markdown. This guide covers how they work, which ones perform best, and how to integrate them into your scraping pipeline.
Key Takeaways
- Roughly 70% of modern websites require JavaScript execution to display meaningful content
- SearchHive ScrapeForge handles JS rendering automatically — set
render_js=trueand get rendered content - Headless browser rendering adds 2-10 seconds per request compared to static HTML fetching
- Stealth detection matters — sites use browser fingerprinting to detect headless Chrome
- Serverless rendering APIs (SearchHive, ScrapingBee, ZenRows) are cheaper than self-hosting for most workloads
Why JavaScript Rendering Matters
When you send an HTTP request to a server-side rendered page (like a traditional WordPress site), the response contains the complete HTML. Everything you need is in the initial response.
When you request a client-side rendered page (React SPA, Next.js with useSWR, infinite scroll), the initial HTML is a shell — a <div id="root"></div> with a JavaScript bundle that constructs the actual page in the browser.
Without JS rendering, your scraper gets:
- Empty content divs
- Loading spinners
- No product data, no prices, no reviews
This is why JavaScript rendering APIs exist. They run a real browser engine on the server, wait for the page to finish loading, and return the fully rendered content.
Detecting Whether a Page Needs JS Rendering
Before burning credits on headless rendering, check whether a page actually requires it. This saves both cost and latency.
import requests
def needs_js_rendering(url):
# Fetch with a basic HTTP client
resp = requests.get(url, timeout=10)
html = resp.text
# Heuristic checks for JS-rendered content
signals = [
'<div id="root"></div>' in html, # React root
'<div id="__next"></div>' in html, # Next.js
'window.__NUXT__' in html, # Nuxt.js
'<noscript>' in html and len(html) < 5000, # Fallback content only
'Loading' in html and html.count('<') < 50, # Minimal HTML with loading text
]
return any(signals)
# Check a URL before deciding whether to use JS rendering
url = "https://example.com/products"
if needs_js_rendering(url):
print("JS rendering needed — use ScrapeForge with render_js=true")
else:
print("Static HTML — use a simple request")
This pre-check avoids unnecessary headless rendering on static pages, cutting your API costs significantly.
SearchHive ScrapeForge — Integrated JS Rendering
SearchHive's ScrapeForge API runs headless Chrome with stealth patches, executes the page's JavaScript, and returns clean markdown or raw HTML. Anti-bot bypass and proxy rotation are built in.
import requests
API_KEY = "your-searchhive-key"
# Render a JavaScript-heavy page
resp = requests.get(
"https://api.searchhive.dev/v1/scrape",
headers={"Authorization": f"Bearer {API_KEY}"},
params={
"url": "https://spa-example.com/products",
"render_js": "true",
"wait_for": 3000, # Wait 3 seconds for JS to execute
"format": "markdown"
}
)
data = resp.json()
print(data["markdown"][:1000])
# Combine search + scrape: find JS-rendered pages then extract them
search_resp = requests.get(
"https://api.searchhive.dev/v1/swift/search",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"query": "site:example.com products", "limit": 5}
)
for result in search_resp.json().get("results", []):
scrape_resp = requests.get(
"https://api.searchhive.dev/v1/scrape",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"url": result["url"], "render_js": "true", "format": "markdown"}
)
print(f"--- {result['title']} ---")
print(scrape_resp.json().get("markdown", "")[:300])
Key features:
- Stealth-mode headless Chrome (evades browser fingerprinting)
- Configurable wait time and wait-for-selectors
- Proxy rotation included
- Markdown conversion built in
- Same credit system as search and AI extraction
Pricing: Credits work across all APIs. A JS-rendered scrape costs more credits than a static fetch, but you're not paying a separate service for it.
/blog/best-apis-for-bulk-web-scraping-at-scale
ScrapingBee — Headless Browser Specialist
ScrapingBee focuses on headless browser rendering with rotating proxies. Their API is straightforward — send a URL, get rendered HTML back.
import requests
resp = requests.get(
"https://app.scrapingbee.com/api/v1/",
params={
"api_key": "YOUR_KEY",
"url": "https://example.com",
"render_js": "true",
"wait": 3000,
"premium_proxy": "true"
}
)
print(resp.text[:500])
Pricing: JS-rendered requests start at ~$0.05/request. At 100K pages with JS rendering, expect costs around $4,900/month. That's 100x what SearchHive charges for equivalent volume.
Strengths: Reliable rendering, good proxy network, screenshot API Weaknesses: Very expensive at scale, no markdown conversion, no AI extraction
ZenRows — Anti-Bot + Rendering
ZenRows combines anti-bot bypass with headless browser rendering. Their fingerprint rotation and proxy network handle even aggressive bot protection.
import requests
resp = requests.get(
"https://api.zenrows.com/v1/",
params={
"apikey": "YOUR_KEY",
"url": "https://protected-site.com",
"js_render": "true",
"antibot": "true",
"premium_proxies": "true"
}
)
print(resp.text[:500])
Pricing: $49/50K requests. JS rendering and anti-bot both increase per-request cost. Expect 2-3x the base rate for fully protected JS-rendered pages.
Strengths: Best anti-bot bypass in the market Weaknesses: Expensive with all features enabled, no markdown or AI features
Self-Hosted: Playwright and Puppeteer
If you control the infrastructure, running your own headless browser is an option. Playwright (Python/Node) and Puppeteer (Node) give you full control over browser behavior.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://example.com")
page.wait_for_load_state("networkidle")
content = page.content()
browser.close()
print(content[:1000])
Cost: Free software, but you pay for servers, proxies, and maintenance. At scale, a single headless Chrome instance uses 200-500MB RAM. Running 100 concurrent instances requires significant infrastructure.
Strengths: Full control, no per-request costs Weaknesses: Infrastructure management, proxy rotation, anti-detection patches, scaling complexity
Performance Considerations
Latency: JS-rendered requests take 2-10 seconds depending on page complexity. Static HTML fetching completes in 200-500ms. Budget your time accordingly.
Concurrency: Most rendering APIs limit concurrent browser instances. SearchHive handles this transparently with queue management. Self-hosted solutions need explicit orchestration.
Cost optimization:
- Pre-check pages for JS rendering needs (see detection code above)
- Cache rendered results for pages that don't change frequently
- Use
wait_forselectors instead of fixed delays when possible - Batch requests to reduce per-request overhead
Best Practices
- Always test with and without JS rendering before committing to headless mode. Many pages serve static HTML for the initial content and only add interactive elements via JS.
- Set appropriate wait times. Too short and the content isn't loaded. Too long and you're burning credits on idle browser time. 2-5 seconds covers most cases.
- Handle infinite scroll carefully. JS-rendered pages often load more content as you scroll. You may need to simulate scroll events or use the API's pagination/crawl features.
- Monitor for rendering failures. JavaScript errors, network timeouts, and memory limits in headless browsers can cause silent failures. Log and retry these cases.
Getting Started with SearchHive
SearchHive's ScrapeForge API handles JavaScript rendering, anti-bot bypass, and data extraction in one call. Sign up for free credits and start scraping JS-rendered pages in minutes.
Get 500 free credits — no credit card. Documentation includes examples for rendering, extraction, and search + scrape workflows.