Extract Semrush domain authority, organic and paid traffic, top keywords, backlinks, AI-search visibility, and competitor data for any domain — in Python, no Semrush login required.
Working Python example that fetches the full Semrush domain overview through the Semrush All-in-One Scraper Apify actor. One actor call batches a domain's organic search posture, paid ad spend, top keywords by country, backlink profile, AI-search referral traffic, top competitors, and (optionally) Moz Domain Authority + Spam Score — every signal SEO teams pull from Semrush, in a single JSON record.
Semrush's official Domain Overview API and Trends API cost upwards of $140–$450 per month with per-call quotas on each endpoint. This example runs the same kind of audit through Apify's pay-as-you-go pricing: pay $0.50–$2 per 1,000 records, no monthly minimum. The actor batches every endpoint Semrush serves for one domain into a single JSON-RPC call, so a 100-domain audit completes in well under a minute. Use it for competitive intelligence, monthly SEO reporting, lead scoring, or simply to keep a private rolling history of how your competitors' organic traffic moves over time.
- Monthly SEO competitor reporting — script a Monday-morning snapshot of 20 competitor domains and post the diff into Slack.
- Sales territory planning — pull organic + paid traffic estimates for prospects in your CRM and prioritize outreach by domain authority.
- Investment due diligence — verify a target company's reported organic traffic against an independent Semrush snapshot before signing.
- Content gap analysis — fetch competitors' organic keyword count and traffic per keyword to find under-served topics.
- Backlink prospecting — combine Semrush + Moz to identify high-DA referring domains worth approaching for outreach.
- Market sizing for new verticals — use
top_websitesmode with an industry filter to enumerate the top 1,000 sites in a vertical and country.
- Python 3.10+
- A free Apify account (gives you $5 of free monthly credits)
- No Semrush account needed
git clone https://github.com/pro100chok/semrush-domain-overview-python.git
cd semrush-domain-overview-python
pip install -r requirements.txt
cp .env.example .env
# Paste APIFY_API_TOKEN from console.apify.com/settings/integrations
python main.pymain.py benchmarks five direct-to-consumer brands (Warby Parker, Casper, Allbirds, Harry's, Glossier) — organic vs paid traffic, total backlinks, Moz Domain Authority — and dumps everything to output.json plus a flat output.csv. Edit the BRANDS list to point at your own competitors.
The actor supports two ways of asking Semrush for data:
| Mode | What you pass in | What you get out |
|---|---|---|
domain |
A list of up to 100 domains | A full Semrush snapshot per domain: organic, paid, backlinks, AI search, top keywords, competitors. |
top_websites |
Country + industry filters | A leaderboard of the highest-trafficked websites in that segment. |
main.py posts a JSON request to the pro100chok/semrush-scraper actor via the official apify-client. The actor runs server-side on Apify's residential proxy infrastructure, hits Semrush's internal JSON-RPC endpoints in a single batched call per domain, parses the response into a strongly-typed schema, and pushes one record per domain to a dataset. The Python script then iterates the dataset and writes it to disk.
If you set include_moz: true, the actor additionally fetches Moz's Domain Authority, Spam Score, top linking domains, and ranking distribution through a separate Moz endpoint (Cloudflare-protected, served via residential proxy).
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_API_TOKEN"])
run = client.actor("pro100chok/semrush-scraper").call(run_input={
"mode": "domain",
"urls": ["asana.com", "monday.com", "trello.com", "clickup.com", "notion.so"],
"database": "us",
"include_moz": True,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
overview = item["overview"]
print(
f"{item['domain']:<18} "
f"organic: {overview['organic']['traffic']:>10,} "
f"keywords: {overview['organic']['keywords']:>7,} "
f"DA: {item.get('moz', {}).get('domain_authority', '—')}"
){
"domain": "asana.com",
"overview": {
"organic": {
"traffic": 3215460,
"keywords": 412300,
"cost": 412500.0,
"top_keywords": [
{ "keyword": "asana", "position": 1, "search_volume": 1500000 },
{ "keyword": "project management", "position": 4, "search_volume": 165000 }
]
},
"paid": { "traffic": 88200, "keywords": 410, "cost": 35200.0 },
"backlinks": { "total": 25430800, "ref_domains": 92140 },
"ai_search": { "traffic": 14200 },
"competitors": [
{ "domain": "monday.com", "common_keywords": 8100 },
{ "domain": "clickup.com", "common_keywords": 7200 }
]
},
"moz": {
"domain_authority": 87,
"spam_score": 1,
"top_linking_domains": [
{ "domain": "github.com", "domain_authority": 95 }
]
}
}| Parameter | Type | Required | Description |
|---|---|---|---|
mode |
string | yes | Either domain (per-domain overview) or top_websites (industry leaderboard). |
urls |
string[] | when mode=domain |
Up to 100 domains. Each one triggers a single batched JSON-RPC call to Semrush. |
database |
string | no | Country database (us, gb, de, ru, etc.) — controls which country's organic/paid data is returned. Default worldwide. |
include_moz |
boolean | no | If true, also fetch Moz Domain Authority, Spam Score, top linking domains. Slower. Default false. |
country_top_websites |
string | when mode=top_websites |
Country filter for the leaderboard. |
industry_top_websites |
string | when mode=top_websites |
Industry filter (e.g. finance, retail, healthcare, ecommerce). |
concurrency |
integer | no | Number of domains processed in parallel. Default 5. |
| Path | Meaning |
|---|---|
overview.organic.traffic |
Monthly organic visit estimate. |
overview.organic.keywords |
Number of keywords the domain ranks for. |
overview.organic.top_keywords[] |
Highest-traffic ranking keywords with position and volume. |
overview.paid.traffic |
Estimated paid traffic from Google Ads. |
overview.paid.cost |
Estimated monthly ad spend. |
overview.backlinks.total |
Total backlinks pointing to the domain. |
overview.backlinks.ref_domains |
Unique referring domains. |
overview.ai_search.traffic |
Referral traffic from AI engines (ChatGPT, Perplexity, etc.). |
overview.competitors[] |
Top organic competitors by common-keyword count. |
moz.domain_authority |
Moz DA (only if include_moz: true). |
| File | Demonstrates |
|---|---|
examples/01_basic_usage.py |
Single-domain overview in 12 lines. |
examples/02_top_websites_by_industry.py |
Enumerate the top 50 finance sites in the US. |
examples/03_backlink_audit.py |
Backlink profile + Moz DA + top linking domains. |
examples/04_export_to_csv.py |
Bulk SaaS audit with pandas filtering and sorting. |
examples/05_export_to_google_sheets.py |
Append snapshots to a shared Google Sheet. |
How much does this cost? Semrush domain records are charged per result. Apify's free tier ($5/month) covers a few thousand domains per month before you pay anything.
Do I need a Semrush login or API token? No. The actor runs against Semrush's public-facing endpoints through residential proxies — no Semrush account needed.
How accurate is the data compared with Semrush's UI? The numbers are pulled from the same internal endpoints that power semrush.com — organic traffic, keywords, backlinks, and Moz metrics match the values you'd see in a paid Semrush dashboard.
Why batch through Apify instead of using Semrush's official API?
The official Semrush API tiers reportedly start around $140/month and charge per endpoint call (Domain Overview, Keyword Analytics, Backlinks are all separate). This actor returns all of those in one record per domain, on a true pay-as-you-go pricing model.
Can I monitor a domain on a schedule? Yes. Use Apify's scheduler to run the actor daily/weekly and feed the dataset into your data warehouse via Apify's webhooks or Google Sheets integration.
What countries are supported?
A hundred-plus — every major Semrush country database including US, UK, all of EU, India, Brazil, Russia, Japan, and most of Latin America. See the database enum in the actor's input schema.
Can I use my own proxies?
The actor uses Apify Proxy by default. If you have a higher-tier Apify plan, you can pass proxyConfiguration with custom URLs.
What about pricing for top_websites mode?
top_websites returns a list of domains in your chosen country + industry. It's charged the same way — per result item — and supports limits up to 1,000 results per call.
- SimilarWeb Traffic Data Scraper — same domain list, different data source.
- Ahrefs All-in-One SEO Scraper — domain rating, backlinks, keyword difficulty.
- Website Contact Scraper — enrich competitor domains with bulk emails / phones / socials.
See all my actors at apify.com/pro100chok.
MIT — see LICENSE.
Built on top of the Semrush All-in-One Scraper Apify actor.