senrecep/web-scraping-comparison.md

Created March 12, 2026 12:41

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/senrecep/2aff258a6978347e962b843eff247155.js"></script>
Save senrecep/2aff258a6978347e962b843eff247155 to your computer and use it in GitHub Desktop.

Web Scraping Services Comparison: Cloudflare Browser Rendering vs Firecrawl vs Scrapling (features + cost + code examples)

Raw

web-scraping-comparison.md

Web Scraping Services Comparison: Cloudflare Browser Rendering vs Firecrawl vs Scrapling

Date: 2026-03-12 | Author: @senrecep

1. Overview
2. Cloudflare Browser Rendering - /crawl Endpoint
3. Firecrawl
4. Scrapling
5. Feature Comparison
6. Pricing Comparison
7. Infrastructure Cost (Scrapling Self-Host)
8. Total Cost of Ownership (TCO)
9. Decision Matrix
10. Code Examples

1. Overview

	Cloudflare Browser Rendering	Firecrawl	Scrapling
Type	Infrastructure service (Workers ecosystem)	Managed scraping SaaS	Open-source Python framework
Approach	Low-level, flexible	High-level, easy to use	Full control, self-hosted
Pricing model	Browser seconds (time-based)	Credit-based (per page)	Free (infrastructure cost only)
Self-host	No	Yes (open source)	Yes (Python/Docker)
SDK	REST API only	Python, Node.js, Go, Rust	Python only
Cloudflare bypass	No	No	Yes (built-in)
License	Commercial	Open source + SaaS	Open source
GitHub	-	github.com/mendableai/firecrawl	github.com/D4Vinci/Scrapling

2. Cloudflare Browser Rendering - /crawl Endpoint

API Details

Endpoint: https://api.cloudflare.com/client/v4/accounts/<account_id>/browser-rendering/crawl
Methods: POST (initiate), GET (retrieve results), DELETE (cancel)
Auth: Authorization: Bearer <apiToken> - requires Browser Rendering - Edit permission

Workflow

1. POST /crawl        --> Get job ID (async)
2. GET /crawl/{id}    --> Poll for results (status: running -> completed)
3. DELETE /crawl/{id} --> (Optional) Cancel the job

Parameters

Parameter	Default	Description
`url`	(required)	Starting URL
`limit`	10	Maximum pages to crawl (max: 100,000)
`depth`	100,000	Link follow depth
`formats`	`["html"]`	`html`, `markdown`, `json`
`render`	`true`	Execute JS. `false` = HTML-only fetch (fast + cheap)
`source`	`all`	URL discovery source: `all`, `sitemaps`, `links`
`maxAge`	86,400 (1 day)	Cache freshness in seconds (max: 604,800)
`modifiedSince`	-	Unix timestamp - only crawl pages modified after this time
`options.includeExternalLinks`	false	Follow external domain links
`options.includeSubdomains`	false	Follow subdomain links
`options.includePatterns`	-	Wildcard patterns (``, `*`) to include
`options.excludePatterns`	-	Wildcard patterns to exclude

Advanced parameters: authenticate, cookies, gotoOptions, setExtraHTTPHeaders, waitForSelector, rejectResourceTypes, userAgent

Job Status Values

Status	Meaning
`running`	Crawl in progress
`completed`	Successfully finished
`cancelled_due_to_timeout`	Exceeded 7-day max runtime
`cancelled_due_to_limits`	Hit account limits
`cancelled_by_user`	Manually cancelled
`errored`	Encountered an error

Limits

Max runtime: 7 days
Result retention: 14 days
Max pages/job: 100,000
Respects robots.txt (mandatory)
CAPTCHA/Turnstile bypass: None
Default User-Agent: CloudflareBrowserRenderingCrawler/1.0

Pros

Native integration with Workers, R2, KV
render: false for very cheap static HTML fetch
14-day result retention (longest)
100K pages/job limit (highest)
Browser seconds pricing - cheap for simple pages

Cons

Workers Paid Plan required ($5/mo minimum)
No CAPTCHA/Cloudflare bypass
No Webhook/WebSocket (polling required)
REST API only (no SDK)
Significant limitations on Free plan

3. Firecrawl

API Details

Endpoint: POST /v2/crawl
Auth: Authorization: Bearer fc-YOUR-API-KEY
SDKs: Python, Node.js, Go, Rust

Workflow

# Blocking (automatic polling + pagination)
result = firecrawl.crawl(url, limit=50)

# Non-blocking
job = firecrawl.start_crawl(url, limit=50)
status = firecrawl.get_crawl_status(job['id'])

Parameters

Parameter	Default	Description
`url`	(required)	Starting URL
`limit`	10,000	Maximum pages to crawl
`maxDiscoveryDepth`	-	Link follow depth
`includePaths`	-	Regex patterns (include)
`excludePaths`	-	Regex patterns (exclude)
`crawlEntireDomain`	false	Include sibling/parent URLs
`allowSubdomains`	false	Follow subdomain links
`allowExternalLinks`	false	Follow external links
`sitemap`	`"include"`	`"include"`, `"skip"`, `"only"`
`ignoreQueryParameters`	false	Avoid re-scraping same path with different query params
`delay`	-	Delay between requests (seconds)
`maxConcurrency`	-	Concurrent request limit

Pricing

Plan	Price/mo	Credits	Concurrent	Extra Credits
Free	$0	500 (one-time)	2	-
Hobby	$16	3,000	5	$9/1K
Standard	$83	100,000	50	$47/35K
Growth	$333	500,000	100	$177/175K
Scale	$599	1,000,000	150	-
Enterprise	Custom	Custom	Custom	-

Credit usage:

Feature	Cost
Scrape/Crawl	1 credit/page
JSON mode (AI)	+4 credits/page
Enhanced proxy	+4 credits/page
PDF parsing	1 credit/PDF page
Search	2 credits/10 results
Browser	2 credits/minute

Pros

Rich SDK support (Python, Node, Go, Rust)
Webhook (signed) + WebSocket (real-time) support
Self-host option (open source)
Regex-based URL filtering
Blocking SDK method (automatic polling)
Geo-location targeting
Error tracking endpoint (/crawl/{id}/errors)

Cons

Credit-based - expensive at high volume
JSON/AI extraction costs +4 credits (5x cost)
Result retention only 24 hours (shortest)
No CAPTCHA/Cloudflare bypass
Credits reset monthly (no rollover)

4. Scrapling

Installation

pip install scrapling
scrapling install  # install browsers

# or Docker
docker pull d4vinci/scrapling

3 Fetcher Types

Fetcher	Use Case	CF Bypass	Speed	Resource Usage
Fetcher	Normal sites, APIs	No	Fastest	Low
StealthyFetcher	Cloudflare-protected sites	Yes (Turnstile + Interstitial)	Medium	Medium
DynamicFetcher	JS-heavy SPAs	Via Playwright	Slowest	High

Anti-Bot Bypass Capabilities

Protection	Support
Cloudflare Turnstile	Built-in
Cloudflare Interstitial	Built-in
TLS Fingerprint	Chrome, Firefox, Safari, Edge impersonation
Akamai, DataDome, Kasada	Via Hyper Solutions integration (external service)

Key Features

Adaptive Parsing: Automatically relocates elements when page structure changes
Spider Framework: Scrapy-like API with concurrent crawling
Pause/Resume: Ctrl+C to pause, restart to continue
Multi-session: Use different fetchers within the same spider
MCP Server: AI-assisted scraping with Claude/Cursor
CLI: Scrape from terminal without writing code
Async/await support
Proxy rotation

Pros

Free and open source
Cloudflare Turnstile/Interstitial bypass (biggest differentiator)
Adaptive parsing (works even when site changes)
Full spider framework (Scrapy-like)
TLS fingerprint impersonation
Zero dependency on managed services

Cons

Python only
Self-hosting required (infrastructure cost)
Server maintenance, browser updates are your responsibility
No Webhook/WebSocket
No managed monitoring
IP ban risk (may need proxies)
Scaling is your responsibility

5. Feature Comparison

Feature	CF Browser Rendering	Firecrawl	Scrapling
JS rendering	Yes (`render: true`)	Yes (default)	Yes (Playwright)
Output formats	HTML, Markdown, JSON	HTML, Markdown, JSON	HTML (with parsing)
AI extraction	Workers AI	Built-in LLM	MCP server
Sitemap support	Yes	Yes	Via spider
URL filtering	Wildcard (``, `*`)	Regex patterns	Python code
Subdomain following	Yes	Yes	Yes
External link following	Yes	Yes	Yes
Auth support	Basic auth, cookies, headers	Basic auth, cookies, headers	Full control
Proxy	Cloudflare network (built-in)	Yes (extra credits)	Self-configured
Geo-location	No	Yes	Possible via proxy
Webhook	No	Yes (signed)	No
WebSocket (real-time)	No	Yes	No
Rate limit control	Automatic	`delay` + `maxConcurrency`	Full control
Result retention	14 days	24 hours	Unlimited (your disk)
Max job runtime	7 days	Not specified	Unlimited
Max pages/job	100,000	10,000 (default)	Unlimited
robots.txt compliance	Mandatory	Optional	Optional
CAPTCHA bypass	No	No	Yes (CF Turnstile)
Error tracking	No	Yes	Via logging
Adaptive parsing	No	No	Yes
Spider framework	No	No	Yes
Pause/Resume	No	No	Yes

6. Pricing Comparison

Cloudflare Browser Rendering

Prerequisite: Workers Paid Plan ($5/mo)
Pricing: Browser seconds based
render: false for HTML-only fetch - very cheap
render: true for JS rendering - cost varies by site complexity

Firecrawl

Plan	Price/mo	Pages/mo	Cost per Page
Free	$0	500 (one-time)	$0
Hobby	$16	3,000	$0.0053
Standard	$83	100,000	$0.00083
Growth	$333	500,000	$0.00067
Scale	$599	1,000,000	$0.00060

Scrapling

Software: $0 (open source)
Infrastructure: Google Cloud, AWS, or your own server
See next section for details

7. Infrastructure Cost (Scrapling Self-Host)

Why Infrastructure Matters

Scrapling runs headless Chromium. Each browser instance uses ~300-500MB RAM. With concurrent scraping, this multiplies. Memory is the critical resource.

Google Cloud Compute Engine

VM Type	vCPU	RAM	Monthly Price	Concurrent Scrapes
e2-small	0.5-2	2 GB	~$13/mo	2-3 pages
e2-medium	1-2	4 GB	~$27/mo	5-8 pages
e2-standard-2	2	8 GB	~$49/mo	10-15 pages
e2-standard-4	4	16 GB	~$97/mo	25-30 pages

Additional costs:

Disk (30GB SSD): ~$5/mo
Egress: first 1GB free, then ~$0.12/GB
Static IP (optional): ~$3/mo

Google Cloud Run

Pricing:

vCPU: $0.00002400/vCPU-second
Memory: $0.00000250/GiB-second
Requests: $0.40/million
Free tier: 2M requests, 360K vCPU-sec, 180K GiB-sec/mo

Per scrape request (CF bypass ~10 sec, 2 vCPU, 4GB RAM):

CPU: 10s x 2vCPU x $0.000024 = $0.00048
RAM: 10s x 4GiB x $0.0000025 = $0.0001
Cost per page: ~$0.00058

Cloud Run caveats:

Cold start adds 5-10s for browser launch
Solved with min instances (but adds cost)
Max timeout 60min - long crawl jobs need to be split

8. Total Cost of Ownership (TCO)

Low Volume: 3,000 pages/month

Service	Monthly Cost	Notes
Scrapling + Cloud Run	~$2	Benefits from free tier
CF Browser Rendering	~$5-7	Workers paid plan required
Firecrawl Hobby	$16	3K credits included
Scrapling + Compute Engine	~$19	24/7 VM (overkill)

Winner: Scrapling + Cloud Run (~~$2) If CF bypass needed: Scrapling + Cloud Run (~~$2) - only real option anyway

Medium Volume: 50,000 pages/month

Service	Monthly Cost	Notes
CF Browser Rendering	~$20-40	Browser seconds based
Scrapling + Cloud Run	~$44	Min instance required
Scrapling + Compute Engine	~$59	e2-standard-2, stable
Firecrawl Standard	$83	100K credits, easy to use

Winner: CF Browser Rendering (~~$20-40) If CF bypass needed: Scrapling + Cloud Run (~~$44)

High Volume: 500,000 pages/month

Service	Monthly Cost	Notes
CF Browser Rendering	~$100-200	Estimated
Scrapling + Compute Engine	~$234	2x e2-standard-4
Firecrawl Growth	$333	500K credits
Scrapling + Cloud Run	~$350	Gets expensive at this scale
Firecrawl Scale	$599	1M credits

Winner: CF Browser Rendering (~~$100-200) If CF bypass needed: Scrapling + Compute Engine (~~$234)

Hidden Costs

Cost Item	Scrapling (self-host)	Managed Services
Server maintenance	You manage	None
Browser updates	You update	Automatic
Proxy (if IP banned)	$20-100+/mo	Built-in/extra fee
Monitoring/logging	$5-20/mo (Cloud Monitoring)	Built-in
Developer time	High (debug, maintain)	Low
Downtime risk	Present	Low

9. Decision Matrix

Cloudflare Bypass Required

Volume	Recommendation	Cost
Low (3K)	Scrapling + Cloud Run	~$2/mo
Medium (50K)	Scrapling + Cloud Run	~$44/mo
High (500K)	Scrapling + Compute Engine	~$234/mo

Cloudflare Bypass Not Required

Volume	Recommendation	Cost
Low (3K)	CF Browser Rendering	~$5-7/mo
Medium (50K)	CF Browser Rendering	~$20-40/mo
High (500K)	CF Browser Rendering	~$100-200/mo

Other Scenarios

Scenario	Recommendation	Reason
Quick prototype	Firecrawl Free/Hobby	Working in 5 minutes with SDK
Zero-cost trial	Scrapling + local machine	Run on your own computer
Minimum management	Firecrawl	SDK + webhook + zero ops
Using Workers ecosystem	CF Browser Rendering	Native integration
Site structure changes often	Scrapling	Adaptive parsing
Real-time result streaming	Firecrawl	Webhook + WebSocket
Data control (compliance)	Scrapling self-host	Data never leaves your infra
Crawl from different countries	Firecrawl	Geo-location targeting

10. Code Examples

Cloudflare Browser Rendering

# 1. Start crawl
curl -X POST \
  'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
  -H 'Authorization: Bearer <API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "limit": 50,
    "depth": 3,
    "formats": ["markdown"],
    "options": {
      "includePatterns": ["**/blog/**"],
      "excludePatterns": ["**/admin/**"]
    }
  }'

# 2. Retrieve results
curl -X GET \
  'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}?status=completed' \
  -H 'Authorization: Bearer <API_TOKEN>'

# 3. Cancel
curl -X DELETE \
  'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}' \
  -H 'Authorization: Bearer <API_TOKEN>'

// JavaScript polling example
async function crawlAndWait(accountId, apiToken, targetUrl) {
  const startRes = await fetch(
    `https://api.cloudflare.com/client/v4/accounts/${accountId}/browser-rendering/crawl`,
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiToken}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({ url: targetUrl, limit: 50, formats: ['markdown'] })
    }
  );
  const { result: jobId } = await startRes.json();

  for (let i = 0; i < 60; i++) {
    const res = await fetch(
      `https://api.cloudflare.com/client/v4/accounts/${accountId}/browser-rendering/crawl/${jobId}`,
      { headers: { Authorization: `Bearer ${apiToken}` } }
    );
    const data = await res.json();
    if (data.result.status !== 'running') return data.result;
    await new Promise(r => setTimeout(r, 5000));
  }
}

Firecrawl

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR-API-KEY")

# Blocking (automatic polling)
result = app.crawl(
    url="https://example.com",
    limit=50,
    scrape_options={
        "formats": ["markdown"],
        "includePaths": ["/blog/*"],
        "excludePaths": ["/admin/*"]
    }
)

for page in result['data']:
    print(page['metadata']['title'])
    print(page['markdown'][:200])

import FirecrawlApp from '@mendable/firecrawl-js';

const app = new FirecrawlApp({ apiKey: "fc-YOUR-API-KEY" });

const result = await app.crawlUrl("https://example.com", {
  limit: 50,
  scrapeOptions: { formats: ["markdown"] }
});

result.data.forEach(page => {
  console.log(page.metadata.title);
  console.log(page.markdown.substring(0, 200));
});

Scrapling

from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher

# --- Simple HTTP request ---
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()

# --- With Cloudflare bypass ---
with StealthyFetcher.create_session(headless=True, solve_cloudflare=True) as session:
    page = session.fetch('https://cloudflare-protected-site.com')
    title = page.css('h1::text').get()
    links = page.css('a::attr(href)').getall()

# --- For JS-heavy SPAs ---
with DynamicFetcher.create_session(headless=True, network_idle=True) as session:
    page = session.fetch('https://react-spa-site.com')
    data = page.css('.product-card')

# --- Full Spider ---
from scrapling import Spider, Response, Request

class MyScraper(Spider):
    name = "my_scraper"
    start_urls = ["https://example.com"]
    concurrent_requests = 10

    async def parse(self, response: Response):
        for item in response.css('.item'):
            yield {
                "title": item.css('.title::text').get(),
                "url": item.css('a::attr(href)').get(),
            }

        # Follow pagination
        next_page = response.css('.next a::attr(href)').get()
        if next_page:
            yield Request(next_page, callback=self.parse)

# Run (Ctrl+C to pause, restart to resume)
MyScraper(crawldir="./crawl_data").start()

References

Raw

web-scraping-comparison.tr.md

Web Scraping Servisleri Karsilastirmasi: Cloudflare Browser Rendering vs Firecrawl vs Scrapling

Tarih: 2026-03-12 | Yazar: @senrecep

Icindekiler

1. Genel Bakis
2. Cloudflare Browser Rendering - /crawl Endpoint
3. Firecrawl
4. Scrapling
5. Ozellik Karsilastirmasi
6. Fiyatlandirma Karsilastirmasi
7. Altyapi Maliyeti (Scrapling Self-Host)
8. Toplam Maliyet Karsilastirmasi (TCO)
9. Karar Matrisi
10. Kod Ornekleri

1. Genel Bakis

	Cloudflare Browser Rendering	Firecrawl	Scrapling
Tip	Altyapi servisi (Workers ekosistemi)	Managed scraping SaaS	Acik kaynak Python framework
Yaklasim	Dusuk seviye, esnek	Yuksek seviye, kolay kullanim	Tam kontrol, self-host
Fiyat modeli	Browser seconds (sure bazli)	Credit bazli (sayfa basi)	Ucretsiz (altyapi maliyeti var)
Self-host	Yok	Var (acik kaynak)	Var (Python/Docker)
SDK	Sadece REST API	Python, Node.js, Go, Rust	Sadece Python
Cloudflare bypass	Yok	Yok	Var (dahili)
Lisans	Ticari	Acik kaynak + SaaS	Acik kaynak
GitHub	-	github.com/mendableai/firecrawl	github.com/D4Vinci/Scrapling

2. Cloudflare Browser Rendering - /crawl Endpoint

API Bilgileri

Endpoint: https://api.cloudflare.com/client/v4/accounts/<account_id>/browser-rendering/crawl
Metod: POST (baslat), GET (sonuc al), DELETE (iptal)
Auth: Authorization: Bearer <apiToken> - Browser Rendering - Edit permission gerekli

Akis

1. POST /crawl        --> Job ID al (asenkron)
2. GET /crawl/{id}    --> Polling ile sonuc al (status: running -> completed)
3. DELETE /crawl/{id} --> (Opsiyonel) Isi iptal et

Parametreler

Parametre	Varsayilan	Aciklama
`url`	(zorunlu)	Baslangic URL'i
`limit`	10	Maksimum sayfa sayisi (max: 100,000)
`depth`	100,000	Link takip derinligi
`formats`	`["html"]`	`html`, `markdown`, `json`
`render`	`true`	JS calistir. `false` = sadece HTML fetch (hizli + ucuz)
`source`	`all`	URL kesif kaynagi: `all`, `sitemaps`, `links`
`maxAge`	86,400 (1 gun)	Cache suresi (saniye, max: 604,800)
`modifiedSince`	-	Unix timestamp - sadece bu tarihten sonra degisen sayfalari tara
`options.includeExternalLinks`	false	Harici domain link'lerini takip et
`options.includeSubdomains`	false	Subdomain link'lerini takip et
`options.includePatterns`	-	Wildcard pattern'ler (``, `*`)
`options.excludePatterns`	-	Haric tutulacak wildcard pattern'ler

Gelismis parametreler: authenticate, cookies, gotoOptions, setExtraHTTPHeaders, waitForSelector, rejectResourceTypes, userAgent

Job Status Degerleri

Status	Anlam
`running`	Crawl devam ediyor
`completed`	Basariyla tamamlandi
`cancelled_due_to_timeout`	7 gun max sureyi asti
`cancelled_due_to_limits`	Hesap limitine ulasti
`cancelled_by_user`	Manuel iptal
`errored`	Hata olustu

Limitler

Max calisma suresi: 7 gun
Sonuc saklama: 14 gun
Max sayfa/job: 100,000
robots.txt'e uyar (zorunlu)
CAPTCHA/Turnstile bypass yok
Default User-Agent: CloudflareBrowserRenderingCrawler/1.0

Avantajlar

Workers, R2, KV ile native entegrasyon
render: false ile cok ucuz statik HTML fetch
14 gun sonuc saklama (en uzun)
100K sayfa/job limiti (en yuksek)
Browser seconds bazli - basit sayfalar icin ucuz

Dezavantajlar

Workers Paid Plan zorunlu ($5/ay minimum)
CAPTCHA/Cloudflare bypass yok
Webhook/WebSocket yok (polling zorunlu)
Sadece REST API (SDK yok)
Free plan'da ciddi kisitlamalar

3. Firecrawl

API Bilgileri

Endpoint: POST /v2/crawl
Auth: Authorization: Bearer fc-YOUR-API-KEY
SDK'lar: Python, Node.js, Go, Rust

Akis

# Blocking (otomatik polling + pagination)
result = firecrawl.crawl(url, limit=50)

# Non-blocking
job = firecrawl.start_crawl(url, limit=50)
status = firecrawl.get_crawl_status(job['id'])

Parametreler

Parametre	Varsayilan	Aciklama
`url`	(zorunlu)	Baslangic URL'i
`limit`	10,000	Maksimum sayfa sayisi
`maxDiscoveryDepth`	-	Link takip derinligi
`includePaths`	-	Regex pattern'ler (dahil)
`excludePaths`	-	Regex pattern'ler (haric)
`crawlEntireDomain`	false	Kardes/ust URL'leri dahil et
`allowSubdomains`	false	Subdomain link'lerini takip et
`allowExternalLinks`	false	Harici link'leri takip et
`sitemap`	`"include"`	`"include"`, `"skip"`, `"only"`
`ignoreQueryParameters`	false	Ayni path'i farkli query ile tekrar tarama
`delay`	-	Istekler arasi bekleme suresi (sn)
`maxConcurrency`	-	Eslenik istek limiti

Fiyatlandirma

Plan	Fiyat/ay	Credit	Concurrent	Ek Credit
Free	$0	500 (tek seferlik)	2	-
Hobby	$16	3,000	5	$9/1K
Standard	$83	100,000	50	$47/35K
Growth	$333	500,000	100	$177/175K
Scale	$599	1,000,000	150	-
Enterprise	Ozel	Ozel	Ozel	-

Credit kullanimi:

Ozellik	Maliyet
Scrape/Crawl	1 credit/sayfa
JSON mode (AI)	+4 credit/sayfa
Enhanced proxy	+4 credit/sayfa
PDF parsing	1 credit/PDF sayfa
Search	2 credit/10 sonuc
Browser	2 credit/dakika

Avantajlar

Zengin SDK destegi (Python, Node, Go, Rust)
Webhook (imzali) + WebSocket (real-time) destegi
Self-host secenegi (acik kaynak)
Regex bazli URL filtreleme
Blocking SDK metodu (polling otomatik)
Geo-location targeting
Hata tracking endpoint (/crawl/{id}/errors)

Dezavantajlar

Credit bazli - yuksek hacimde pahali
JSON/AI extraction +4 credit (5x maliyet)
Sonuc saklama sadece 24 saat (en kisa)
CAPTCHA/Cloudflare bypass yok
Credit'ler aylik sifirlanir (tasimaz)

4. Scrapling

Kurulum

pip install scrapling
scrapling install  # browser yukle

# veya Docker
docker pull d4vinci/scrapling

3 Fetcher Tipi

Fetcher	Kullanim	CF Bypass	Hiz	Kaynak Tuketimi
Fetcher	Normal siteler, API'ler	Yok	En hizli	Dusuk
StealthyFetcher	Cloudflare korumali siteler	Var (Turnstile + Interstitial)	Orta	Orta
DynamicFetcher	JS-heavy SPA'lar	Playwright ile	En yavas	Yuksek

Anti-Bot Bypass Yetenekleri

Koruma	Destek
Cloudflare Turnstile	Dahili
Cloudflare Interstitial	Dahili
TLS Fingerprint	Chrome, Firefox, Safari, Edge taklidi
Akamai, DataDome, Kasada	Hyper Solutions entegrasyonu (ek servis)

Ozellikler

Adaptive Parsing: Site yapisi degisse bile element'leri otomatik bulur
Spider Framework: Scrapy benzeri API, concurrent crawling
Pause/Resume: Ctrl+C ile durdur, tekrar calistir devam etsin
Multi-session: Ayni spider'da farkli fetcher'lar
MCP Server: Claude/Cursor ile AI-assisted scraping
CLI: Kod yazmadan terminal'den scrape
Async/await destegi
Proxy rotasyonu

Avantajlar

Ucretsiz ve acik kaynak
Cloudflare Turnstile/Interstitial bypass (en buyuk fark)
Adaptive parsing (site degisse bile calisir)
Tam spider framework (Scrapy benzeri)
TLS fingerprint taklidi
Sifir bagimlilik managed servise

Dezavantajlar

Sadece Python
Self-host zorunlu (altyapi maliyeti)
Sunucu bakimi, browser guncellemeleri senin sorunun
Webhook/WebSocket yok
Managed monitoring yok
IP ban riski (proxy gerekebilir)
Olceklendirme senin sorumlulugunda

5. Ozellik Karsilastirmasi

Ozellik	CF Browser Rendering	Firecrawl	Scrapling
JS rendering	Var (`render: true`)	Var (varsayilan)	Var (Playwright)
Cikti formatlari	HTML, Markdown, JSON	HTML, Markdown, JSON	HTML (parse ile)
AI veri cikarma	Workers AI	Dahili LLM	MCP server
Sitemap destegi	Var	Var	Spider ile
URL filtreleme	Wildcard (``, `*`)	Regex pattern	Python kodu ile
Subdomain takibi	Var	Var	Var
External link takibi	Var	Var	Var
Auth destegi	Basic auth, cookies, headers	Basic auth, cookies, headers	Tam kontrol
Proxy	Cloudflare agi (dahili)	Var (ek credit)	Kendin ayarla
Geo-location	Yok	Var	Proxy ile mumkun
Webhook	Yok	Var (imzali)	Yok
WebSocket (real-time)	Yok	Var	Yok
Rate limit kontrolu	Otomatik	`delay` + `maxConcurrency`	Tam kontrol
Sonuc saklama	14 gun	24 saat	Sinirsiz (kendi disk)
Max job suresi	7 gun	Belirtilmemis	Sinirsiz
Max sayfa/job	100,000	10,000 (varsayilan)	Sinirsiz
robots.txt uyumu	Zorunlu	Opsiyonel	Opsiyonel
CAPTCHA bypass	Yok	Yok	Var (CF Turnstile)
Error tracking	Yok	Var	Loglama ile
Adaptive parsing	Yok	Yok	Var
Spider framework	Yok	Yok	Var
Pause/Resume	Yok	Yok	Var

6. Fiyatlandirma Karsilastirmasi

Cloudflare Browser Rendering

On kosul: Workers Paid Plan ($5/ay)
Ucretlendirme: Browser seconds bazli
render: false ile sadece HTML fetch - cok ucuz
render: true ile JS rendering - site karmasikligina gore degisir

Firecrawl

Plan	Fiyat/ay	Sayfa/ay	Sayfa Basi Maliyet
Free	$0	500 (tek seferlik)	$0
Hobby	$16	3,000	$0.0053
Standard	$83	100,000	$0.00083
Growth	$333	500,000	$0.00067
Scale	$599	1,000,000	$0.00060

Scrapling

Yazilim: $0 (acik kaynak)
Altyapi: Google Cloud, AWS veya kendi sunucun
Detay icin sonraki bolume bak

7. Altyapi Maliyeti (Scrapling Self-Host)

Neden Altyapi Onemli?

Scrapling headless Chromium calistirir. Her browser instance ~300-500MB RAM kullanir. Concurrent scraping'de bu katlanir. Bu nedenle bellek kritik.

Google Cloud Compute Engine

VM Tipi	vCPU	RAM	Aylik Fiyat	Concurrent Scrape
e2-small	0.5-2	2 GB	~$13/ay	2-3 sayfa
e2-medium	1-2	4 GB	~$27/ay	5-8 sayfa
e2-standard-2	2	8 GB	~$49/ay	10-15 sayfa
e2-standard-4	4	16 GB	~$97/ay	25-30 sayfa

Ek maliyetler:

Disk (30GB SSD): ~$5/ay
Egress (cikis trafigi): ilk 1GB ucretsiz, sonra ~$0.12/GB
Statik IP (opsiyonel): ~$3/ay

Google Cloud Run

Fiyatlandirma:

vCPU: $0.00002400/vCPU-saniye
Bellek: $0.00000250/GiB-saniye
Request: $0.40/milyon
Free tier: 2M request, 360K vCPU-sn, 180K GiB-sn/ay

Her scrape istegi icin (CF bypass ile ~10 sn, 2 vCPU, 4GB RAM):

CPU: 10sn x 2vCPU x $0.000024 = $0.00048
RAM: 10sn x 4GiB x $0.0000025 = $0.0001
Sayfa basi maliyet: ~$0.00058

Cloud Run sorunlari:

Cold start'ta browser launch 5-10sn ekler
Min instance ile cozulur ama ek maliyet
Max timeout 60dk - uzun crawl job'lari bolmen gerekir

8. Toplam Maliyet Karsilastirmasi (TCO)

Dusuk Hacim: 3,000 sayfa/ay

Servis	Aylik Maliyet	Notlar
Scrapling + Cloud Run	~$2	Free tier'den faydalanir
CF Browser Rendering	~$5-7	Workers paid plan zorunlu
Firecrawl Hobby	$16	3K credit dahil
Scrapling + Compute Engine	~$19	7/24 calisan VM (overkill)

Kazanan: Scrapling + Cloud Run (~~$2) CF bypass gerekiyorsa: Scrapling + Cloud Run (~~$2) - tek secenek zaten

Orta Hacim: 50,000 sayfa/ay

Servis	Aylik Maliyet	Notlar
CF Browser Rendering	~$20-40	Browser seconds bazli
Scrapling + Cloud Run	~$44	Min instance gerekli
Scrapling + Compute Engine	~$59	e2-standard-2, stabil
Firecrawl Standard	$83	100K credit, kolay kullanim

Kazanan: CF Browser Rendering (~~$20-40) CF bypass gerekiyorsa: Scrapling + Cloud Run (~~$44)

Yuksek Hacim: 500,000 sayfa/ay

Servis	Aylik Maliyet	Notlar
CF Browser Rendering	~$100-200	Tahmini
Scrapling + Compute Engine	~$234	2x e2-standard-4
Firecrawl Growth	$333	500K credit
Scrapling + Cloud Run	~$350	Bu hacimde pahalilasir
Firecrawl Scale	$599	1M credit

Kazanan: CF Browser Rendering (~~$100-200) CF bypass gerekiyorsa: Scrapling + Compute Engine (~~$234)

Gizli Maliyetler

Maliyet Kalemi	Scrapling (self-host)	Managed Servisler
Sunucu bakimi	Sen yonetirsin	Yok
Browser guncellemeleri	Sen guncellersin	Otomatik
Proxy (IP ban durumunda)	$20-100+/ay	Dahili/ek ucret
Monitoring/logging	$5-20/ay (Cloud Monitoring)	Dahili
Gelistirici zamani	Yuksek (debug, maintain)	Dusuk
Downtime riski	Var	Dusuk

9. Karar Matrisi

Cloudflare Bypass LAZIM

Hacim	Oneri	Maliyet
Dusuk (3K)	Scrapling + Cloud Run	~$2/ay
Orta (50K)	Scrapling + Cloud Run	~$44/ay
Yuksek (500K)	Scrapling + Compute Engine	~$234/ay

Cloudflare Bypass LAZIM DEGIL

Hacim	Oneri	Maliyet
Dusuk (3K)	CF Browser Rendering	~$5-7/ay
Orta (50K)	CF Browser Rendering	~$20-40/ay
Yuksek (500K)	CF Browser Rendering	~$100-200/ay

Diger Senaryolar

Senaryo	Oneri	Neden
Hizli prototip	Firecrawl Free/Hobby	SDK ile 5 dakikada calisir
Sifir maliyet denemek	Scrapling + lokal makine	Kendi bilgisayarinda
Minimum yonetim	Firecrawl	SDK + webhook + sifir ops
Workers ekosistemi kullaniliyor	CF Browser Rendering	Native entegrasyon
Site yapisi sik degisiyor	Scrapling	Adaptive parsing
Real-time sonuc akisi	Firecrawl	Webhook + WebSocket
Veri kontrolu (compliance)	Scrapling self-host	Veri hicbir yere gitmez
Farkli ulkelerden crawl	Firecrawl	Geo-location targeting

10. Kod Ornekleri

Cloudflare Browser Rendering

# 1. Crawl baslat
curl -X POST \
  'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
  -H 'Authorization: Bearer <API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "limit": 50,
    "depth": 3,
    "formats": ["markdown"],
    "options": {
      "includePatterns": ["**/blog/**"],
      "excludePatterns": ["**/admin/**"]
    }
  }'

# 2. Sonuclari al
curl -X GET \
  'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}?status=completed' \
  -H 'Authorization: Bearer <API_TOKEN>'

# 3. Iptal et
curl -X DELETE \
  'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}' \
  -H 'Authorization: Bearer <API_TOKEN>'

// JavaScript polling ornegi
async function crawlAndWait(accountId, apiToken, targetUrl) {
  const startRes = await fetch(
    `https://api.cloudflare.com/client/v4/accounts/${accountId}/browser-rendering/crawl`,
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiToken}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({ url: targetUrl, limit: 50, formats: ['markdown'] })
    }
  );
  const { result: jobId } = await startRes.json();

  for (let i = 0; i < 60; i++) {
    const res = await fetch(
      `https://api.cloudflare.com/client/v4/accounts/${accountId}/browser-rendering/crawl/${jobId}`,
      { headers: { Authorization: `Bearer ${apiToken}` } }
    );
    const data = await res.json();
    if (data.result.status !== 'running') return data.result;
    await new Promise(r => setTimeout(r, 5000));
  }
}

Firecrawl

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR-API-KEY")

# Blocking (otomatik polling)
result = app.crawl(
    url="https://example.com",
    limit=50,
    scrape_options={
        "formats": ["markdown"],
        "includePaths": ["/blog/*"],
        "excludePaths": ["/admin/*"]
    }
)

for page in result['data']:
    print(page['metadata']['title'])
    print(page['markdown'][:200])

import FirecrawlApp from '@mendable/firecrawl-js';

const app = new FirecrawlApp({ apiKey: "fc-YOUR-API-KEY" });

const result = await app.crawlUrl("https://example.com", {
  limit: 50,
  scrapeOptions: { formats: ["markdown"] }
});

result.data.forEach(page => {
  console.log(page.metadata.title);
  console.log(page.markdown.substring(0, 200));
});

Scrapling

from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher

# --- Basit HTTP istegi ---
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()

# --- Cloudflare bypass ile ---
with StealthyFetcher.create_session(headless=True, solve_cloudflare=True) as session:
    page = session.fetch('https://cloudflare-korumali-site.com')
    title = page.css('h1::text').get()
    links = page.css('a::attr(href)').getall()

# --- JS-heavy SPA icin ---
with DynamicFetcher.create_session(headless=True, network_idle=True) as session:
    page = session.fetch('https://react-spa-site.com')
    data = page.css('.product-card')

# --- Tam Spider ---
from scrapling import Spider, Response, Request

class MyScraper(Spider):
    name = "my_scraper"
    start_urls = ["https://example.com"]
    concurrent_requests = 10

    async def parse(self, response: Response):
        for item in response.css('.item'):
            yield {
                "title": item.css('.title::text').get(),
                "url": item.css('a::attr(href)').get(),
            }

        # Sonraki sayfaya git
        next_page = response.css('.next a::attr(href)').get()
        if next_page:
            yield Request(next_page, callback=self.parse)

# Calistir (Ctrl+C ile durdur, tekrar calistir devam etsin)
MyScraper(crawldir="./crawl_data").start()

senrecep/web-scraping-comparison.md

Web Scraping Services Comparison: Cloudflare Browser Rendering vs Firecrawl vs Scrapling

Table of Contents

1. Overview

2. Cloudflare Browser Rendering - /crawl Endpoint

API Details

Workflow

Parameters

Job Status Values

Limits

Pros

Cons

3. Firecrawl

API Details

Workflow

Parameters

Pricing

Pros

Cons

4. Scrapling

Installation

3 Fetcher Types

Anti-Bot Bypass Capabilities

Key Features

Pros

Cons

5. Feature Comparison

6. Pricing Comparison

Cloudflare Browser Rendering

Firecrawl

Scrapling

7. Infrastructure Cost (Scrapling Self-Host)

Why Infrastructure Matters

Google Cloud Compute Engine

Google Cloud Run

8. Total Cost of Ownership (TCO)

Low Volume: 3,000 pages/month

Medium Volume: 50,000 pages/month

High Volume: 500,000 pages/month

Hidden Costs

9. Decision Matrix

Cloudflare Bypass Required

Cloudflare Bypass Not Required

Other Scenarios

10. Code Examples

Cloudflare Browser Rendering

Firecrawl

Scrapling

References

Web Scraping Servisleri Karsilastirmasi: Cloudflare Browser Rendering vs Firecrawl vs Scrapling

Icindekiler

1. Genel Bakis

2. Cloudflare Browser Rendering - /crawl Endpoint

API Bilgileri

Akis

Parametreler

Job Status Degerleri

Limitler

Avantajlar

Dezavantajlar

3. Firecrawl

API Bilgileri

Akis

Parametreler

Fiyatlandirma

Avantajlar

Dezavantajlar

4. Scrapling

Kurulum

3 Fetcher Tipi

Anti-Bot Bypass Yetenekleri

Ozellikler

Avantajlar

Dezavantajlar

5. Ozellik Karsilastirmasi

6. Fiyatlandirma Karsilastirmasi

Cloudflare Browser Rendering

Firecrawl

Scrapling

7. Altyapi Maliyeti (Scrapling Self-Host)