A practical field guide from Automation Ace.
Short answer: to automate web scraping with Firecrawl, pick the endpoint that matches the job (scrape for one page, batch scrape for a list, map or crawl for a whole site, search when you don't know the URLs), request clean markdown or structured JSON, and run it on a schedule or trigger from Zapier, Make, n8n, or a cron job. Store results by URL with a timestamp, alert only on changes, and set page and run limits so credits stay predictable.
Endpoint names and options reflect Firecrawl's v2 documentation at the time of writing. Always follow each site's terms and robots rules.
Short answer
- Choose scrape, batch scrape, map, crawl, or search.
- Ask for markdown (for reading and AI) or JSON (for data).
- Schedule it or trigger it from your automation platform.
- Store by URL; compare with the last run; alert on changes.
- Cap pages and runs; log failures.
What Firecrawl does
Firecrawl turns websites into clean, LLM-ready content. It renders JavaScript-heavy pages, strips navigation and boilerplate, and returns markdown, HTML, links, screenshots, or structured JSON, through an API, SDKs, and integrations such as Zapier, Make, n8n, and an MCP server for AI tools. For the concepts, see web parsing vs web scraping vs web crawling.
Firecrawl endpoints at a glance
| Endpoint | What it does | Use it for |
|---|---|---|
| Scrape | Turns one URL into clean markdown, HTML, links, a screenshot, or JSON | A single page: pricing page, article, product |
| Batch scrape | Scrapes a list of URLs as one job | Hundreds of known URLs |
| Map | Returns the URLs it can find on a site, quickly, without scraping content | Discovering pages before you scrape |
| Crawl | Follows links from a start URL and scrapes the pages it finds (asynchronous job) | A whole docs site, blog, or section |
| Search | Runs a web search and can return the content of each result | Finding pages when you don't know the URLs |
| Change tracking / monitoring | Compares a page with its previous version | Alerts when pricing, policies, or listings change |
Step 1: Decide what to scrape and how often
| Goal | Pattern |
|---|---|
| Same pages on a schedule | Scheduled workflow → list of URLs → scrape or batch scrape |
| Only act when something changes | Change tracking or monitoring → alert on diffs |
| Whole site refresh | Map or crawl on a schedule → store pages by URL |
| On demand | A form, Slack command, or webhook triggers a scrape |
| Discovery | Search → scrape the top results |
Write down the URLs or sections, the fields you need, and how fresh the data must be. Daily is plenty for most pricing and content monitoring.
Step 2: Scrape a page
# Scrape one page as markdown (v2 API; check the current reference)
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H "Authorization: Bearer $FIRECRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/pricing",
"formats": ["markdown", "links"],
"onlyMainContent": true
}'
# -> { "success": true, "data": { "markdown": "...", "links": [...], "metadata": {...} } }
Use markdown when people or AI will read the content; use JSON when you need fields (see turning websites into structured data).
Step 3: Crawl or map a whole site
# Crawl a section, then poll for results
curl -X POST https://api.firecrawl.dev/v2/crawl \
-H "Authorization: Bearer $FIRECRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/blog",
"limit": 50,
"includePaths": ["^/blog/.*"],
"scrapeOptions": { "formats": ["markdown"], "onlyMainContent": true }
}'
# -> { "success": true, "id": "crawl_123", "url": ".../v2/crawl/crawl_123" }
curl https://api.firecrawl.dev/v2/crawl/crawl_123 \
-H "Authorization: Bearer $FIRECRAWL_API_KEY"
# -> { "status": "scraping" | "completed", "data": [ ...pages... ], "next": ... }
Run map first to see the URLs, then crawl with include or exclude paths and a page limit. Crawls are asynchronous: start the job, then poll or use a webhook option if offered.
Step 4: Schedule it with Zapier, Make, n8n, or code
Zapier / Make / n8n: weekly competitor pricing snapshot
Trigger Schedule: every Monday 7:00
Step 1 Google Sheets / Airtable: get the list of competitor pricing URLs
Step 2 Loop over rows (Looping by Zapier, Make iterator, n8n Split in Batches)
Step 3 Firecrawl: Scrape URL -> markdown (or JSON with a schema)
Step 4 AI step: "Summarize plan names and prices; flag changes vs last week"
Step 5 Airtable: update the row with today's snapshot and summary
Step 6 Slack: post only rows where something changed
For code, a scheduled function works well; see Cloudflare Cron Triggers. Polling vs on-demand: API pull vs poll.
Step 5: Store, dedupe, and alert
- Store each page by URL with the scrape time in Airtable, a sheet, or a database.
- Compare with the previous version, or use Firecrawl's change tracking, and only alert when something changed.
- Alert to one Slack channel or a daily digest; see daily digests.
- Log failures and retry later; see 429 errors.
Best practices
| Practice | Why |
|---|---|
| Scrape the main content only | Smaller output, fewer tokens if AI reads it |
| Use map before crawl | See what you will pay for, and include or exclude paths |
| Set limits | Cap pages per crawl and runs per day to control credits |
| Store by URL with a timestamp | Easy updates, history, and dedupe |
| Handle failures | Retry with backoff; log URLs that fail; alert on spikes |
| Respect site rules | Follow terms of service and robots rules; avoid personal data you don't need; don't overload sites |
Next steps
- Extract fields with AI: turn any website into structured data using Firecrawl and AI.
- Let an agent decide what to read: build AI agents that research websites using Firecrawl.
- Monitoring ideas: website monitoring, summarization, and competitor research.
Frequently asked questions
How do I automate web scraping with Firecrawl?
Choose the right endpoint (scrape, batch scrape, map, crawl, or search), request markdown or JSON, and run it on a schedule or trigger from Zapier, Make, n8n, or a cron job. Store results by URL, alert only on changes, and cap pages and runs.
What is the difference between Firecrawl scrape, crawl, and map?
Scrape returns the content of one URL. Map quickly returns the URLs found on a site without scraping them. Crawl follows links from a starting URL and scrapes the pages it finds as an asynchronous job.
Does Firecrawl work with Zapier, Make, and n8n?
Yes. Firecrawl has integrations with Zapier, Make, and n8n, so you can scrape pages on a schedule or trigger and pass the results to other apps. Check each platform for the current actions.
Can Firecrawl detect when a website changes?
Yes. Firecrawl offers change tracking and monitoring features that compare a page with its previous version, so you can alert only when something changes. Check current pricing for change tracking modes.
Is automated web scraping legal?
It depends on the site’s terms, the data, and the law where you operate. Follow terms of service and robots rules, avoid unnecessary personal data, and don’t overload sites. This is not legal advice.
Disclaimer: Firecrawl endpoints, formats, credits, and plans change, and you are responsible for following each website's terms, robots rules, and applicable law; details reflect public documentation and coverage at the time of writing, so confirm current details with each provider before building. Automation Ace is not affiliated with Firecrawl beyond the referral links marked on this page. This article may include links to apps, products, or services; some links may be affiliate links, which means Automation Ace may earn a commission at no extra cost to you.