Web Data Automation

How to Automate Web Scraping with Firecrawl

Scrape, map, crawl, or search, then schedule it and act only on what changed.

FirecrawlWeb ScrapingAutomation

By Troy Tessalone · · 9 minutes

Web Data Automation

A practical field guide from Automation Ace.

Short answer: to automate web scraping with Firecrawl, pick the endpoint that matches the job (scrape for one page, batch scrape for a list, map or crawl for a whole site, search when you don't know the URLs), request clean markdown or structured JSON, and run it on a schedule or trigger from Zapier, Make, n8n, or a cron job. Store results by URL with a timestamp, alert only on changes, and set page and run limits so credits stay predictable.

Endpoint names and options reflect Firecrawl's v2 documentation at the time of writing. Always follow each site's terms and robots rules.

Short answer

  1. Choose scrape, batch scrape, map, crawl, or search.
  2. Ask for markdown (for reading and AI) or JSON (for data).
  3. Schedule it or trigger it from your automation platform.
  4. Store by URL; compare with the last run; alert on changes.
  5. Cap pages and runs; log failures.

What Firecrawl does

Firecrawl turns websites into clean, LLM-ready content. It renders JavaScript-heavy pages, strips navigation and boilerplate, and returns markdown, HTML, links, screenshots, or structured JSON, through an API, SDKs, and integrations such as Zapier, Make, n8n, and an MCP server for AI tools. For the concepts, see web parsing vs web scraping vs web crawling.

Firecrawl endpoints at a glance

EndpointWhat it doesUse it for
ScrapeTurns one URL into clean markdown, HTML, links, a screenshot, or JSONA single page: pricing page, article, product
Batch scrapeScrapes a list of URLs as one jobHundreds of known URLs
MapReturns the URLs it can find on a site, quickly, without scraping contentDiscovering pages before you scrape
CrawlFollows links from a start URL and scrapes the pages it finds (asynchronous job)A whole docs site, blog, or section
SearchRuns a web search and can return the content of each resultFinding pages when you don't know the URLs
Change tracking / monitoringCompares a page with its previous versionAlerts when pricing, policies, or listings change

Step 1: Decide what to scrape and how often

GoalPattern
Same pages on a scheduleScheduled workflow → list of URLs → scrape or batch scrape
Only act when something changesChange tracking or monitoring → alert on diffs
Whole site refreshMap or crawl on a schedule → store pages by URL
On demandA form, Slack command, or webhook triggers a scrape
DiscoverySearch → scrape the top results

Write down the URLs or sections, the fields you need, and how fresh the data must be. Daily is plenty for most pricing and content monitoring.

Step 2: Scrape a page

# Scrape one page as markdown (v2 API; check the current reference)
curl -X POST https://api.firecrawl.dev/v2/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "url": "https://example.com/pricing",
        "formats": ["markdown", "links"],
        "onlyMainContent": true
      }'
# -> { "success": true, "data": { "markdown": "...", "links": [...], "metadata": {...} } }

Use markdown when people or AI will read the content; use JSON when you need fields (see turning websites into structured data).

Step 3: Crawl or map a whole site

# Crawl a section, then poll for results
curl -X POST https://api.firecrawl.dev/v2/crawl \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "url": "https://example.com/blog",
        "limit": 50,
        "includePaths": ["^/blog/.*"],
        "scrapeOptions": { "formats": ["markdown"], "onlyMainContent": true }
      }'
# -> { "success": true, "id": "crawl_123", "url": ".../v2/crawl/crawl_123" }

curl https://api.firecrawl.dev/v2/crawl/crawl_123 \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY"
# -> { "status": "scraping" | "completed", "data": [ ...pages... ], "next": ... }

Run map first to see the URLs, then crawl with include or exclude paths and a page limit. Crawls are asynchronous: start the job, then poll or use a webhook option if offered.

Step 4: Schedule it with Zapier, Make, n8n, or code

Zapier / Make / n8n: weekly competitor pricing snapshot

Trigger   Schedule: every Monday 7:00
Step 1    Google Sheets / Airtable: get the list of competitor pricing URLs
Step 2    Loop over rows (Looping by Zapier, Make iterator, n8n Split in Batches)
Step 3    Firecrawl: Scrape URL -> markdown (or JSON with a schema)
Step 4    AI step: "Summarize plan names and prices; flag changes vs last week"
Step 5    Airtable: update the row with today's snapshot and summary
Step 6    Slack: post only rows where something changed

For code, a scheduled function works well; see Cloudflare Cron Triggers. Polling vs on-demand: API pull vs poll.

Step 5: Store, dedupe, and alert

  • Store each page by URL with the scrape time in Airtable, a sheet, or a database.
  • Compare with the previous version, or use Firecrawl's change tracking, and only alert when something changed.
  • Alert to one Slack channel or a daily digest; see daily digests.
  • Log failures and retry later; see 429 errors.

Best practices

PracticeWhy
Scrape the main content onlySmaller output, fewer tokens if AI reads it
Use map before crawlSee what you will pay for, and include or exclude paths
Set limitsCap pages per crawl and runs per day to control credits
Store by URL with a timestampEasy updates, history, and dedupe
Handle failuresRetry with backoff; log URLs that fail; alert on spikes
Respect site rulesFollow terms of service and robots rules; avoid personal data you don't need; don't overload sites

Next steps

Frequently asked questions

How do I automate web scraping with Firecrawl?

Choose the right endpoint (scrape, batch scrape, map, crawl, or search), request markdown or JSON, and run it on a schedule or trigger from Zapier, Make, n8n, or a cron job. Store results by URL, alert only on changes, and cap pages and runs.

What is the difference between Firecrawl scrape, crawl, and map?

Scrape returns the content of one URL. Map quickly returns the URLs found on a site without scraping them. Crawl follows links from a starting URL and scrapes the pages it finds as an asynchronous job.

Does Firecrawl work with Zapier, Make, and n8n?

Yes. Firecrawl has integrations with Zapier, Make, and n8n, so you can scrape pages on a schedule or trigger and pass the results to other apps. Check each platform for the current actions.

Can Firecrawl detect when a website changes?

Yes. Firecrawl offers change tracking and monitoring features that compare a page with its previous version, so you can alert only when something changes. Check current pricing for change tracking modes.

Is automated web scraping legal?

It depends on the site’s terms, the data, and the law where you operate. Follow terms of service and robots rules, avoid unnecessary personal data, and don’t overload sites. This is not legal advice.

FirecrawlWeb ScrapingAutomation

Disclaimer: Firecrawl endpoints, formats, credits, and plans change, and you are responsible for following each website's terms, robots rules, and applicable law; details reflect public documentation and coverage at the time of writing, so confirm current details with each provider before building. Automation Ace is not affiliated with Firecrawl beyond the referral links marked on this page. This article may include links to apps, products, or services; some links may be affiliate links, which means Automation Ace may earn a commission at no extra cost to you.

Build Better Systems

Ready to automate with confidence?

Share your tools, process, and goals. Automation Ace can design the Zapier, AI, and automation setup that fits your business.

Start a Project →