A practical field guide from Automation Ace.
Short answer: to turn any website into structured data with Firecrawl and AI, define a JSON schema for the fields you want (for example plan name, price, and features), call Firecrawl's scrape endpoint with a JSON format that includes the schema and a short prompt, and Firecrawl returns data shaped to your schema instead of raw HTML. Validate the output, store it by URL in Airtable, a sheet, or a database, and re-run on a schedule from Zapier, Make, or n8n to keep it fresh. For many pages, use batch scrape or crawl with the same schema.
Request formats and credit costs reflect Firecrawl's documentation at the time of writing; check the current API reference.
Short answer
- Write a small JSON schema for the fields you need.
- Scrape with the JSON format: schema plus prompt.
- Validate types and required fields; send failures to review.
- Upsert by URL with a timestamp.
- Schedule refreshes and alert on changes.
How it works
Firecrawl renders the page, extracts the main content, and uses an LLM to fill your schema from it. You get JSON back with the page metadata. Because the output follows your schema, the next step in your workflow can map fields directly, with no regex or HTML selectors that break when the site's layout changes. For the difference between parsing, scraping, and crawling, see web parsing vs web scraping vs web crawling.
What you can extract
| Website type | Fields to extract | Where it goes |
|---|---|---|
| Competitor pricing pages | Plan, price, billing period, key limits | Airtable table with weekly history |
| Company websites (leads) | Industry, size signals, products, contact page, tech hints | CRM enrichment fields |
| Job boards and careers pages | Title, location, department, date posted | Recruiting or sales-signal sheet |
| Directories and listings | Name, category, address, phone, website | Prospect list (respect terms and privacy) |
| Product catalogs | SKU, name, price, availability | Price monitoring and alerts |
| Event or news pages | Title, date, location, link, summary | Content calendar or digest |
Step 1: Design the schema
| Tip | Why |
|---|---|
| Keep the schema small | Fewer fields, more accurate extraction |
| Allow null for optional fields | The model can say “not found” instead of guessing |
| Use enums for categories | Consistent values you can filter on |
| Ask for numbers as numbers | No currency symbols or commas to clean later |
| Add a source or evidence field when it matters | Easier review and fewer hallucinated values |
| Version your schema | Changes don't silently break downstream steps |
Step 2: Extract with Firecrawl
# Ask Firecrawl for JSON that matches your schema (v2 format; check the current reference)
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H "Authorization: Bearer $FIRECRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/pricing",
"formats": [{
"type": "json",
"prompt": "Extract every pricing plan shown on the page.",
"schema": {
"type": "object",
"properties": {
"plans": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": { "type": "string" },
"monthly_price": { "type": ["number", "null"] },
"currency": { "type": "string" },
"billing_note": { "type": "string" },
"features": { "type": "array", "items": { "type": "string" } }
},
"required": ["name"]
}
}
},
"required": ["plans"]
}
}]
}'
# -> data.json = { "plans": [ { "name": "Pro", "monthly_price": 49, ... } ] }
Schema, prompt, or your own AI step
| Approach | How it works | When to use it |
|---|---|---|
| Schema plus prompt | You define fields and types; the prompt explains what to look for | Production data you will store or act on |
| Prompt only | Describe what you want; the model chooses the structure | Exploration and one-off research |
| Markdown, then your own AI step | Scrape markdown, then extract with your own model and prompt | You need a specific model, extra context, or custom logic |
| Multi-page extraction | Crawl or batch scrape with JSON format per page | Many similar pages, such as listings or products |
Firecrawl has also offered dedicated extraction and agent endpoints for multi-page or discovery-heavy jobs; check the current docs for what fits. To have an agent decide which pages to read, see AI agents that research websites using Firecrawl.
Step 3: Validate before you trust it
- Check required fields exist and types match (numbers are numbers, dates are dates).
- Range-check values (a $49,000 monthly price is probably wrong).
- Send invalid results to a review view with the source markdown; see handling invalid JSON from AI steps.
- Spot-check a sample against the live page when you change the schema or prompt.
Step 4: Store and keep it fresh
- Upsert by URL so re-runs update rows instead of duplicating them.
- Keep
extracted_atand, for prices, a history table. - Re-run on a schedule; alert only when key fields change.
- Use change tracking where available to skip unchanged pages.
Scheduling patterns: how to automate web scraping with Firecrawl.
A complete pipeline
Pipeline: URLs -> structured rows
1. Source Airtable view "To extract" (url, type)
2. Extract Firecrawl scrape with the JSON schema for that type
3. Validate Check required fields, types, and ranges (code step)
invalid -> "Needs review" view with the raw markdown attached
4. Upsert Write fields to the row, matched by URL; set extracted_at
5. Compare If price or key fields changed since last run -> Slack alert
6. Schedule Re-run weekly for rows older than 7 days
Costs and limits
- Firecrawl bills in credits; at the time of writing its docs describe JSON extraction as costing more per page than a plain scrape (for example 5 credits per page instead of 1), so check current pricing.
- Crawling with JSON multiplies that by the number of pages, so set page limits.
- If you run your own AI step on markdown, count the model's token costs too; see what AI costs to run.
Responsible extraction
- Follow each site's terms and robots rules.
- Avoid collecting personal data you don't need, and handle what you do collect under applicable privacy law.
- Keep request rates reasonable.
Related: extracting emails, phones, and IDs with regex for simple patterns that don't need AI.
Frequently asked questions
How do I turn a website into structured data?
Define a JSON schema for the fields you want, then use Firecrawl’s scrape endpoint with a JSON format that includes the schema and a prompt. Firecrawl returns data shaped to your schema, which you validate and store by URL.
Does Firecrawl use AI to extract data?
Yes. In JSON mode, Firecrawl uses an LLM to fill your schema or follow your prompt from the page content, so you don’t need CSS selectors or regex for each site.
Should I use a schema or just a prompt?
Use a schema for production data you will store or act on, because it gives consistent fields and types. Prompt-only extraction is fine for exploration and one-off research.
How much does Firecrawl JSON extraction cost?
Firecrawl bills in credits. At the time of writing, its documentation describes JSON extraction as costing more per page than a plain scrape, for example 5 credits instead of 1. Check current pricing.
How do I keep extracted website data up to date?
Store results by URL with a timestamp, re-run extraction on a schedule, alert only when key fields change, and use change tracking where available to skip unchanged pages.
Disclaimer: Firecrawl endpoints, formats, credits, and plans change, and you are responsible for following each website's terms, robots rules, and applicable law; details reflect public documentation and coverage at the time of writing, so confirm current details with each provider before building. Automation Ace is not affiliated with Firecrawl beyond the referral links marked on this page. This article may include links to apps, products, or services; some links may be affiliate links, which means Automation Ace may earn a commission at no extra cost to you.