A practical field guide from Automation Ace.
Short answer: to build a self-updating AI knowledge base with Context.dev, crawl or scrape your sources into clean markdown, chunk and store them with their URL and fetch date, then create Context.dev monitors on the pages that matter. When a monitor detects a change it sends a signed webhook; your handler re-scrapes that page, replaces its old chunks, and logs the update. Your AI assistant, support bot, or agent then always answers from current content, with citations and a “last updated” date.
Monitor and webhook details reflect Context.dev's documentation and announcements at the time of writing; verify request and payload fields in Context.dev's documentation.
Short answer
- Load: crawl or scrape sources to markdown.
- Store: chunks with URL, title, and fetched date.
- Watch: monitors on important pages.
- Refresh: on webhook, re-scrape and replace that page's chunks.
- Answer: cite sources and show freshness.
Why knowledge bases go stale
Most AI knowledge bases are loaded once. Then pricing changes, policies update, and docs get rewritten, and the AI keeps answering from old copies, confidently. Re-crawling everything daily is slow and expensive; refreshing only what changed is the fix.
The architecture
Sources (your docs, help center, partner sites, regulations, competitor pages)
│ 1. Initial load: crawl -> markdown pages
▼
Processing: clean -> chunk -> embed -> store with URL, title, fetched_at, hash
▼
Knowledge store (vector DB, search index, or your AI tool's sources)
▲
│ 3. Re-fetch only changed pages -> re-chunk -> replace old chunks
Change detection: Context.dev monitors -> signed webhook "change.detected"
│ 2. Watch each source page on a schedule
▼
AI assistant / support bot / agent answers with citations + "last updated" date
Step 1: Choose and load sources
| Source type | How to load | How to keep fresh |
|---|---|---|
| Your docs or help center | Crawl with a URL pattern | Monitors on key sections, plus a weekly full re-crawl |
| Product and pricing pages | Scrape a URL list | Monitors on each page |
| Partner or vendor docs | Crawl allowed sections | Monitors; review terms of use |
| Regulations and policies | Scrape official pages | Monitors with alerts to a person |
| Industry news | Search with freshness filters | Scheduled search, not monitors |
Crawl and extraction basics: turning website data into automated workflows.
Step 2: Chunk, embed, and store with metadata
- Split markdown by headings into chunks of a few hundred words.
- Store each chunk with its URL, page title, section heading, fetched date, and a content hash.
- Use your vector database, a search index, or your AI tool's source manager.
Step 3: Watch pages with monitors
# Create a monitor that calls your webhook when a page changes
# (shape per Context.dev docs and examples at the time of writing; check the reference)
curl -X POST https://api.context.dev/v1/monitors \
-H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Pricing page",
"target": "https://example.com/pricing",
"webhook": { "url": "https://your-app.example.com/hooks/context-dev",
"events": ["change.detected"] }
}'
# Monitors can check frequently (as often as every 10 minutes per Context.dev);
# pick the slowest schedule that keeps answers fresh enough.
Step 4: Refresh on change via webhook
// Webhook handler sketch (Node.js): verify, then refresh one page
app.post("/hooks/context-dev", express.raw({ type: "*/*" }), async (req, res) => {
// Context.dev signs webhooks with HMAC-SHA256 in the X-Context-Signature header
// (timestamp + signature). Verify exactly as the docs describe before trusting it.
if (!verifyContextSignature(req.body, req.get("X-Context-Signature"), process.env.CONTEXT_DEV_WEBHOOK_SECRET))
return res.status(401).end();
const evt = JSON.parse(req.body.toString());
res.status(200).end(); // acknowledge fast
const url = evt.target ?? evt.url; // field names: check the payload docs
const page = await scrapeToMarkdown(url); // Context.dev scrape
if (hash(page.markdown) === await storedHash(url)) return; // nothing material changed
await deleteChunksFor(url);
await upsertChunks(url, chunk(page.markdown), { fetched_at: new Date() });
await notify(`Knowledge updated: ${url}`);
});
Webhook handling patterns: receiving webhooks with Cloudflare Workers and duplicate webhook deliveries.
Step 5: Answer with citations and dates
- Return source URLs and fetched dates with every answer.
- Prefer the newest chunk when sources conflict.
- Tell users when information may be out of date.
Quality practices
| Practice | Why |
|---|---|
| Store URL, title, and fetched date with every chunk | Citations and “last updated” in answers |
| Replace chunks per URL, don't append | No stale or duplicate answers |
| Hash content to skip trivial changes | Fewer re-embeddings and costs |
| Strip navigation and boilerplate | Better retrieval; scrape the main content |
| Keep a change log | Audit what changed and when |
| Alert a person on high-stakes changes | Policies and pricing need human eyes |
Where the knowledge lives
- Support bots: update a support agent's sources when pages change; see automating support with Chatbase.
- Internal assistants and agents: a vector database your agent queries as a tool; see how agents use tools.
- Research agents: combine the knowledge base with live search; see web research agents with Context.dev.
Costs and limits
- Each scrape uses Context.dev credits; monitors save money by triggering re-scrapes only on change.
- Re-embedding costs depend on your model; content hashes avoid unnecessary work.
- Choose monitor schedules by how fast each source changes and how much staleness you can accept.
- Follow each site's terms and robots rules for sources you don't own.
Related: scheduled scraping with Firecrawl.
Frequently asked questions
How do I build a self-updating AI knowledge base?
Crawl or scrape your sources into markdown, chunk and store them with URL and fetch date, set up Context.dev monitors on important pages, and when a change webhook arrives, re-scrape that page and replace its chunks so answers stay current.
What are Context.dev monitors?
Monitors watch a web page on a schedule and send a webhook when it changes. According to Context.dev, checks can run as often as every 10 minutes and webhooks are signed with HMAC-SHA256 in an X-Context-Signature header.
How do I keep RAG data from going stale?
Track a fetch date and content hash for every page, refresh only pages that changed using monitors or scheduled checks, replace old chunks per URL instead of appending, and show sources and dates in answers.
Should I re-crawl my whole site every day?
Usually not. Use monitors to refresh only changed pages and run a slower full re-crawl, such as weekly, to catch new pages. This keeps costs down and freshness high.
How do I verify Context.dev webhooks?
Use the webhook secret to verify the HMAC-SHA256 signature in the X-Context-Signature header, including its timestamp, exactly as Context.dev documents, before trusting the payload.
Disclaimer: Context.dev endpoints, parameters, credits, and features change, and you are responsible for following each website's terms, robots rules, and applicable law; details reflect public documentation and coverage at the time of writing, so confirm current details with each provider before building. Automation Ace is not affiliated with Context.dev beyond the referral links marked on this page. This article may include links to apps, products, or services; some links may be affiliate links, which means Automation Ace may earn a commission at no extra cost to you.