Knowledge System

What it is: URL-based content that agents can reference during conversations. Knowledge sources are websites or documents that get crawled, indexed, and made searchable for RAG (Retrieval Augmented Generation).

Why it exists: Agents need domain expertise to answer questions accurately. Instead of embedding all knowledge in agent instructions, knowledge sources provide dynamic, updatable content that agents retrieve on-demand.

What is a Knowledge Source?

A knowledge source is a URL that Gnosari crawls and indexes. Once loaded, agents can reference the content to answer questions. Sources are assigned directly to agents.

Example (recommended — omit type, let the server figure out the best ingestion strategy):

python

gnosari_manage_knowledge(action="create", name="Product Documentation", url="https://docs.example.com")

When a user asks "How do I integrate your API?", the agent searches this knowledge source and retrieves relevant documentation to answer accurately.

Source Types

Omit type for whole-site auto-resolve — this is the recommended default, not just an edge case. When type is not passed on gnosari_manage_knowledge(action="create"), the server probes the URL for a sitemap (/sitemap.xml, then /sitemap_index.xml, then any Sitemap: directives in robots.txt) and uses the first one that lists at least one page. If none is found, it falls back to discovery — a crawl-based ingestion with no sitemap required. You only need to pass type explicitly when you want to force a specific ingestion strategy (see Manage Resources reference for the full probe order and error contract).

Knowledge sources come in three types:

website

What it does: Recursively crawls a website starting from the URL.

Behavior:

Starts at the provided URL
Follows internal links
Crawls up to configured depth (default: 3 levels)
Respects robots.txt
Indexes all discovered pages

Best for:

Documentation sites
Marketing websites
Blogs
Any site with interconnected pages

Example:

python

{
    "name": "Help Center",
    "url": "https://help.example.com",
    "type": "website"
}

Crawl process: Starts at /, discovers /getting-started, /api, /faq, etc., and indexes all pages.

sitemap

What it does: Loads URLs from a sitemap.xml file.

Behavior:

Fetches sitemap.xml at the URL
Extracts all listed URLs
Indexes each URL directly
No additional crawling

Best for:

Large sites where recursive crawl would be slow
Sites with well-maintained sitemaps
Targeting specific pages
Avoiding crawl depth limits

Example:

python

{
    "name": "Documentation",
    "url": "https://docs.example.com/sitemap.xml",
    "type": "sitemap"
}

Performance: Faster and more precise than recursive crawl. Use when available.

discovery

What it does: Whole-site, crawl-based discovery via NeoReader at load time — no sitemap needed.

Behavior:

No sitemap.xml or robots.txt Sitemap: directive found at the URL
NeoReader's Discovery API crawls the site starting from the URL when loading runs
Indexes discovered pages, same as a website crawl, but driven by NeoReader instead of a recursive in-process crawler

Best for:

Sites without a maintained sitemap
The automatic fallback target when auto-resolve can't find a usable sitemap — you rarely set discovery explicitly; omit type instead and let the server land here on its own

Example (explicit, forces discovery even if a sitemap exists):

python

gnosari_manage_knowledge(action="create", name="Marketing Site", url="https://example.com", type="discovery")

Loading States

Knowledge sources progress through states as they're processed:

unloaded -> loading -> loaded
                  |
               failed

unloaded

What it means: Source created but crawling hasn't started.

When: Immediately after creation, before background job picks it up.

Duration: Seconds to minutes depending on queue load.

User action: Wait for processing to begin.

loading

What it means: Crawling in progress. Pages are being fetched and indexed.

When: Background job is actively crawling the URL.

Duration: Seconds to hours depending on:

Source type (sitemap is faster than website)
Number of pages
Site response time
Crawl depth

User action: Monitor progress. Check pages_indexed count if available.

loaded

What it means: Crawling complete. Content indexed and ready for RAG retrieval.

When: All pages successfully crawled and indexed.

User action: Knowledge is ready. Agents can use this source immediately.

failed

What it means: Crawling encountered an error and stopped.

Common causes:

URL not accessible (404, 500, DNS error)
robots.txt blocks crawling
Timeout (site too slow)
Invalid sitemap.xml
Authentication required (knowledge sources must be public)

User action: Check error message, fix URL or permissions, try again.

Assigning Sources to Agents

Sources are assigned directly to agents. A single source can be shared across multiple agents.

Create sources and assign to agents:

python

# Create sources once — omit `type` (or pass "website") and let the server auto-resolve the whole site (recommended)
faq_src = gnosari_manage_knowledge(action="create", name="FAQ", url="https://example.com/faq")
docs_src = gnosari_manage_knowledge(action="create", name="Docs", url="https://docs.example.com")
# Opt out of whole-site ingestion with single_page when you only want one page
tutorials_src = gnosari_manage_knowledge(action="create", name="Tutorials Page", url="https://example.com/tutorials", single_page=True)

# Assign to multiple agents
gnosari_manage_knowledge(action="assign", source_ids=[faq_src.id, docs_src.id, tutorials_src.id], gnosari_id=support_bot_id)
gnosari_manage_knowledge(action="assign", source_ids=[faq_src.id, docs_src.id, tutorials_src.id], gnosari_id=onboarding_bot_id)
gnosari_manage_knowledge(action="assign", source_ids=[faq_src.id, docs_src.id, tutorials_src.id], gnosari_id=sales_bot_id)

Benefits:

Reuse across agents -- one source, many agents
Update knowledge in one place (re-crawl the source once)
Standardize knowledge sets
Easier maintenance

RAG Integration

RAG (Retrieval Augmented Generation) is how agents use knowledge during conversations.

How RAG Works

User asks question: "How do I reset my password?"
Agent detects knowledge need: Question requires domain-specific information
Semantic search: Agent queries knowledge index for relevant content
Context retrieval: Top matching pages/chunks returned
Response generation: Agent uses retrieved context + LLM to answer
Citation: Agent may cite sources in response

Example Flow

User: "What's your refund policy?"
        |
Agent searches knowledge index for "refund policy"
        |
Retrieved: "Refunds are available within 30 days of purchase..."
        |
Agent: "We offer refunds within 30 days of purchase. You can request
       a refund by contacting support@example.com. Would you like
       me to explain the process?"

Query Optimization

The agent automatically:

Reformulates user questions for better search
Filters by relevance score
Combines results from multiple sources
Prioritizes recent/updated content

You don't configure RAG behavior - it's automatic when knowledge sources are attached.

Monitoring Knowledge Sources

Listing Sources

python

# All sources
gnosari_manage_knowledge(action="list")

# With search filter
gnosari_manage_knowledge(action="list", search="docs")

Checking Load Status

python

# Get specific source
source = gnosari_manage_knowledge(action="get", source_id=123)

# Check status
if source.loading_status == "loaded":
    print(f"Ready! Indexed {source.document_count} documents")
elif source.loading_status == "loading":
    print("Still crawling...")
elif source.loading_status == "failed":
    print(f"Error: {source.error_message}")

Troubleshooting Failed Sources

Error	Cause	Fix
"404 Not Found"	URL doesn't exist	Check URL, fix typo
"403 Forbidden"	robots.txt blocks	Get permission or use different URL
"Timeout"	Site too slow	Use sitemap.xml instead of recursive crawl
"Invalid sitemap"	XML format error	Validate sitemap.xml structure
"Authentication required"	URL behind login	Knowledge sources must be public

Best Practices

Choosing Source Type

Omit type (or pass "website") as the default — whole-site auto-resolve probes for a sitemap and falls back to discovery automatically. This is the recommended path for almost every knowledge source; you don't need to know in advance whether the target site has a sitemap.

Override the auto-resolve result only when you have a reason to:

Use single_page=true when:

You want ONLY the given URL loaded, not the whole site
A single pricing/landing/FAQ page is all the agent needs
Stored as data_type="website" (one page), no probe

Use type="discovery" when:

No sitemap.xml available and you want to force crawl-based discovery
You want to skip the sitemap probe and go straight to discovery

Use type="sitemap" when:

Large sites (1000+ pages) with a maintained sitemap.xml
You want faster loading and precise control over indexed pages
You want to point at a specific sitemap URL and skip the probe

Organizing Knowledge

Single source per domain concept:

python

# Good - focused sources
{"name": "API Docs", "url": "https://docs.example.com/api"}
{"name": "User Guide", "url": "https://docs.example.com/guide"}

# Bad - one massive source
{"name": "Everything", "url": "https://example.com"}

Performance Optimization

Sitemap over website crawl: 5-10x faster for large sites

Update frequency: Re-crawl when content changes significantly, not on every edit

Size limits: Keep individual sources under 10,000 pages. Split large sites into multiple sources.

Automatic Behaviors

When you create a knowledge source:

Status starts as "unloaded"
Background job queues for processing
Crawling begins automatically (no manual trigger needed)

When you attach a source to an agent:

Agent can use it immediately if status is "loaded"
If still "loading", agent waits until ready
If "failed", agent won't use it (no error thrown)

When you delete a knowledge source:

Source removed from all agents that reference it
Indexed content is deleted from OpenSearch
PostgreSQL record is removed

Agent Lifecycle: How to attach knowledge to agents
Knowledge Loading Flow: Complete crawling workflow
Agent Creation Guide: Creating agents with knowledge

Knowledge System ​

What is a Knowledge Source? ​

Source Types ​

website ​

sitemap ​

discovery ​

Loading States ​

unloaded ​

loading ​

loaded ​

failed ​

Assigning Sources to Agents ​

RAG Integration ​

How RAG Works ​

Example Flow ​

Query Optimization ​

Monitoring Knowledge Sources ​

Listing Sources ​

Checking Load Status ​

Troubleshooting Failed Sources ​

Best Practices ​

Choosing Source Type ​

Organizing Knowledge ​

Performance Optimization ​

Automatic Behaviors ​

Related Concepts ​

Knowledge System

What is a Knowledge Source?

Source Types

website

sitemap

discovery

Loading States

unloaded

loading

loaded

failed

Assigning Sources to Agents

RAG Integration

How RAG Works

Example Flow

Query Optimization

Monitoring Knowledge Sources

Listing Sources

Checking Load Status

Troubleshooting Failed Sources

Best Practices

Choosing Source Type

Organizing Knowledge

Performance Optimization

Automatic Behaviors

Related Concepts