Knowledge Loading Flow

Trigger: Knowledge source created (inline during agent creation OR standalone via knowledge tools).

Purpose: Crawl website/sitemap content, extract documents, index into OpenSearch for RAG retrieval, and make knowledge available to agents.

Overview

Knowledge sources provide domain expertise to agents through Retrieval-Augmented Generation (RAG). When a source is created:

Source record created with loading_status = "unloaded"
Background crawler starts (if auto_load=True)
Content fetched from URLs (website crawl or sitemap parse)
Documents extracted and chunked
Content indexed into OpenSearch with embeddings
Status transitions to loaded (success) or failed (error)

Key principle: Loading is asynchronous. Agent can be used immediately; knowledge becomes available as loading completes.

Flow Steps

1. Source Record Created

Auto-resolve probe (when type is omitted or "website"):

When gnosari_manage_knowledge(action="create") is called with type omitted OR type="website" (and single_page is not set), the server calls resolve_website_source(url) before the UoW opens (the resolver does outbound HTTP, so it must run before any transaction). It probes, in order:

{url}/sitemap.xml
{url}/sitemap_index.xml
robots.txt Sitemap: directives (may be several, tried in order)

The first candidate with at least one <loc> entry wins → data_type="sitemap". If none yield a usable sitemap → data_type="discovery" (NeoReader crawls the site at load time). Only type="sitemap" or type="discovery" skip this probe entirely — the URL is used as-is. single_page=true is the single-page opt-out: it stores data_type="website" with no probe, regardless of type.

type omitted / "website" → resolve_website_source(url) → sitemap (if found) OR discovery (fallback)
type "sitemap" / "discovery" → used as-is, no probe
single_page=true → single-page website load, no probe

Two creation paths:

After agent creation (using gnosari_manage_knowledge)

python

result = gnosari_create(
    name="Support Bot",
    instructions="...",
    empty_state_title="Ask us anything",
    empty_state_description="We'll get back to you",
    data_collection=DataCollectionInput(...)
)

# Attach knowledge after creation
gnosari_manage_knowledge(
    action="create",
    name="Documentation",
    url="https://docs.example.com/sitemap.xml",
    type="sitemap",
    gnosari_id=result.agent.id  # creates AND assigns in one call
)

What happens:

gnosari_manage_knowledge(action="create") calls KnowledgeService.create_knowledge_source()
Parameters: auto_load=True (triggers loading automatically)
Source record created in PostgreSQL
Source immediately assigned to the agent

Standalone (separate tool call, no agent assigned yet)

python

gnosari_manage_knowledge(
    action="create",
    name="Product Docs",
    url="https://example.com/docs",
    type="website"
    # No gnosari_id — creates source only, assign later
)

What happens:

KnowledgeService creates source record with auto_load=True
Loading starts immediately
Source can later be assigned to one or more agents with action="assign"

2. Initial State

Source record created:

json

{
  "id": 456,
  "name": "Documentation",
  "data_type": "sitemap",
  "paths": ["https://docs.example.com/sitemap.xml"],
  "loading_status": "unloaded",
  "status": "active",
  "document_count": 0,
  "last_synced_at": null,
  "created_at": "2024-03-15T10:30:00Z"
}

data_type can be 'website' | 'sitemap' | 'discovery'. When type was omitted at create time, data_type reflects whichever the auto-resolve probe settled on (sitemap or discovery) — never the literal string "auto".

Loading status states:

unloaded: Created but not started
loading: Crawler running, indexing in progress
loaded: Complete and ready for RAG
failed: Error during loading

Status field (separate from loading_status):

active: Enabled for agent use
inactive: Disabled (not used for RAG even if loaded)

3. Loading Starts (Async)

Trigger: auto_load=True OR manual call to sync_knowledge_source()

What happens:

Background job queued (via Redis/Celery)
loading_status → "loading"
Job parameters:
- source_id: 456
- data_type: "website" or "sitemap"
- paths: URLs to crawl
- force: True (reload even if previously loaded)

Job execution:

Independent of agent creation flow
Doesn't block API response
Can take 1-30 minutes depending on site size

Agent availability: Agent can be used immediately even with loading_status="loading". RAG will use available knowledge (may be zero initially).

4. Content Crawling

For data_type = "sitemap":

Fetch sitemap.xml from URL
Parse XML for <url> entries
Extract list of page URLs
Fetch each page's HTML
Extract text content from HTML
Clean and normalize text

For data_type = "website":

Start at base URL
Fetch HTML page
Extract all links on page
Follow links recursively (up to configured depth)
Extract text content from each page
De-duplicate pages (by URL)

Crawl behavior:

Respects robots.txt
Rate limiting (configurable, default: 10 requests/sec)
Timeout per page (default: 30 seconds)
Max pages (default: 1000 pages per source)

Content extraction:

HTML → plain text (removes tags, scripts, styles)
Headings preserved for context
Links extracted for reference
Metadata captured (title, description, URL)

5. Document Chunking

Why chunking: LLM context windows and vector search require smaller units

Chunking strategy:

Target chunk size: 500-1000 tokens (~2000-4000 characters)
Overlap: 100 tokens (prevents context loss at boundaries)
Split on logical boundaries (paragraphs, headers)

Chunk metadata:

json

{
  "source_id": 456,
  "source_name": "Documentation",
  "url": "https://docs.example.com/api/authentication",
  "title": "Authentication Guide",
  "chunk_index": 2,
  "total_chunks": 5,
  "text": "To authenticate API requests, include your API key..."
}

Purpose: Each chunk becomes a searchable document in OpenSearch

6. OpenSearch Indexing

Index pattern: gnosari-{account_id} (tenant isolation)

For each chunk:

Generate embedding vector (via OpenAI/Cohere embedding model)
Index into OpenSearch with:
- text: Original chunk text
- embedding: Vector representation (1536 dimensions for text-embedding-ada-002)
- metadata: Source, URL, title, chunk position
- account_id: For filtering
- source_id: For source-specific queries

Index settings:

json

{
  "mappings": {
    "properties": {
      "embedding": {"type": "knn_vector", "dimension": 1536},
      "text": {"type": "text"},
      "url": {"type": "keyword"},
      "source_id": {"type": "integer"},
      "account_id": {"type": "integer"}
    }
  }
}

KNN (K-Nearest Neighbors): Enables semantic search by vector similarity

7. Loading Complete (Success)

Final state update:

json

{
  "loading_status": "loaded",
  "document_count": 247,
  "last_synced_at": "2024-03-15T10:45:32Z"
}

Document count: Total chunks indexed (not pages crawled)

Agent integration: Knowledge immediately available for RAG queries

RAG usage:

User asks: "How do I authenticate?"
Query embedded into vector
OpenSearch KNN search finds relevant chunks
Chunks injected into LLM context
LLM generates answer using knowledge

8. Loading Failed

Failure triggers:

Invalid URL (404, DNS failure)
Sitemap parse error (invalid XML)
Authentication required (401, 403)
Timeout (site too slow to respond)
Crawl limit exceeded (too many pages)

Final state:

json

{
  "loading_status": "failed",
  "error_message": "Failed to fetch sitemap: 404 Not Found",
  "document_count": 0,
  "last_synced_at": null
}

Recovery:

Check URL accessibility
Verify sitemap exists and is valid XML
Ensure no authentication walls
Fix issue
Retry: sync_knowledge_source(source_id=456, force=True)

Monitoring Loading Status

Via gnosari_get

Query:

python

gnosari_get(gnosari_id=123)

Response includes knowledge sources (via GnosariOverview.knowledge_sources):

json

{
  "gnosari_id": 123,
  "knowledge_sources": [
    {
      "id": 456,
      "name": "Documentation",
      "loading_status": "loading",
      "document_count": 0,
      "last_synced_at": null
    },
    {
      "id": 789,
      "name": "Blog Posts",
      "loading_status": "loaded",
      "document_count": 142,
      "last_synced_at": "2024-03-10T08:20:00Z"
    }
  ]
}

Status interpretation:

loading: Still in progress, check back later
loaded: Ready, document_count > 0
failed: Error, check error_message

Via gnosari_manage_knowledge(action="list")

Query all sources:

python

gnosari_manage_knowledge(action="list")

Filter by search term:

python

gnosari_manage_knowledge(action="list", search="documentation")

Response (PaginatedKnowledgeSourcesResponse):

json

{
  "data": [
    {
      "id": 456,
      "name": "Documentation",
      "type": "sitemap",
      "loading_status": "loading",
      "document_count": 0,
      "created_at": "2024-03-15T10:30:00Z",
      "last_synced_at": null
    }
  ],
  "total": 1
}

Polling Strategy

Recommended:

Create source with auto_load=True
Wait 5 seconds
Poll get_agent() every 10 seconds
Check if loading_status is "loaded" or "failed"
Stop polling when status changes from "loading"

Example:

python

# Create agent
result = gnosari_create(
    name="Support Bot",
    instructions="...",
    empty_state_title="Ask us anything",
    empty_state_description="We'll get back to you",
    data_collection=DataCollectionInput(...)
)
agent_id = result.agent.id

# Attach knowledge (loading starts automatically)
gnosari_manage_knowledge(
    action="create",
    name="Docs",
    url="https://docs.example.com/sitemap.xml",
    type="sitemap",
    gnosari_id=agent_id
)

# Poll for completion
import time

for _ in range(30):  # Max 5 minutes
    time.sleep(10)
    overview = gnosari_get(gnosari_id=agent_id)
    sources = overview.knowledge_sources

    if all(s.loading_status == "loaded" for s in sources):
        print(f"All sources loaded! Total docs: {sum(s.document_count for s in sources)}")
        break
    elif any(s.loading_status == "failed" for s in sources):
        print("Loading failed for some sources")
        break
    else:
        print("Still loading...")

Typical Loading Times

Site Size	Type	Estimated Time
Small sitemap (< 50 pages)	sitemap	1-2 minutes
Medium sitemap (< 500 pages)	sitemap	5-10 minutes
Large sitemap (> 1000 pages)	sitemap	15-30 minutes
Small website (< 100 pages)	website	3-5 minutes
Medium website (< 500 pages)	website	10-20 minutes
Large website (> 1000 pages)	website	20-40 minutes

Factors affecting time:

Page count
Page size (heavy content = slower)
Site responsiveness (slow server = slower crawl)
Network latency
Rate limiting (intentional throttling)

Optimization: Use sitemap type when available - faster than recursive website crawl.

Automatic Behaviors

Event	Automatic Action
Source created with `auto_load=True`	Loading starts immediately
Source created with `auto_load=False`	Stays "unloaded" until manual sync
Loading starts	Status → "loading", background job queued
Page fetched	Content extracted, chunked, embedded
Chunk created	Indexed into OpenSearch immediately
All pages processed	Status → "loaded", document_count updated
Error during crawl	Status → "failed", error_message set
Agent uses knowledge	OpenSearch KNN search queries indexed chunks

Side Effects

Database writes:

Source record created with initial status
Status updates during loading
document_count incremented as chunks indexed
last_synced_at timestamp updated on completion

OpenSearch writes:

Index created (if first source for account)
Documents added for each chunk
Embeddings stored as KNN vectors
Metadata indexed for filtering

Background jobs:

Crawler process spawned
HTTP requests to target URLs
Embedding API calls (OpenAI/Cohere)
OpenSearch bulk index operations

No agent downtime: Agent remains usable throughout loading process.

Error Handling

Common Failures

404 Not Found:

Cause: URL doesn't exist or moved
Fix: Verify URL, update paths if relocated
Retry: Update source with correct URL

Invalid sitemap XML:

Cause: Malformed XML, not a valid sitemap
Fix: Validate sitemap format, use sitemap validator
Retry: Fix sitemap or switch to data_type="website"

403 Forbidden / 401 Unauthorized:

Cause: Site requires authentication
Fix: Make content public OR use website scraping service
Retry: Not automatic - requires manual intervention

Timeout:

Cause: Site too slow, network issues
Fix: Check site performance, retry during off-peak
Retry: sync_knowledge_source(force=True)

Rate limiting:

Cause: Too many requests, site blocks crawler
Fix: Reduce crawl rate, contact site admin
Retry: Wait and retry with lower rate

Retry Mechanism

Manual retry:

python

sync_knowledge_source(source_id=456, force=True)

What force=True does:

Deletes existing indexed documents for this source
Re-crawls all pages from scratch
Re-indexes all content

When to retry:

After fixing URL errors
After site issues resolved
To refresh stale content (site updated)
After failed loading attempt

Automatic retry: Not implemented. All retries are manual.

Flow Diagram

Agent Creation Flow: Inline knowledge source creation
Knowledge System: Sources, RAG
Agent Creation Guide: Setup walkthrough

Knowledge Loading Flow ​

Overview ​

Flow Steps ​

1. Source Record Created ​

After agent creation (using gnosari_manage_knowledge) ​

Standalone (separate tool call, no agent assigned yet) ​

2. Initial State ​

3. Loading Starts (Async) ​

4. Content Crawling ​

5. Document Chunking ​

6. OpenSearch Indexing ​

7. Loading Complete (Success) ​

8. Loading Failed ​

Monitoring Loading Status ​

Via gnosari_get ​

Via gnosari_manage_knowledge(action="list") ​

Polling Strategy ​

Typical Loading Times ​

Automatic Behaviors ​

Side Effects ​

Error Handling ​

Common Failures ​

Retry Mechanism ​

Flow Diagram ​

Related Flows ​

Knowledge Loading Flow

Overview

Flow Steps

1. Source Record Created

After agent creation (using gnosari_manage_knowledge)

Standalone (separate tool call, no agent assigned yet)

2. Initial State

3. Loading Starts (Async)

4. Content Crawling

5. Document Chunking

6. OpenSearch Indexing

7. Loading Complete (Success)

8. Loading Failed

Monitoring Loading Status

Via gnosari_get

Via gnosari_manage_knowledge(action="list")

Polling Strategy

Typical Loading Times

Automatic Behaviors

Side Effects

Error Handling

Common Failures

Retry Mechanism

Flow Diagram

Related Flows