Knowledge Loading Guide

Knowledge sources provide context for AI agents through retrieval-augmented generation (RAG). This guide covers the complete flow from creation to search.

Overview

The knowledge loading system uses an asynchronous architecture:

API Endpoint → Queue Event → Background Worker → Engine → OpenSearch

Key principle: Loading happens in the background. The API returns immediately while processing continues asynchronously.

See: CLAUDE.md Knowledge Loading Flow

Step 1: Create Knowledge Source

Quickest path: let the server resolve the site

For a website source, the simplest request is a name plus the URL — omit data_type (or pass "auto") and the server resolves the whole site automatically (sitemap first, crawl-based discovery as fallback):

bash

curl -X POST "https://api.gnosari.com/api/v1/knowledge?auto_load=true" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Company Documentation",
    "paths": ["https://docs.company.com"]
  }'

Response:

json

{
  "id": 123,
  "identifier": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "name": "Company Documentation",
  "data_type": "sitemap",
  "paths": ["https://docs.company.com/sitemap.xml"],
  "loading_status": "unloaded",
  "loaded_at": null,
  "created_at": "2025-02-15T10:00:00Z"
}

data_type in the response reflects what the server resolved (sitemap if one was found, discovery otherwise) — not what you sent. name/description can be omitted too; they're derived from the URL's hostname when blank.

Explicit data type

Pass a concrete data_type when you already know the ingestion strategy (e.g. a Confluence space, or to force website/sitemap/discovery instead of letting the server pick):

bash

curl -X POST "https://api.gnosari.com/api/v1/knowledge?auto_load=true" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Company Documentation",
    "data_type": "confluence",
    "paths": ["https://docs.company.com/space/DOCS"],
    "description": "Internal documentation for all teams"
  }'

Response:

json

{
  "id": 123,
  "identifier": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "name": "Company Documentation",
  "data_type": "confluence",
  "paths": ["https://docs.company.com/space/DOCS"],
  "loading_status": "unloaded",
  "loaded_at": null,
  "created_at": "2025-02-15T10:00:00Z"
}

Note: Even with auto_load=true, the initial response shows loading_status: "unloaded" because loading is asynchronous. Poll the status to track progress.

Supported data types:

confluence: Confluence Wiki
website: Public web pages (single page or explicit URL list)
sitemap: Reads a sitemap.xml/sitemap_index.xml directly
discovery: Crawl-based whole-site discovery via NeoReader, used when no sitemap is found
pdf: PDF documents
markdown: Markdown files
text: Plain text documents

See: Knowledge API Reference

Step 2: Poll Loading Status

Knowledge sources transition through these states:

Status	Description	Action
`unloaded`	Not yet loaded	Wait or trigger manual load
`loading`	Currently processing	Poll every 2 seconds
`loaded`	Ready for use	Agent can search
`failed`	Loading failed	Check logs, retry

Poll for status updates:

bash

# Poll every 2 seconds while loading
curl "https://api.gnosari.com/api/v1/knowledge/123" \
  -H "Authorization: Bearer $TOKEN"

Response when loaded:

json

{
  "id": 123,
  "name": "Company Documentation",
  "loading_status": "loaded",
  "loaded_at": "2025-02-15T10:05:30Z",
  "document_count": 342,
  "last_indexed_at": "2025-02-15T10:05:30Z"
}

Step 3: Verify Indexing

Once loading_status is loaded, the knowledge source is indexed in OpenSearch.

Check index stats:

bash

curl "https://api.gnosari.com/api/v1/knowledge/123/stats" \
  -H "Authorization: Bearer $TOKEN"

Response:

json

{
  "knowledge_source_id": 123,
  "index_name": "gnosari-456-knowledge",
  "document_count": 342,
  "index_size_bytes": 15728640,
  "last_indexed_at": "2025-02-15T10:05:30Z"
}

List indexed documents:

bash

curl "https://api.gnosari.com/api/v1/knowledge/123/documents?limit=10" \
  -H "Authorization: Bearer $TOKEN"

Response:

json

{
  "documents": [
    {
      "id": "doc_abc123",
      "title": "Getting Started Guide",
      "content_preview": "Welcome to our platform...",
      "url": "https://docs.company.com/getting-started",
      "indexed_at": "2025-02-15T10:05:20Z"
    }
  ],
  "total": 342,
  "limit": 10,
  "offset": 0
}

Step 4: Search Knowledge

Test the knowledge source with a direct search:

bash

curl -X POST "https://api.gnosari.com/api/v1/knowledge/search" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "authentication setup",
    "knowledge_source_ids": [123],
    "limit": 5
  }'

Response:

json

{
  "results": [
    {
      "content": "## Authentication Setup\n\nTo configure authentication...",
      "metadata": {
        "title": "Authentication Guide",
        "url": "https://docs.company.com/auth"
      },
      "score": 0.89
    }
  ],
  "total_results": 5,
  "query": "authentication setup"
}

Search parameters:

query: Search string
knowledge_source_ids: Filter to specific sources
limit: Max results (default: 10, max: 50)
min_score: Minimum relevance score (0.0-1.0)

Manual Preload/Reload

Force reload a knowledge source (e.g., after source content changes):

bash

curl -X POST "https://api.gnosari.com/api/v1/knowledge/123/preload?force=true" \
  -H "Authorization: Bearer $TOKEN"

Response:

json

{
  "success": true,
  "message": "Knowledge source loading has been queued",
  "event_id": "knowledge_load_123_xyz",
  "status": "queued",
  "note": "Check the knowledge source loading_status field to track progress"
}

When to reload:

Source content has been updated
Initial load failed and you've fixed the configuration
OpenSearch index was manually deleted

Agent Integration

Once knowledge is loaded, agents with the knowledge-retrieval tool can search it:

bash

# 1. Add knowledge-retrieval tool to agent
curl -X POST "https://api.gnosari.com/api/v1/agents/42/tools/knowledge-retrieval" \
  -H "Authorization: Bearer $TOKEN"

# 2. Agent now searches all knowledge sources for the account automatically
# The agent uses the knowledge-retrieval tool when it needs context

How agents use knowledge:

User asks a question requiring context
Agent decides to use the knowledge-retrieval tool
Tool searches ALL knowledge sources for the account
Relevant chunks are injected into the conversation context
Agent answers using the retrieved information

See: Agent Setup Guide

Data Flow: API → Engine → Library

The knowledge loading chain:

1. API (constructs index_name)
   ↓ index_name = "gnosari-{account_id}-knowledge"
   ↓
2. Dispatcher (enqueues to Redis Streams)
   ↓ Event: {"knowledge_source_id": 123, "account_id": 456}
   ↓
3. KnowledgeLoadingHandler (processes event)
   ↓ Calls engine.KnowledgeLoaderService
   ↓
4. Engine (orchestrates loading)
   ↓ Delegates to gnosisllm-knowledge
   ↓
5. gnosisllm-knowledge (OpenSearch operations)
   ↓ Indexes documents with embeddings
   ↓
6. OpenSearch (stores vectors + metadata)

Key files:

API dispatcher: core/services/knowledge/knowledge_orchestrator.py
Queue handler: app/queue/handlers/knowledge_loading_handler.py
Engine service: engine/services/knowledge_loader_service.py
Library: libraries/gnosisllm-knowledge/knowledge.py

Troubleshooting

Status stuck on "loading"

Check:

Background worker is running: docker-compose logs -f queue-worker
Redis connection is working
OpenSearch is accessible: curl http://opensearch:9200/_cluster/health
API logs for error messages

Status changes to "failed"

Check:

API logs: docker-compose logs python-api | grep knowledge
Worker logs: docker-compose logs queue-worker | grep knowledge
OpenSearch logs for indexing errors
Environment variables (see Configuration section below)

Search returns no results

Check:

Knowledge source is loaded: GET /knowledge/123
Documents were indexed: GET /knowledge/123/stats shows document_count > 0
Query is relevant to indexed content
Try lowering min_score threshold

Agent doesn't use knowledge

Check:

Agent has knowledge-retrieval tool: GET /agents/42/tools
Knowledge source is loaded
User query requires external context (agent decides when to use tools)
LLM API keys are configured

Configuration

Knowledge loading requires these environment variables in the API .env:

bash

# OpenSearch connection (from gnosisllm-knowledge)
OPENSEARCH_HOST=opensearch
OPENSEARCH_PORT=9200
OPENSEARCH_USER=admin
OPENSEARCH_PASSWORD=admin
OPENSEARCH_USE_SSL=false

# Index naming (single shared knowledge index; tenant isolation via metadata.account_id filter)
GNOSARI_KNOWLEDGE_INDEX_NAME=gnosari-knowledge

# Embeddings (from gnosisllm-knowledge)
OPENAI_API_KEY=sk-...

# Queue system
REDIS_URL=redis://redis:6379/0

Critical: The API must include ALL environment variables needed by both the engine AND gnosisllm-knowledge.

See: Configuration Reference

Best Practices

Content Organization

One source per domain: Don't mix HR docs with engineering docs
Descriptive names: "Q4 Sales Documentation" not "Docs 1"
Regular reloads: Schedule weekly reloads for frequently updated sources

Performance

Chunk size: Default is 1000 characters. Smaller chunks = more precise, larger chunks = more context
Query specificity: Specific queries ("JWT authentication flow") return better results than vague queries ("security")
Result limits: Request only what you need (default 10 is usually sufficient)

Security

Private sources: Knowledge sources inherit account isolation. Source A's data never appears in Source B's searches.
Sensitive data: Don't index secrets, credentials, or PII
Access control: Knowledge sources respect agent access levels

Next Steps

Set up entity extraction: Data Collection Guide
Create event listeners: Event-Driven Automation
Deploy to production: Background Systems

Knowledge Loading Guide ​

Overview ​

Step 1: Create Knowledge Source ​

Quickest path: let the server resolve the site ​

Explicit data type ​

Step 2: Poll Loading Status ​

Step 3: Verify Indexing ​

Step 4: Search Knowledge ​

Manual Preload/Reload ​

Agent Integration ​

Data Flow: API → Engine → Library ​

Troubleshooting ​

Status stuck on "loading" ​

Status changes to "failed" ​

Search returns no results ​

Agent doesn't use knowledge ​

Configuration ​

Best Practices ​

Content Organization ​

Performance ​

Security ​

Next Steps ​

Related References ​

Knowledge Loading Guide

Overview

Step 1: Create Knowledge Source

Quickest path: let the server resolve the site

Explicit data type

Step 2: Poll Loading Status

Step 3: Verify Indexing

Step 4: Search Knowledge

Manual Preload/Reload

Agent Integration

Data Flow: API → Engine → Library

Troubleshooting

Status stuck on "loading"

Status changes to "failed"

Search returns no results

Agent doesn't use knowledge

Configuration

Best Practices

Content Organization

Performance

Security

Next Steps

Related References