Data Collection System

What it is: Automatic extraction of structured data from conversations with agents. Instead of forms or surveys, users chat naturally and the AI extracts the data you need.

Why it exists: Traditional data collection requires interrupting users with forms. Gnosari agents collect data conversationally, making it feel natural while capturing structured information for CRM, analytics, and automation.

What is Data Collection?

Data collection in Gnosari transforms conversations into structured records. As users chat with your agent, the AI:

1. Identifies relevant information based on your template
2. Extracts data into structured fields
3. Assigns confidence scores
4. Stores records for review and action

Example conversation:

User: "I'm interested in your enterprise plan for my company."
Agent: "Great! I can help with that. What's your company name?"
User: "Acme Corp. I'm the CTO, John Smith. Email is john@acme.com"

Extracted data:

{
  "company": "Acme Corp",
  "title": "CTO",
  "name": "John Smith",
  "email": "john@acme.com",
  "interest": "enterprise plan"
}

Templates

Templates define what data to collect. A template is a named collection of fields with types, validation, and AI hints.

Template Structure

Property	Purpose
template_name	Identifier (e.g., "Lead Info", "Support Ticket")
fields	Array of field definitions
collection_mode	How aggressive to be (passive, opportunistic, active, guided)
custom_prompt	Optional instructions for guided mode

Field Definition

Each field has:

Attribute	Purpose	Required
name	Field identifier (snake_case)	Yes
type	Data type (text, email, number, date, boolean)	Yes
required	Whether field must be filled	No (default: false)
ai_hint	Guidance for extraction AI	No (recommended)

Example template:

{
    "template_name": "Lead Capture",
    "fields": [
        {
            "name": "full_name",
            "type": "text",
            "required": True,
            "ai_hint": "Person's complete name"
        },
        {
            "name": "email",
            "type": "email",
            "required": True,
            "ai_hint": "Contact email address"
        },
        {
            "name": "company",
            "type": "text",
            "required": False,
            "ai_hint": "Company or organization name"
        },
        {
            "name": "interest",
            "type": "text",
            "required": False,
            "ai_hint": "What product/service they're interested in"
        }
    ],
    "collection_mode": "active"
}

Field Types

Type	Validation	Example
text	Free-form string	"Product Manager", "Acme Corporation"
email	Email format	"john@example.com"
number	Numeric value	42, 99.95
date	ISO 8601 date	"2024-03-15"
boolean	True/false	true, false

Type validation: The extraction engine validates data types and flags mismatches (e.g., "john at example dot com" would fail email validation).

AI Hints

AI hints guide the extraction engine. Good hints improve accuracy:

Poor hint: "name" (too vague) Better hint: "Person's full name (first and last)"

Poor hint: "info" (what kind?) Better hint: "Description of the technical issue they're experiencing"

Best practice: Write hints as if explaining to a human what to extract.

Collection Modes

Collection modes control how aggressive the agent is about collecting data.

passive

Behavior: Extract data silently. Never ask questions. Only capture what the user volunteers.

Best for:

• Sentiment analysis
• Topic tracking
• Passive monitoring
• Cases where asking feels intrusive

Example use case: Customer support agent that logs issues mentioned but doesn't ask for details - just extracts what's naturally mentioned.

User experience: Seamless - user doesn't know data is being collected.

opportunistic

Behavior: Capture data when mentioned. Ask brief follow-up questions if important fields are missing.

Best for:

• Support ticket logging
• Feedback collection
• Cases where some follow-up is acceptable

Example conversation:

User: "The app keeps crashing on my phone."
Agent: "I'm sorry to hear that. To help you better, what's your email address so I can follow up?"

User experience: Mostly natural with occasional targeted questions.

active

Behavior: Proactively ask for required fields. Guide the conversation toward data collection.

Best for:

• Lead capture
• Job applications
• Booking/registration
• Cases where data collection is the primary goal

Example conversation:

Agent: "I'd love to help you get started! First, what's your name?"
User: "Sarah Chen"
Agent: "Great to meet you, Sarah! What email should I use to send you the details?"

User experience: Clear data collection intent. Feels like a structured conversation.

guided

Behavior: Follow a custom script/flow defined in custom_prompt. Ask questions in order.

Best for:

• Multi-step workflows
• Surveys with specific question order
• Booking with availability checks
• Complex qualification flows

Requires: custom_prompt parameter with step-by-step instructions

Example:

{
    "collection_mode": "guided",
    "custom_prompt": """
    1. Greet warmly and ask what service they need
    2. Ask for their preferred date and time
    3. Collect name and contact info
    4. Confirm booking details
    """
}

User experience: Structured interview-style conversation.

Data Lifecycle

Collected data moves through states as it's processed:

NEW → REVIEWED → ACTIONED → ARCHIVED

NEW

What it means: Data just extracted from a conversation. Needs review.

Typical actions:

• Review for accuracy
• Verify contact information
• Qualify lead
• Triage support ticket

REVIEWED

What it means: Human verified the data. Confirmed accurate and actionable.

Typical actions:

• Export to CRM
• Assign to sales rep
• Schedule follow-up
• Begin processing

ACTIONED

What it means: Action taken on the data (email sent, ticket created, deal initiated).

Typical actions:

• Monitor for response
• Track to closure
• Update status in external system

ARCHIVED

What it means: Data is complete/closed. Kept for historical record.

Typical actions:

• Remove from active dashboard
• Include in reports
• Retain for compliance

Note: Status transitions are manual via the UI or API. The MCP server provides read-only access to collected data.

Confidence Scores

Every extracted field has a confidence score (0.0-1.0) indicating AI certainty:

Score Range	Meaning	Action
0.9-1.0	Very confident	Likely accurate, safe to automate
0.7-0.89	Confident	Probably accurate, quick review recommended
0.5-0.69	Uncertain	Needs verification
0.0-0.49	Very uncertain	Manual review required

Example:

{
  "email": {
    "value": "john@acme.com",
    "confidence": 0.98
  },
  "company": {
    "value": "Acme Corp maybe?",
    "confidence": 0.52
  }
}

Best practice: Set automation thresholds based on your tolerance for errors. Conservative: 0.9+. Aggressive: 0.7+.

How Extraction Works

During Conversation

1. User sends message: "I'm Sarah from Acme Corp, interested in your API"
2. Agent responds naturally: "Great to meet you, Sarah! Let me tell you about our API..."
3. Background extraction: AI analyzes message for template fields
4. Data captured: name="Sarah", company="Acme Corp", interest="API"
5. Confidence scored: Each field gets a confidence score
6. Storage: Record saved to database with status=NEW

After Conversation

The collected data is available via:

• list_collected_data() - Filter and search
• get_collection_stats() - Aggregated statistics
• Gnosari UI - Dashboard views

Real-time vs batch: Extraction happens during the conversation. Data is immediately available for retrieval.

Monitoring Collected Data

Listing Data

Filter collected data with list_collected_data():

# Get all NEW leads from the last 7 days
list_collected_data(
    template_name="Lead Capture",
    status="NEW",
    days=7
)

# Find high-confidence enterprise leads
list_collected_data(
    agent_id=123,
    min_confidence=0.9,
    search="enterprise"
)

Statistics

View collection performance with get_collection_stats():

# Last 30 days across all agents
get_collection_stats(days=30)

# Per-agent performance
get_collection_stats(agent_id=123, days=90)

Returns:

• Total entries by template
• Breakdown by status (NEW, REVIEWED, ACTIONED, ARCHIVED)
• Collection rate trends

Best Practices

Template Design

Do:

• Keep templates focused (5-10 fields max)
• Use clear field names (full_name not fn)
• Provide specific AI hints
• Mark truly required fields only

Don't:

• Create kitchen-sink templates (split into multiple templates)
• Use ambiguous field names (info, data, field1)
• Skip AI hints (they improve accuracy significantly)
• Mark everything as required (users will abandon)

Collection Mode Selection

Goal	Mode
Don't interrupt conversation	passive
Capture feedback naturally	opportunistic
Maximize data capture	active
Follow specific script	guided

Confidence Thresholds

High-value actions (auto-email, create account): 0.9+ Medium-value (CRM entry, notification): 0.7+ Review queue: 0.5-0.7 Flag for manual review: <0.5

Related Concepts

• Agent Lifecycle: How to configure agents with data collection
• Data Collection Flow: Complete extraction workflow
• Data Collection Guide: Step-by-step setup, including the enable_attachments toggle for file uploads/attachments (a separate per-agent capability, not a data-collection field)