Data Collection Flow

Trigger: Conversation with an agent that has data collection configured (entity_type_configs present).

Purpose: Automatically extract structured data from natural conversations, assign confidence scores, and make data available for review and action.

Overview

Data collection transforms conversational interactions into structured records. As users chat naturally, the agent:

Monitors conversation for template-relevant information
Extracts data fields based on AI hints and field types
Assigns confidence scores to each field
Creates a data record with status NEW
Makes record available via list_collected_data tool

Key principle: Data extraction happens during the conversation, not after. Records are immediately available.

Flow Steps

1. Conversation Happens (joina.chat)

Where: User interacts with agent on joina.chat/{domain}/{agent-slug}

What user sees: Natural chat interface, no forms or structured input required

Behind the scenes: Agent has data collection configured:

json

{
  "entity_type_configs": [
    {
      "entity_type_id": 123,
      "collection_mode": "active",
      "custom_prompt": null
    }
  ]
}

Template fields (example Lead Capture):

json

{
  "fields": [
    {"name": "full_name", "type": "text", "required": true, "ai_hint": "Person's complete name"},
    {"name": "email", "type": "email", "required": true, "ai_hint": "Contact email"},
    {"name": "company", "type": "text", "required": false, "ai_hint": "Company name"},
    {"name": "interest", "type": "text", "required": false, "ai_hint": "Product interest"}
  ]
}

2. Data Extraction (Real-time)

When: After each user message in the conversation

Who: Gnosari extraction engine (runs alongside conversation LLM)

What happens:

User message analyzed against template fields
For each field:
- Check if information is present
- Extract value matching field type
- Validate against type rules (email format, number type, etc.)
- Calculate confidence score (0.0-1.0)
Merge with previously extracted data from this session
Track which fields are filled vs. missing

Example extraction:

User message:

"Hi, I'm Sarah Chen from Acme Corp. I'm interested in your enterprise plan."

Extracted data:

json

{
  "full_name": {
    "value": "Sarah Chen",
    "confidence": 0.98,
    "extracted_from": "message_1"
  },
  "company": {
    "value": "Acme Corp",
    "confidence": 0.95,
    "extracted_from": "message_1"
  },
  "interest": {
    "value": "enterprise plan",
    "confidence": 0.92,
    "extracted_from": "message_1"
  },
  "email": null  // Not yet provided
}

Confidence calculation factors:

Explicit statement: "My name is X" (high confidence)
Contextual inference: "I'm X" (medium-high confidence)
Ambiguous reference: "X mentioned this" (low confidence)
Field type validation: Valid email format (confidence boost)

3. Collection Mode Behavior

Collection mode (from entity_type_config) determines agent response:

passive

Behavior: Extract silently, never prompt for missing fields

Agent response (to message above):

"Great to meet you, Sarah! Let me tell you about our enterprise plan..."

No mention of missing email - just continues conversation naturally.

Data record created: With partial data (name, company, interest - no email)

opportunistic

Behavior: Extract + ask brief follow-up if critical fields missing

Agent response:

"Great to meet you, Sarah! I'd be happy to share details about our enterprise plan. What's the best email to send you information?"

Brief, natural request for the missing required field (email).

Data record created: After email is provided (or without it if conversation ends)

active

Behavior: Proactively guide conversation toward data collection

Agent response:

"Welcome! I'd love to help you learn about our enterprise plan. First, what's your name?"
User: "Sarah Chen"
"Great to meet you, Sarah! What company are you with?"
User: "Acme Corp"
"Perfect! What email should I use to send you details?"

Structured questioning to fill all required fields systematically.

Data record created: Once all required fields collected

guided

Behavior: Follow custom script defined in custom_prompt

custom_prompt example:

1. Greet warmly and ask what service they need
2. Collect their name and company
3. Ask for preferred contact method (email or phone)
4. Confirm details before proceeding

Agent follows the script regardless of what user volunteers.

Data record created: After script completion

4. Data Record Created

Trigger: Conversation ends OR sufficient data collected OR session timeout

What happens:

Call to Gnosari API: POST /api/v1/entities

Payload includes:

json

{
  "agent_id": 123,
  "entity_type_id": 456,
  "session_id": "sess_abc123",
  "attributes": {
    "full_name": "Sarah Chen",
    "email": "sarah@acme.com",
    "company": "Acme Corp",
    "interest": "enterprise plan"
  },
  "confidence": 0.94,
  "status": "NEW",
  "is_manual": false
}

Record saved to PostgreSQL
Status set to NEW

Automatic fields:

id: Auto-generated UUID
created_at: Timestamp
updated_at: Timestamp
confidence: Overall extraction confidence (average of field confidences)
status: Always "NEW" on creation
is_manual: false (AI extracted, not manually entered)

5. State Transitions

Data records move through states as they're processed:

NEW → REVIEWED → ACTIONED → ARCHIVED

NEW

Initial state: Just extracted from conversation

What it means: Needs human review for accuracy

Available via:

python

list_collected_data(status="NEW")

Typical workflow:

Appears in dashboard "New Leads" section
Human reviews for accuracy
Verifies email format, company name
Checks for spam/test entries

Next step: Mark as REVIEWED (via UI or API, not MCP)

REVIEWED

State transition: Human verified the data is accurate

What it means: Confirmed legitimate and actionable

Typical workflow:

Export to CRM (HubSpot, Salesforce)
Assign to sales rep
Send welcome email
Create support ticket

Available via:

python

list_collected_data(status="REVIEWED")

Next step: Mark as ACTIONED after taking action

ACTIONED

State transition: Action taken on the data

What it means: Email sent, ticket created, deal started, etc.

Typical workflow:

Monitor for response
Track to closure
Update in external systems
Follow-up scheduling

Available via:

python

list_collected_data(status="ACTIONED")

Next step: Mark as ARCHIVED when complete

ARCHIVED

State transition: Data processing complete

What it means: Closed/completed, kept for historical record

Typical workflow:

Remove from active views
Include in analytics reports
Retain for compliance/auditing

Available via:

python

list_collected_data(status="ARCHIVED", include_archived=True)

Note: Archived entries excluded from default queries unless include_archived=True

Querying Collected Data

list_collected_data Tool

Purpose: Filter and search collected data records

Common queries:

All NEW leads from last 7 days:

python

list_collected_data(
    template_name="Lead Capture",
    status="NEW",
    days=7
)

High-confidence enterprise leads:

python

list_collected_data(
    search="enterprise",
    min_confidence=0.9,
    days=30
)

Specific agent's data:

python

list_collected_data(
    agent_id=123,
    status="NEW",
    days=14
)

Search by email:

python

list_collected_data(
    search="sarah@acme.com"
)

Query Parameters

Parameter	Purpose	Example
`agent_id`	Filter by specific agent	123
`template_name`	Filter by template (e.g., "Lead Capture")	"Lead Capture"
`days`	Relative date filter (last N days)	7 (last week)
`status`	Filter by status	"NEW", "REVIEWED"
`search`	Search in field values	"acme.com"
`min_confidence`	Minimum AI confidence (0.0-1.0)	0.9 (very confident)
`skip`	Pagination offset	0
`limit`	Max results (default 10)	25

Response Format

Returns: List of data entries with summary

json

{
  "entries": [
    {
      "id": 789,
      "template_name": "Lead Capture",
      "template_icon": "i-heroicons-user",
      "agent_name": "Sales Assistant",
      "status": "NEW",
      "created_at": "2024-03-15T10:30:00Z",
      "attributes_summary": {
        "full_name": "Sarah Chen",
        "email": "sarah@acme.com",
        "company": "Acme Corp"
      },
      "confidence": 0.94
    }
  ],
  "total": 42,
  "skip": 0,
  "limit": 10
}

Note: Only first 3 attributes shown in summary. Use Gnosari UI for full details.

Statistics Dashboard

get_collection_stats Tool

Purpose: Aggregate statistics for performance monitoring

Query:

python

get_collection_stats(
    agent_id=123,  # Optional - per-agent stats
    days=30
)

Response:

json

{
  "total": 156,
  "status_counts": {
    "NEW": 42,
    "REVIEWED": 67,
    "ACTIONED": 38,
    "ARCHIVED": 9
  },
  "type_counts": {
    "Lead Capture": 98,
    "Support Ticket": 47,
    "Feedback": 11
  },
  "period_days": 30
}

Use cases:

Dashboard overview ("42 new leads this month")
Agent performance comparison
Template effectiveness analysis
Conversion funnel tracking (NEW → REVIEWED → ACTIONED)

Automatic Behaviors

Event	Automatic Action
Conversation starts	Extraction engine initialized with agent's templates
User sends message	Content analyzed against all template fields
Field detected	Value extracted, confidence calculated, stored in session
Conversation ends	Data record created with status NEW
Required field missing (active mode)	Agent asks for it naturally
Invalid field value (type validation)	Confidence score reduced, value flagged
Session timeout (30 min idle)	Partial data saved with lower confidence

Side Effects

Database writes:

Entity record created in entities table
Attributes stored as JSONB
Session reference saved
Agent and template associations created

No external API calls: Extraction happens locally. Export to CRM/external systems is manual (or via webhook integration, outside MCP scope).

Real-time availability: Data appears in list_collected_data() immediately after record creation.

Error Handling

Extraction Failures

Invalid field type:

Example: User says "email is john at example dot com" (not valid email format)
Action: Field marked with low confidence (< 0.5), value stored as-is
Human review required

Ambiguous data:

Example: Multiple names mentioned in conversation
Action: Most confident extraction used, confidence score reflects ambiguity
Human verification recommended

Missing required fields:

Example: User never provides email (required field)
Action: Record created with partial data, confidence score reduced
Follow-up needed

State Transition Errors

Note: State transitions (NEW → REVIEWED → ACTIONED → ARCHIVED) happen via Gnosari UI or API, not via MCP tools.

The MCP server provides read-only access to collected data. Status updates require direct API access or UI interaction.

Flow Diagram

Agent Creation Flow: How to configure data collection
Data Collection System: Templates, modes, fields
Data Collection Guide: Step-by-step setup

Data Collection Flow ​

Overview ​

Flow Steps ​

1. Conversation Happens (joina.chat) ​

2. Data Extraction (Real-time) ​

3. Collection Mode Behavior ​

passive ​

opportunistic ​

active ​

guided ​

4. Data Record Created ​

5. State Transitions ​

NEW ​

REVIEWED ​

ACTIONED ​

ARCHIVED ​

Querying Collected Data ​

list_collected_data Tool ​

Query Parameters ​

Response Format ​

Statistics Dashboard ​

get_collection_stats Tool ​

Automatic Behaviors ​

Side Effects ​

Error Handling ​

Extraction Failures ​

State Transition Errors ​

Flow Diagram ​

Related Flows ​

Data Collection Flow

Overview

Flow Steps

1. Conversation Happens (joina.chat)

2. Data Extraction (Real-time)

3. Collection Mode Behavior

passive

opportunistic

active

guided

4. Data Record Created

5. State Transitions

NEW

REVIEWED

ACTIONED

ARCHIVED

Querying Collected Data

list_collected_data Tool

Query Parameters

Response Format

Statistics Dashboard

get_collection_stats Tool

Automatic Behaviors

Side Effects

Error Handling

Extraction Failures

State Transition Errors

Flow Diagram

Related Flows