Per-Embed Metrics Dashboard Guide
AI widgets live in many places: docs, pricing, partner pages, in-product overlays. Without per-embed metrics you cannot prove ROI or find accuracy hotspots. This guide shows how to instrument impressions, opens, chats, messages, containment, and feedback for every placement.
1. Establish embed identity
- Generate a deterministic embed_id from tenant, domain, and placement.
- Store metadata: placement_type (hero, sticky, footer), language, stage.
- Persist embed configuration (prompt version, theme, relevance threshold) so you can correlate changes with performance.
2. Capture the funnel
| Event | Why it matters | Notes |
|---|---|---|
| impression | Measures reach | Fire once per page load when widget script executes. |
| open | Shows intent | Trigger when user expands or focuses the assistant. |
| chat_started | Gauges real engagement | Record when the first user message is sent. |
| messages_total | Tracks effort | Include user and assistant counts. |
Pair these events with page_path, referrer, resolved language, and tenant plan to segment behavior.
3. Track containment and fallbacks
- Containment = resolved_without_handoff / chats_started.
- Log fallback_reason (low_score, no_context, provider_error, timeout). CrawlBot captures this automatically for each turn.
- Monitor retrieval_score_p95 to understand if embeddings or thresholds need tuning.
- Alert when containment drops below target or fallbacks spike for a specific embed_id.
4. Blend qualitative feedback
- Capture thumbs, flag reason (incorrect, outdated, unclear, off-scope), and optional free text.
- Join transcripts with crawl_version and prompt_version so ops can see whether content or prompting caused the issue.
- Pipe negative feedback into Google Chat or Pager channel per AGENTS runbook.
5. Visualize in a dashboard
Suggested tiles:
- Impressions → opens → chats waterfall per embed.
- Containment vs fallback reasons scatter plot.
- Top unanswered or flagged queries.
- Language distribution vs path for multi-lingual embeds.
- Trend lines for crawl_version and prompt_version changes.
Use alerts and annotations whenever a prompt or crawl change occurs so stakeholders know why metrics moved.
6. Operational actions
- Low impressions? Promote the CTA or reposition the widget.
- Low opens vs impressions? Test copy, button color, or language auto detect.
- Low containment? Check crawl coverage, threshold tuning, or escalate to live chat for that page.
- High fallback_reason = provider_error? Verify LLM gateway timeouts and failover.
Per-embed metrics turn anecdotes into accountable workflows. CrawlBot ships these fields by default, but even DIY stacks should mimic them to keep AI quality measurable.***