> ## Documentation Index > Fetch the complete documentation index at: https://docs.getbluejay.ai/llms.txt > Use this file to discover all available pages before exploring further. # Quickstart > Go from zero to a full Bluejay evaluation workflow — via the UI or the API Get from account creation to running Simulations, evaluating production calls, and optimizing your Agent quality in under 15 minutes. ## Prerequisites * A Bluejay account ([book a demo](https://cal.com/rohanv/15min) if you don't have one) * For API steps: an API key from the [API Keys page](https://app.getbluejay.ai/api-keys) ## Basic Everything you need to run your first Simulation and connect production monitoring. An Agent represents the conversational AI system you want to test and monitor. Navigate to the **Agents** page in the Bluejay dashboard and click **Add Agent**. Fill in the Agent name, system prompt, knowledge base, and goals. Learn more in the [Agents overview](/key-concepts/agents/overview) or jump to the [Add Agent API reference](/api-reference/endpoint/add-agent). A Simulation groups Digital Humans together so you can run them in parallel against your Agent — think of it as a test suite. Open your Agent and click **Create Simulation**. Give it a name and optionally configure settings like max call duration or sequential calling. Learn more in the [Simulations overview](/test/simulations/overview) or explore [Simulation Types](/test/simulations/types). Digital Humans are the synthetic callers that interact with your Agent during Simulations. Inside your Simulation, click **Generate Digital Humans**. Choose from goal adherence, red teaming, load testing, or Customer Persona modes. You can also manually create individual Digital Humans for specific scenarios. Learn more in the [Digital Humans overview](/key-concepts/digital-humans/overview). Click the **Run** button on your Simulation. Bluejay's Digital Humans will call your Agent, and results are evaluated automatically. You can watch conversations happen in real time and review results as they complete. Learn more about [Simulation Runs](/test/simulations/simulation-runs). Custom Metrics define the exact quality signals you care about — compliance checks, empathy scoring, task completion, or anything specific to your domain. Navigate to the **Custom Metrics** section of your Agent, click **Create Metric**, and define the name, description, response type, and scoring guidance. Learn more in the [Custom Metrics overview](/key-concepts/custom-metrics/overview). Observability evaluates your production conversations against the same Custom Metrics you use in Simulations. Go to the **Observability** tab for your Agent. You can connect a native integration ([Retell](/integrations/retell), [Vapi](/integrations/vapi), [Bland](/integrations/bland), [ElevenLabs](/integrations/eleven-labs)) or configure webhook-based ingestion directly from the Dashboard. Learn more in the [Observability overview](/monitor/observability/overview) or follow the [API integration tutorial](/monitor/observability/api-integration-tutorial). *** ## Advanced Level up with Alerts, Dashboards, optimization, automated testing, and Prompt Versioning. Alerts notify your team when a Custom Metric crosses a threshold so issues get caught before they compound. Navigate to the **Alerts** configuration for your Agent. Set the Custom Metric, threshold, and delivery channel ([Slack](/integrations/slack), email). Learn more in the [Alerts overview](/key-concepts/alerts/overview). Dashboards aggregate Simulation and production data into a single view. Open the **Dashboards** section to see at-a-glance health scores, trend sparklines, and Alert badges across all your Agents. Customize views to focus on the Custom Metrics that matter most. Learn more in the [Dashboards overview](/key-concepts/dashboards/overview). Metrics Lab lets you prototype, test, and refine Custom Metrics before deploying them. Open **Metrics Lab** from the sidebar. Import or draft Custom Metrics, annotate sample conversations, and run side-by-side comparisons to find the most reliable scoring approach. Promote validated Custom Metrics to production with one click. Learn more in the [Metrics Lab overview](/key-concepts/metrics-lab/overview). For ElevenLabs, Vapi, Retell, and Google Dialogflow CX agents, the **Workflow** tab in Agent Settings renders your provider's multi-agent workflow as an editable graph. Sync it from the provider, edit nodes and edges in Bluejay, push changes back, and generate digital humans that cover every path through the flow. Learn more in the [Workflows overview](/key-concepts/agents/workflow). Prompt Versioning tracks changes to your Agent's system prompt over time. Open the **Prompts** section of your Agent. Create a new version with an updated prompt, add a commit message, and optionally tag it with labels like `production` or `staging`. Compare performance across Simulation Runs and roll back when needed. See the [Create Prompt Version API reference](/api-reference/endpoint/create-prompt-version). ## Basic Everything you need to run your first Simulation and connect production monitoring. ```python theme={null} import requests API_KEY = "your-api-key" BASE_URL = "https://api.getbluejay.ai/v1" ``` An Agent represents the conversational AI system you want to test and monitor. You'll attach Simulations, Custom Metrics, and Observability pipelines to it. ```python theme={null} agent = requests.post( f"{BASE_URL}/add-agent", headers={"X-API-Key": API_KEY}, json={ "name": "Support Agent", "system_prompt": "You are a helpful customer support agent for Acme Corp.", "knowledge_base": "Acme Corp FAQ and product documentation.", "goals": [ "Resolve customer issues accurately", "Maintain a professional and empathetic tone", "Escalate when unable to resolve" ], "type": "INBOUND", "mode": "VOICE" } ).json() agent_id = agent["agent_id"] print(f"Agent created: {agent_id}") ``` Save the `agent_id` from the response — you'll use it in every subsequent step. Learn more in the [Agents overview](/key-concepts/agents/overview). See the full [Add Agent API reference](/api-reference/endpoint/add-agent) for all available fields like `connection_type`, `phone_number`, and `external_agent_id`. A Simulation groups Digital Humans together so you can run them in parallel against your Agent — think of it as a test suite. ```python theme={null} simulation = requests.post( f"{BASE_URL}/create-simulation", headers={"X-API-Key": API_KEY}, json={ "agent_id": str(agent_id), "name": "Support Agent — Regression Suite", "max_call_duration": 5, "max_call_duration_units": "minutes" } ).json() simulation_id = simulation["simulation_id"] print(f"Simulation created: {simulation_id}") ``` Learn more in the [Simulations overview](/test/simulations/overview) and explore [Simulation Types](/test/simulations/types). See the full [Create Simulation API reference](/api-reference/endpoint/create-simulation) for options like `sequential_calling`, `max_concurrent`, and `selected_custom_metrics`. Digital Humans are the synthetic callers that interact with your Agent during Simulations. You can generate them automatically from your Agent's goals or create them manually for specific scenarios. **Generate automatically from goals:** ```python theme={null} digital_humans = requests.post( f"{BASE_URL}/generate-digital-humans", headers={"X-API-Key": API_KEY}, json={ "agent_id": str(agent_id), "simulation_id": str(simulation_id), "goal_adherence": [ {"goal": "Resolve billing dispute", "num_calls": 3}, {"goal": "Process a return request", "num_calls": 2} ], "red_teaming": 2 } ).json() print(f"Generated {len(digital_humans['digital_humans'])} Digital Humans") ``` **Or create one manually:** ```python theme={null} dh = requests.post( f"{BASE_URL}/create-digital-human", headers={"X-API-Key": API_KEY}, json={ "simulation_id": str(simulation_id), "name": "Frustrated Customer", "scenario": "Customer received the wrong item and wants a full refund.", "success_criteria": "Agent processes the return and confirms refund timeline." } ).json() ``` Learn more in the [Digital Humans overview](/key-concepts/digital-humans/overview). See [Generate Digital Humans](/api-reference/endpoint/generate-digital-humans) and [Create Digital Human](/api-reference/endpoint/create-digital-human) API references for all options. Queue a Simulation Run to start the conversations. Bluejay's Digital Humans will call your Agent, and results are evaluated automatically. ```python theme={null} run = requests.post( f"{BASE_URL}/queue-simulation-run", headers={"X-API-Key": API_KEY}, json={ "simulation_id": str(simulation_id) } ).json() print(f"Simulation run queued: {run['simulation_run_id']}") print(f"Result IDs: {run['simulation_result_ids']}") ``` Use the [Get Simulation Runs](/api-reference/endpoint/get-simulation-runs) endpoint to poll for completion, or set up a [webhook](/api-reference/webhook/events-webhook) to get notified when runs finish. Learn more about [Simulation Runs](/test/simulations/simulation-runs). See the full [Queue Simulation Run API reference](/api-reference/endpoint/queue-simulation-run) for options like `phone_number`, `digital_human_ids`, and `prompt_id`. Custom Metrics define the exact quality signals you care about — compliance checks, empathy scoring, task completion, or anything specific to your domain. ```python theme={null} metric = requests.post( f"{BASE_URL}/create-custom-metric", headers={"X-API-Key": API_KEY}, json={ "agent_ids": [agent_id], "name": "Identity Verification", "description": "Did the agent verify the customer's identity before sharing account details?", "response_type": "pass_fail", "scoring_guidance": "Pass if the agent asks for at least two identifying pieces of information before disclosing any account-specific data. Fail otherwise." } ).json() print(f"Metric created: {metric['id']}") ``` Learn more in the [Custom Metrics overview](/key-concepts/custom-metrics/overview). See the [Create Custom Metric API reference](/api-reference/endpoint/create-custom-metric) for all response types (`pass_fail`, `yes_no`, `qualitative`, `quantitative`, `json`, `enum`). Observability evaluates your production conversations against the same Custom Metrics you use in Simulations. Send calls to Bluejay via the API, webhooks, or native integrations. ```python theme={null} evaluation = requests.post( f"{BASE_URL}/evaluate", headers={"X-API-Key": API_KEY}, json={ "agent_id": str(agent_id), "transcript": [ {"speaker": "CUSTOMER", "utterance": "Hi, I need help with my recent order."}, {"speaker": "AGENT", "utterance": "Of course! Could you provide your order number?"}, {"speaker": "CUSTOMER", "utterance": "It's 12345."}, {"speaker": "AGENT", "utterance": "I see order 12345. It looks like it shipped yesterday. You should receive it by Friday."} ], "participants": [ {"role": "AGENT", "spoke_first": False, "name": "Support Agent"}, {"role": "CUSTOMER", "spoke_first": True, "name": "Customer"} ], "call_direction": "INBOUND" } ).json() print(f"Evaluation queued: {evaluation['call_id']}") ``` You can also send audio recordings via `recording_url` or `recording_base64` instead of transcripts. Learn more in the [Observability overview](/monitor/observability/overview) or follow the [API integration tutorial](/monitor/observability/api-integration-tutorial). See the full [Evaluate API reference](/api-reference/endpoint/evaluate). | Ingestion Method | Best For | | ---------------------------------------------------- | -------------------------------- | | [Evaluate API](/api-reference/endpoint/evaluate) | Direct backend integration | | [Webhook ingestion](/key-concepts/webhooks/overview) | Platforms with outbound webhooks | | [Retell](/integrations/retell) | Retell-powered agents | | [Vapi](/integrations/vapi) | Vapi-powered agents | | [Bland](/integrations/bland) | Bland-powered agents | | [ElevenLabs](/integrations/eleven-labs) | ElevenLabs Conversational AI | *** ## Advanced Level up with Alerts, Dashboards, optimization, automated testing, and Prompt Versioning. Alerts notify your team when a Custom Metric crosses a threshold so issues get caught before they compound. Alerts are configured through the Bluejay Dashboard — set threshold-based triggers on any Custom Metric and route notifications to Slack channels or email. Connect Bluejay to Slack via the [Slack integration](/integrations/slack) to receive alert notifications in your team channels. Learn more in the [Alerts overview](/key-concepts/alerts/overview). Dashboards aggregate Simulation and production data into a single view — health scores, trend sparklines, and Alert badges for every Agent. Dashboards are configured through the Bluejay UI and pull from both Simulation results and Observability evaluations, so once you've completed the basic steps, your Dashboards will populate automatically. Learn more in the [Dashboards overview](/key-concepts/dashboards/overview). Metrics Lab lets you prototype, test, and refine Custom Metrics before deploying them. It's an interactive UI experience — import or draft Custom Metrics, annotate sample conversations, run side-by-side comparisons, and promote validated Custom Metrics to production. Once refined, you can manage Custom Metrics programmatically via the [Custom Metrics API](/api-reference/endpoint/create-custom-metric). Learn more in the [Metrics Lab overview](/key-concepts/metrics-lab/overview). Workflows let you chain Simulation Runs, evaluations, and notifications into automated pipelines. Configure pipelines in the Bluejay dashboard, then attach a cron cadence with the [Create Schedule API](/api-reference/endpoint/create-schedule) for nightly regression suites or weekly quality checks. For **conversation-path** scenarios (React Flow graphs used with simulations and digital humans), see the [Scenario Builder overview](/key-concepts/scenario-builder/overview), the [Scenario Builder cookbook](/cookbook/scenario-builder), and [Create scenario](/api-reference/endpoint/create-scenario) (**POST** `/v1/scenario`). Prompt Versioning tracks changes to your Agent's system prompt over time. Create labeled versions, compare performance across Simulation Runs, and roll back when needed. ```python theme={null} prompt_version = requests.post( f"{BASE_URL}/agents/{agent_id}/prompts/versions", headers={"X-API-Key": API_KEY}, json={ "prompt_text": "You are a helpful customer support agent for Acme Corp. Always verify the customer's identity before sharing account details.", "commit_message": "Added identity verification requirement", "labels": ["staging"] } ).json() print(f"Prompt version {prompt_version['version']} created (id: {prompt_version['id']})") ``` Pass a `prompt_id` when [queuing a Simulation Run](/api-reference/endpoint/queue-simulation-run) to test a specific prompt version against your existing Digital Humans. See [Create Prompt Version](/api-reference/endpoint/create-prompt-version) for the full reference. *** ## Next Steps Deep dive into Agents, Digital Humans, Custom Metrics, and more. Explore goal adherence, red teaming, load testing, and replay strategies. Step-by-step guide to connecting your production pipeline. Full endpoint documentation for every Bluejay API. Ask questions, share what you're building, and get help from the Bluejay team and other builders in our Slack.