KB Content: Work in progress
The OpenAI GPT Realtime implementation is currently tested in active development. This document will be expanded with further information. Until this note is removed, please do not implement the instructions below in a productive environment.
INC Work in Progress
This area of the Knowledge Base is still work in progress (WIP). Expect changes to the layout of this page as improvements are made.
INC Use Case Preview Feature
PREVIEW - This Use Case showcases features that may not yet be available to all customers. Note that functionality, scope, and design of preview features may change significantly in future. As we define the last technical details, requirements and steps may be subject to change.
In this Use Case, we're going to address the following topics in order to set up Virtual User in Nimbus for customer intent analysis. Topics covered are as follows:
- How to use Microsoft Copilot Studio to set up your first AI Agent acting as a Virtual User.
- How to configure Nimbus in order to use the Virtual User in your workflows.
- How to set up the workflow and make service calls for further adjustments.
INC Icon Legend Accordion
Show Icon Legend 💡 = A hint to signal learnings, improvements or useful informati...
Show Icon Legend
| 💡 = A hint to signal learnings, improvements or useful information in context. | 🔍 = Info points out essential notes or related page in context. |
| ☝ = Notifies you about fallacies and tricky parts that help avoid problems. | 🤔 = Asks and answers common questions and troubleshooting points. |
| ❌ = Warns you of actions with irreversible / data-destructive consequence. | ✅ = Intructs you to perform a certain (prerequired) action to complete a related step. |
Preconditions
INC AI and Virtual User Preconditions
PRECONDITIONS: AI and Virtual Users
Admin Consent: AI Features
AI use will require your Tenant data (e.g. caller information) to be processed by external services
✅ Nimbus Tenant Admin rights are required to grant consent to enable the use of AI features. This is done in the Data Privacy Tenant Settings.
💡Both Tenant Admins and OU Admins are able to create various AI Configuration items in the Nimbus Admin Portal. This also includes:
- Access to Service Distribution Settings, e.g. to test workflows with your configured Virtual Users.
- Steer feature visibility via Companion Service Settings (e.g. when activities are supposed to happen unobtrusive in the background).
Virtual User Licensing
✅Nimbus License Management:
-
Contact Center Enterprise Routing service licenses are required to use AI-driven Virtual User functionality.
⮑With the license applied, corresponding Service Features (e.g. workflow activities, configuration tabs) are enabled. -
Virtual User license requirement: each Virtual User requires a separate license. This license can either be applied directly in the Virtual User configuration steps below or - in bulk - via Admin > Licensing view.
⮑The license enables usage of the “Add Virtual User” Workflow Activity. For additional Virtual User licenses on your service, get in touch with your Customer Success representative.
Virtual User Bot Configuration
✅Nimbus Configuration dependencies:
-
ALL: Virtual Users require Bots to be configured.
☝Your bot choice also determines the underlying AI (Large Language Model) in which the Nimbus Virtual User integrates into. As each LLM has different advantages, the choice should be made based on the intended Use Case.Show Bot (LLM) comparison…
INC Language model comparison matrix
Type Primary Use Case Limitations Integration Effort Nimbus AI Services – Audio Intent Analyzer Low latency, conversational AI, real-time audio stream. The bot is communicating directly with the user. Used to detect the intent of a caller and to return the selection done for intelligent routing and IVR replacement. - Specialized for providing intent analysis.
- Uses Nimbus-native AI Model with fixed-purpose instruction set.
- Limited parameter sets for data storage.
Low – Easy to use, configuration effort stays within Nimbus. Parametrization is done in workflows as part of the “Virtual User” activity.
🔎 See: Use Case - Setting up a Nimbus Virtual User for Intent Analysis
Nimbus AI Services - AI Workflow Low latency, conversational AI, real-time audio stream. Used as speech‑in / speech‑out conversational AI with interactions specific to data requests from external systems (e.g. banking or insurance services, ticketing platforms and CRM systems). - Flexible, but depends on external system uptime and responsiveness for seamless customer interaction.
- Requires purpose-driven prompt knowledge and context for all parameters to handle conversation effectively.
Medium - Requires knowledge to handle Web Requests and MCP servers.
Customers can use Nimbus-native AI models, but must ensure that the web requests and required data is sufficiently covered with the AI prompts.
🔎 See: Use Case - Integrating Nimbus Virtual User with external systems
M365 Copilot – Direct Line 3.0 Enterprise voice interactions with Microsoft 365 Copilot, enabling conversation storage, transcription and intent detection of spoken conversations. Grounded in O365 work data. - For generalized use (e.g. data lookup, topics handling) and customer intent detection.
- Higher latency due to Speech-to-Text and Text-to-Speech generation.
- Subject to Microsoft licensing and service capacity limits.
Medium – Requires Microsoft 365 and Copilot ecosystem knowledge. Requires configuration within Nimbus, plus integration with Azure Copilot Studio API to handle topics.
🔎 See: Use Case - Setting up a Nimbus Virtual User using CopilotAzure OpenAI – Audio GPT Realtime1 Low latency, conversational AI, real-time audio stream. Used as speech‑in / speech‑out conversational AI. Allows admins “bring their own” LLMs and data instances to implement intent analysis. - Flexible, but requires customers to bring their own bot configuration.
- Needs custom client and backend integration, with regional availability determined by Microsoft1.
- Limited to intent analysis purposes for now, a new bot type will be introduced for wider range of use cases
High – Requires configuration within Nimbus also requires customers to bring their own LLM and 3rd party integrations.
🔎 See: Use Case - Setting up a Nimbus Virtual User using OpenAI GPT Realtime
Comparison: Types of Bots for Nimbus Virtual Users Notes
1 Due to Microsoft Azure AI Foundry GPT-Realtime availability in limited regions, data processed by Nimbus Virtual User GPT-Realtime integration will temporarily leave your regional boundary:
- DE01, DE02, CH01, CH02, UK01, EU01, AU01 computation will be primarily performed in Sweden Central region.
- US01 cluster computation will be primarily performed in the US East region.
-
OPTIONAL: Specific models (e.g. Microsoft Copilot) may also need further Nimbus configuration items:
- Bot Response Templates, to map bot answers back into Nimbus workflow parameters.
- Speech Recognizers (Transcribers) to convert the Customer voice into text for the bot to process.
🔎For more details and specific configuration steps, refer to our AI Use Cases category and the specific pages linked above.
Azure Preconditions
Microsoft Services: In order to use AI-driven Bots in Nimbus, you will need access to Microsoft Cognitive Services. Azure Bot usage causes additional costs outside of your Nimbus subscription. Please note the licensing remarks below.
INC Azure Billing Cognitive Services
Disclaimer: Support for Azure Billing & Cognitive Services
☝ The usage of Microsoft Azure AI Services for Speech1 and AI2 will cause additional costs outside your Nimbus subscription, which are exclusively determined by Microsoft.
Please note:
- Nimbus Support does not cover pricing discussions or make recommendations on Microsoft Azure Services. → Please inform yourself via the references below on pricing models.
- Nimbus Support can not provide any technical help, insights or solutions on Azure Cognitive Service related technical problems.
1 Azure Speech Services: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/speech-services/
2 Generative AI Services (Copilot Studio):
Use Case Overview
Details will be covered below in this Use Case. As a quick tl;dr, here is an overview of items you will need:
| What? | Where? | Why? |
|---|---|---|
| ✅ Copilot Bot Setup1 | https://copilotstudio.microsoft.com/ | To define the bot role, behavior, topics, triggers and to get the API authentication details |
| ✅Bot Configuration | Nimbus Admin > Configuration | To get the Bot API endpoint and Authentication details into Nimbus |
| ✅Bot Response Templates | Nimbus Admin > Configuration | To map the bot response JSON to Nimbus parameters for further processing within workflows. |
| ✅Virtual Users | Nimbus Admin > Configuration | To define the initial bot instructions as well as tying all other configuration items together. |
| ✅ “Add Virtual User Activity” (in Workflow) | Workflows > Conversation Handling Activities | To invite the Bot to the conversation and route the outcomes accordingly. |
1 Note that behaviors, UI designs, and configuration details are subject to frequent changes by Microsoft and may mismatch from our use case descriptions. Feel free to contact our support to request an update to our use case documentation.
Known Limitations
INC AI and Virtual User Limitations
General note on AI-driven interactions
AI driven replies are not deterministic and depend highly on what the Customer is saying. For example, if the Customer is saying “I have trouble with my internet” it is not necessarily given that the Bot will associate “Router, Modem” as your workflow routing exit, unless specifically handled in your Virtual User integration. In this specific example, AI instructions should also cover alternative wordings like “Router, Internet”, to be handled in topics accordingly.
🔎Refer to our Best Practices - Virtual Users in Nimbus for some considerations and risks when using Virtual Users instead of Human interaction.
General Virtual User limitations
The current Nimbus implementation with AI-driven bots underlies the following limitations:
- Supported Modalities: Virtual Users (Bots) are currently available for Audio/Video modality tasks.
- Virtual User Reporting: Sessions involving Virtual Users are not reflected as dedicated User Session. Virtual User session reporting is planned for a later point this year.
- Outbound Call via Workflow: Virtual Users are supported in Outbound Call with Workflows. While all Virtual Users are planned to be fully supported by Outbound Workflows, a prior “Announcement” workflow element is required before the “Add Virtual User” to give the AI sufficient spin-up time. Nimbus preview program users can already test their scenarios. As testing and development on Bots compatibility is still ongoing, not all features may work as intended yet.
Microsoft Copilot Limitations
- Expect processing delays: Processing AI answers takes a few seconds for voice-to-text-transcription, followed by AI processing and a transcription back into a voiced response. Luware is trying to minimize the delay on Nimbus call infrastructure, but the dependency on external APIs will always incur a delay. The Customer will hear silence during this processing and no audio feedback / silence detection.
-
Ensure you have no ambiguity in your topics. For instance, the word “
Service” may be too generic if you want to transfer to different services. Rather usehealthcare|medical|emergencyas identifiers or use more complex Regular Expressions to identify replies.
Nimbus AI Services - Audio Intent Analyzer
- The Nimbus Audio Intent analyzer Bot runs on pre-defined prompts. Any usage outside of the descriptions specified under Use Case - Setting up a Nimbus Virtual User for Intent Analysis is not supported. → Check Use Case - Integrating Nimbus Virtual User with external systems to see if it covers your criteria. Luware is working on adding new bot types for additional use case coverage.
🔎By design (not a limitation or reportable issue):
-
Fallback handling: If a workflow uses exits with multiple parameters, e.g. to collect
customer nameandinvoice number, and only one parameter can be retrieved, the fallback exit is taken.
Nimbus AI Services - AI Workflow
☝First Web Requests implementation - planned to be improved:
- The “Description” field in the Web Requests itself is used to explain to the AI what the web request does.
- The Web Request itself can be identified via name, using the "Flow Description" field located in the Virtual Users > Extensions > Web Request section.
🔎By design (not a limitation or reportable issue):
-
Fallback handling: If a workflow uses exits with multiple parameters, e.g. to collect
customer nameandinvoice number, and only one parameter can be retrieved, the fallback exit is taken. - Failed Web Request / MCP fallback: If either a Web Request fails or a connection to an MCP server fails to connect, the with the Fallback exit accordingly. Any (optionally user-specified) “Fallback” parameters will be updated with the response generated by the AI.
Azure Open AI - Open GPT Realtime limitations
☝ This feature is not yet ready for productive use and only enabled for selected customers. We will make further adjustments and update Use Case - Setting up a Nimbus Virtual User using OpenAI GPT Realtime accordingly in an upcoming Nimbus release.