spee

In this Use Case, we're building a Copilot Agent which classifies Customer input into dimensions of data you define. This could be intent or sentiment, or categories. It is up to you! 

This generic capability of classifying messages can be used in different scenarios. One of such scenarios is to replace traditional, multi-step IVR by a single, natural language-driven interaction. A traditional IVR could have many steps and could look like:

1. “Welcome, choose your language.”
   → (Customer selects English)
2. “Press 1 if you are a customer, press 2 if you are not yet a customer of BEST Insurance.”
   → (Customer presses 2) 
   ⮑ Nimbus recognizes the Caller as an existing client “Sylvia”, retrieves information about the client.
3. “Welcome, Sylvia! Press 1... press 2.”
   → (Silvia presses 2)
4. “An agent will be available shortly to handle your accident report.” 
   → (Silvia waits for the Agent to connect.)
5. Sylvia is connected to the Agent 
6. Sylvia finally can express her intent.

By replacing this (rather long) IVR interaction with Copilot, it can look like this:

1. “Welcome, what can I do for you?” 
   → (Customer says:) “Hello, I was in a car accident involved and would like to know if the damage is covered by my insurance.” 
   ⮑Use Case - The Copilot Agent processes the input and extracts:
   1. Language: English (EN)
   2. Customer Type: Existing
   3. Topic: Insurance
   4. Subtopic: Coverage
   5. Sentiment: Neutral
2. Based on this understanding, the Copilot routes the Customer directly to the appropriate service or provides an immediate, context-aware response – eliminating the need for multiple IVR steps. 
3. Sylvia is connected to the Agent. 
4. The Agent is already aware of the topic Sylvia wants to speak about.

As you can see, the Copilot approach is much more time-efficient and Customer-friendly while minimizing errors during the Interaction.

Overview of the Nimbus Workflow and Copilot Activities

 

 

Designing the Copilot

Let's start by applying the best practices to our creation process by defining a scenario:

Imagine visitors of a nature reserve. At the entrance a welcome guide listens to each one of them, understanding multiple languages, why they are here and what they want to see. The guide will then call a ranger to come to the entrance and pick up the visitor for a tour. Once the ranger arrives, the guide hands out a datasheet of the visitors conversation and interests so that the ranger can personalize the tour.

Defining the Bot's role

Acting as the Guide, the Copilot bot will gather all the necessary information and perform a classification of all the relevant information. 

Defining the Tasks

• Greet
  • Explain well to arriving visitors why you are here and what you need to know to do your job.
• Listen & Understand
  • Classify the visitor's messages into urgency, topic, sentiment, language.
• Handover …
  • …when language and topic has been detected (it's sufficient, urgency and sentiment are a plus)
  • …when the visitor requests for anything else (do not call the ranger, but point to the park's administration office ==> Agent handover)
  • …as a fallback, in case of too many retries (instruct the ranger to do the default tour ==> fallback language and topic)

Access, checkpoints and thinking

The guide doesn't need access to the park, but he needs to know how to call the rangers and handover the information. The guide does not need to recognize the visitors and doesn't need a long-term memory. There is no need for supervisor approval for giving instructions to the rangers. 

The message classification task requires analytical thinking, we'll be using non-generative AI. 

Building the Copilot

Description	Screenshot
Head to Copilot Studio Creating a new (blank) Agent in Copilot Studio
Go to the system topics.  Leave the “On Error” topic on Toggle everything else off.
Change the “On Error” topic You want to output a Message with the System.Error.Message in the text.
Create a custom topic “On message received”. Here is an overview of the topic flow. We will get into the details in the following steps.
Start with the trigger “Message is received”. 💡This topic runs for every Customer message.
💡The first Customer message will contain the Nimbus TaskID. By handing it over to Copilot using we can use it in the “Initial Message” of the Virtual User we can hand over the TaskId in a variable so that we can update the task when needed. Within Copilot Studio: Add a Condition Fill in the global variable Global.TaskId and check if it is Blank.
If yes, Set its value to the Activity.Text. 💡Activity.Text holds the complete Customer message string. Follow up with your welcome message in the same branch.
💡In the “other” branch, we know that we receive the second Customer message. It will be the reaction to the welcome message. In our case “Hi, how can I help you?”   For that add a new step and select Add a tool > New Agent flow. After clicking that it will directly take you to the designer.    Please go to the Power Automate Flow part below, where we explain how to create the flow.  After that is done you can come back here.    The picture here shows the finished state! The AI model returns a structured JSON output which we need to parse and map to our variables, then respond to Copilot. Language (string) Urgency (string) Sentiment (string) Topic (string)
Now that we classified the message and collected the values in our variables, we can use the Nimbus Connector to Update the Nimbus task.  Add Tool > Connector  Choose Luware Nimbus Update Task → Also See: Flow Actions > Update Task.
Map the Nimbus TaskId to the Global.TaskId from earlier…
…and the Custom Context Parameters (System Fields and Parameters). Use a a powerFX formula to create the required Table object: Table( { name: "Sentiment", value: Global.sentiment }, { name: "Language", value: Global.Language }, { name: "Topic", value: Global.Topic }, { name: "Urgency", value: Global.Urgency }, { name: "Message", value: System.Activity.Text } )
Now, add a message with the Exit criteria response, so the Nimbus workflow activity will exit once the message is classified. Deploy your Copilot Agent.

Power Automate Flow

Flow overview

 

 

Trigger Event for the PA flow will be the “When a agent calls the flow” When you come from the topic this will be added automatically.  Here you add an input and select text.
Next we initialize a variable and name it “System Message” Below you will find the value of the message.  In short we will tell the AI what to do and what output we want and in what format.
Show System Message…. Classify customer messages into ISO 639-1 language code, urgency, sentiment, and topic. Output the result strictly in JSON format according to the provided schema. # Criteria for Classification - **Language**: Classify the language of the input message using the ISO 639-1 language code (e.g., "en" for English, "es" for Spanish). - **Urgency**: Assess how urgent the message is. Options include "low", "medium", "high". - **Sentiment**: Analyze the sentiment of the message. Options include "positive", "neutral", "negative". - **Topic**: Identify the main topic or category of the message (e.g., "billing", "technical support", "general inquiry"). Be concise but descriptive. # Output Format Respond in JSON format that adheres to the following schema: { "language": "string (ISO 639-1 code)", "urgency": "string ('low', 'medium', 'high')", "sentiment": "string ('positive', 'neutral', 'negative')", "topic": "string" } # Additional Notes - When processing input, replace any dynamic placeholders with the actual content if provided (e.g., if the placeholder represents customer message text). - Ensure the text is fully interpreted, and the output adheres to the specified format. # Example **Input:** Classify the following message: I have a problem with my login. Respond in JSON: { "language": "...", "urgency": "...", "sentiment": "...", "topic": "..." } **Output Example:** { "language": "en", "urgency": "high", "sentiment": "negative", "topic": "technical support" }
Subsequently, both the caller's input and the system message are sent to the AI through an HTTP request.  💡This process requires a designated endpoint capable of receiving such data. For our GPT model implementation, we utilise Azure Foundry. Below is the use case outlining the procedure for creating a model: Luware Nimbus - Use Case - Creating an AI Model over Azure AI Foundry with Open AI Once your model is deployed, you will receive a URL for sending requests. Body:  { "messages": [ { "role": "system", "content": "@{variables('System Message')}" }, { "role": "user", "content": "@{triggerBody()?['text']}" } ], "temperature": 0.2, "max_tokens": 200 }
We will then add an Parse JSON step, taking the body output from our HTTP request and put it in the right format.    Schema: { "type": "object", "properties": { "language": { "type": "string" }, "urgency": { "type": "string" }, "sentiment": { "type": "string" }, "topic": { "type": "string" } } }
The last step will be to map the JSON outputs to the outputs needed to send as response back to Nimbus.

Adding the Copilot to Nimbus Configuration

Add the Speech Recognizer

1. Setup your speech services in Azure: 
   1. 💡You can follow Use Case - Setting Up a Speech Recognizer in Azure Portal and Nimbus instructions.
   2. 🧠In the overview of your speech services configuration you find the secret keys, copy one of them.
   3. 🧠Also note the region of your speech service, as it determines where the API call will be made.
2. In Nimbus, go to Admin > Configuration > Speech Recognizers
3. Create a new Speech Recognizer entry. 
4. Define the Name, Organization Unit as needed
5. If not already selected, choose Type “Azure AI Services”
6. 🧠Paste the API Key and specify the region same as defined previously.

Add the Bot

1. In Copilot Studio, go to Settings > Security > Web channel security 
2. 🧠 Copy the Secret 1 key for later.
   
3. In Nimbus, go to Admin > Configuration > Bots. Add a new bot with the following configuration: 
   1. Endpoint: https://directline.botframework.com/v3/directline/conversations
   2. API key: 🧠 Your copied secret key

Disable Authentication

1. In Copilot Studio, go to Settings > Security > Authentication
2. Set the Authentication to “No authentication”
   ☝Attention: If this authentication is not deactivated the Virtual User will not work!

Create a Response Template

1. In Nimbus, go to Admin > Configuration > Bot Response Templates 
2. Add a new default response template for Copilot.
   

Create the Virtual User

1. In Nimbus, go to Admin > Configuration > Virtual Users.
2. Add a new Virtual User with a Description and choose your bot accordingly.
3. On the Initial Message to Bot the most important part is the System Parameter CallId. It corresponds to the Nimbus TaskID.
   💡The CallId is used to start the Copilot “On message received” Trigger mentioned above, which will engage the Bot with a Customer. The next reply by the Customer is then handled exclusively Copilot itself.

Using the Copilot in Nimbus

The final step is to add your newly defined Virtual User to a workflow.

Use the Virtual User in your Workflow

1. Within Nimbus go to Configuration > Workflows.
2. Create a new Audio/Video Workflow or add your new “Add Virtual User” activity to an existing workflow. 
   1. Within the activity, you need to configure the Exit with the Regular Expression ^Understood to ensure that the Virtual User stops after processing the second input. 
      💡The wording “Understood” is adjusted in Copilot.
      
3. You can then add an “Announcement” Custom Context Parameters to listen to their values and work with them according to your needs. 
   1. Example Custom Parameters:
      $(CustomContextParameters.<Language/Topic/Urgency/Sentiment)
      

Known Limitations