Merge pull request #188 from Azure-Samples/users/vdharmaraj/callAutom…

…ationOpenAiSample

Adding call automation ai sample to th repo

callautomation-openai-sample/.env

@@ -0,0 +1,8 @@
+PORT=8080
+CONNECTION_STRING="<YOUR-ACS-CONNECTION_STRING>"
+CALLBACK_URI="<YOUR-DEVTUNNEL-CALLBACK_URI>"
+COGNITIVE_SERVICE_ENDPOINT="<YOUR-COGNITIVE_SERVICE_ENDPOINT>"
+AZURE_OPENAI_SERVICE_KEY = "<YOUR-AZURE_OPENAI_SERVICE_KEY>"
+AZURE_OPENAI_SERVICE_ENDPOINT="<YOUR-AZURE_OPENAI_SERVICE_ENDPOINT>"
+AZURE_OPENAI_DEPLOYMENT_MODEL_NAME="<YOUR-AZURE_OPENAI_DEPLOYMENT_MODEL_NAME>"
+AGENT_PHONE_NUMBER="<YOUR-AGENT_PHONE_NUMBER>"

callautomation-openai-sample/.gitignore

@@ -0,0 +1,30 @@
+# Ignore node_modules directory
+node_modules/
+
+# Ignore environment variables file
+.env
+
+# Ignore build output directory
+dist/
+build/
+public/assets/
+
+# Ignore IDE/Editor-specific files
+.vscode/
+.vs
+.idea/
+
+# Ignore user-specific configuration files
+.npmrc
+.gitconfig
+
+# Ignore log files
+*.log
+
+# Ignore OS-generated files
+.DS_Store
+Thumbs.db
+
+# Ignore package lock files
+package-lock.json
+yarn.lock

callautomation-openai-sample/README.md

@@ -0,0 +1,57 @@
+|page_type|languages|products
+|---|---|---|
+|sample|<table><tr><td>Typescript</tr></td></table>|<table><tr><td>azure</td><td>azure-communication-services</td></tr></table>|
+
+# Call Automation - Quick Start Sample
+
+This sample application shows how the Azure Communication Services - Call Automation SDK can be used with Azure OpenAI Service to enable intelligent conversational agents. It answers an inbound call, does a speech recognition with the recognize API and Cognitive Services, uses OpenAi Services with the input speech and responds to the caller through Cognitive Services' Text to Speech. This sample application configured for accepting input speech until the caller terminates the call or a long silence is detected
+
+## Prerequisites
+
+- Create an Azure account with an active subscription. For details, see [Create an account for free](https://azure.microsoft.com/free/)
+- [Visual Studio Code](https://code.visualstudio.com/download) installed
+- [Node.js](https://nodejs.org/en/download) installed
+- Create an Azure Communication Services resource. For details, see [Create an Azure Communication Resource](https://docs.microsoft.com/azure/communication-services/quickstarts/create-communication-resource). You will need to record your resource **connection string** for this sample.
+- Get a phone number for your new Azure Communication Services resource. For details, see [Get a phone number](https://learn.microsoft.com/en-us/azure/communication-services/quickstarts/telephony/get-phone-number?tabs=windows&pivots=programming-language-csharp)
+- Create Azure AI Multi Service resource. For details, see [Create an Azure AI Multi service](https://learn.microsoft.com/en-us/azure/cognitive-services/cognitive-services-apis-create-account).
+- An Azure OpenAI Resource and Deployed Model. See [instructions](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal).
+
+## Before running the sample for the first time
+
+1. Open an instance of PowerShell, Windows Terminal, Command Prompt or equivalent and navigate to the directory that you would like to clone the sample to.
+2. git clone `https://github.com/Azure-Samples/communication-services-javascript-quickstarts.git`.
+3. cd into the `callautomation-openai-sample` folder.
+4. From the root of the above folder, and with node installed, run `npm install`
+
+### Setup and host your Azure DevTunnel
+
+[Azure DevTunnels](https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/get-started?tabs=windows) is an Azure service that enables you to share local web services hosted on the internet. Use the commands below to connect your local development environment to the public internet. This creates a tunnel with a persistent endpoint URL and which allows anonymous access. We will then use this endpoint to notify your application of calling events from the ACS Call Automation service.
+
+```bash
+devtunnel create --allow-anonymous
+devtunnel port create -p 8080
+devtunnel host
+```
+### Add a Managed Identity to the ACS Resource that connects to the Cognitive Services Resource
+
+Follow the instructions in the [documentation](https://learn.microsoft.com/en-us/azure/communication-services/concepts/call-automation/azure-communication-services-azure-cognitive-services-integration).
+
+### Configuring application
+
+Open the `.env` file to configure the following settings
+
+1. `CONNECTION_STRING`: Azure Communication Service resource's connection string.
+2. `CALLBACK_URI`: Base url of the app. (For local development replace the dev tunnel url)
+3. `COGNITIVE_SERVICE_ENDPOINT`: Azure Cognitive Service endpoint
+4. `AZURE_OPENAI_SERVICE_KEY`: Azure Open AI service key
+5. `AZURE_OPENAI_SERVICE_ENDPOINT`: Azure Open AI endpoint
+6. `AZURE_OPENAI_DEPLOYMENT_MODEL_NAME`: Azure Open AI deployment name
+6. `AGENT_PHONE_NUMBER`: Agent phone number to transfer the call to resolve queries
+
+### Run app locally
+
+1. Open a new Powershell window, cd into the `callautomation-openai-sample` folder and run `npm run dev`
+2. Browser should pop up with the below page. If not navigate it to `http://localhost:8080/`
+3. Register an EventGrid Webhook for the IncomingCall Event that points to your DevTunnel URI. Instructions [here](https://learn.microsoft.com/en-us/azure/communication-services/concepts/call-automation/incoming-call-notification).
+
+Once that's completed you should have a running application. The best way to test this is to place a call to your ACS phone number and talk to your intelligent agent.

callautomation-openai-sample/package.json

@@ -0,0 +1,29 @@
+{
+  "name": "callautomation_openai",
+  "version": "1.0.0",
+  "description": "",
+  "main": "index.js",
+  "scripts": {
+    "build": "tsc",
+    "dev": "nodemon ./src/app.ts"
+  },
+  "keywords": [],
+  "author": "",
+  "license": "ISC",
+  "dependencies": {
+    "@azure/communication-call-automation": "^1.1.0",
+    "@azure/communication-common": "^2.2.0",
+    "@azure/eventgrid": "^4.12.0",
+    "@azure/openai": "^1.0.0-beta.7",
+    "@types/express": "^4.17.17",
+    "@types/node": "^20.2.1",
+    "dotenv": "^16.3.1",
+    "express": "^4.18.2",
+    "uuid": "^9.0.1"
+  },
+  "devDependencies": {
+    "nodemon": "^2.0.22",
+    "ts-node": "^10.9.1",
+    "typescript": "^5.0.4"
+  }
+}

callautomation-openai-sample/src/app.ts

@@ -0,0 +1,254 @@
+import { config } from 'dotenv';
+import express, { Application } from 'express';
+import { PhoneNumberIdentifier, createIdentifierFromRawId } from "@azure/communication-common";
+import { CallAutomationClient, CallConnection, AnswerCallOptions, CallMedia, TextSource, AnswerCallResult, CallMediaRecognizeSpeechOptions, CallIntelligenceOptions, PlayOptions } from "@azure/communication-call-automation";
+import { v4 as uuidv4 } from 'uuid';
+import { AzureKeyCredential, OpenAIClient } from '@azure/openai';
+config();
+
+const PORT = process.env.PORT;
+const app: Application = express();
+app.use(express.json());
+
+let callConnectionId: string;
+let callConnection: CallConnection;
+let acsClient: CallAutomationClient;
+let openAiClient : OpenAIClient;
+let answerCallResult: AnswerCallResult;
+let callerId: string;
+let callMedia: CallMedia;
+let maxTimeout = 2;
+
+const answerPromptSystemTemplate = `You are an assisant designed to answer the customer query and analyze the sentiment score from the customer tone. 
+    You also need to determine the intent of the customer query and classify it into categories such as sales, marketing, shopping, etc.
+    Use a scale of 1-10 (10 being highest) to rate the sentiment score. 
+    Use the below format, replacing the text in brackets with the result. Do not include the brackets in the output: 
+    Content:[Answer the customer query briefly and clearly in two lines and ask if there is anything else you can help with] 
+    Score:[Sentiment score of the customer tone] 
+    Intent:[Determine the intent of the customer query] 
+    Category:[Classify the intent into one of the categories]`;
+
+const helloPrompt = "Hello, thank you for calling! How can I help you today?";
+const timeoutSilencePrompt = "I’m sorry, I didn’t hear anything. If you need assistance please let me know how I can help you.";
+const goodbyePrompt = "Thank you for calling! I hope I was able to assist you. Have a great day!";
+const connectAgentPrompt = "I'm sorry, I was not able to assist you with your request. Let me transfer you to an agent who can help you further. Please hold the line and I'll connect you shortly.";
+const callTransferFailurePrompt = "It looks like all I can’t connect you to an agent right now, but we will get the next available agent to call you back as soon as possible.";
+const agentPhoneNumberEmptyPrompt = "I’m sorry, we're currently experiencing high call volumes and all of our agents are currently busy. Our next available agent will call you back as soon as possible.";
+const EndCallPhraseToConnectAgent = "Sure, please stay on the line. I’m going to transfer you to an agent.";
+
+const transferFailedContext = "TransferFailed";
+const connectAgentContext = "ConnectAgent";
+const goodbyeContext = "Goodbye";
+
+const agentPhonenumber = process.env.AGENT_PHONE_NUMBER;
+const chatResponseExtractPattern = /(?<=: ).*/g;
+
+async function createAcsClient() {
+	const connectionString = process.env.CONNECTION_STRING || "";
+	acsClient = new CallAutomationClient(connectionString);
+	console.log("Initialized ACS Client.");
+}
+
+async function createOpenAiClient() {
+	const openAiServiceEndpoint = process.env.AZURE_OPENAI_SERVICE_ENDPOINT || "";
+	const openAiKey = process.env.AZURE_OPENAI_SERVICE_KEY || "";
+	openAiClient = new OpenAIClient(
+		openAiServiceEndpoint, 
+		new AzureKeyCredential(openAiKey)
+	  );
+	console.log("Initialized Open Ai Client.");
+}
+
+async function hangUpCall() {
+	callConnection.hangUp(true);
+}
+
+async function startRecognizing(callMedia: CallMedia, callerId: string, message: string, context: string){
+	const play : TextSource = { text: message, voiceName: "en-US-NancyNeural", kind: "textSource"}
+	const recognizeOptions: CallMediaRecognizeSpeechOptions = { 
+		endSilenceTimeoutInSeconds: 1, 
+		playPrompt: play, 
+		initialSilenceTimeoutInSeconds: 15, 
+		interruptPrompt: false, 
+		operationContext: context, 
+		kind: "callMediaRecognizeSpeechOptions",
+	}; 
+
+	const targetParticipant = createIdentifierFromRawId(callerId);
+	await callMedia.startRecognizing(targetParticipant, recognizeOptions)
+}
+
+function getSentimentScore(sentimentScore: string){
+	const pattern = /(\d)+/g;
+	const match = sentimentScore.match(pattern);
+	return match ? parseInt(match[0]): -1;
+}
+
+async function handlePlay(callConnectionMedia:CallMedia, textToPlay: string, context: string){
+	const play : TextSource = { text: textToPlay, voiceName: "en-US-NancyNeural", kind: "textSource"}
+	const playOptions : PlayOptions = { operationContext: context };
+	await callConnectionMedia.playToAll([play], playOptions);
+}
+
+async function detectEscalateToAgentIntent(speechInput:string) {
+	return hasIntent(speechInput, "talk to agent");
+}
+
+async function hasIntent(userQuery: string, intentDescription: string){
+	const systemPrompt = "You are a helpful assistant";
+	const userPrompt = `In 1 word: does ${userQuery} have similar meaning as ${intentDescription}?`;
+	const result = await getChatCompletions(systemPrompt, userPrompt);
+	var isMatch = result.toLowerCase().startsWith("yes");
+	console.log("OpenAI results: isMatch=%s, customerQuery=%s, intentDescription=%s", isMatch, userQuery, intentDescription);
+	return isMatch;
+}
+
+async function getChatCompletions(systemPrompt: string, userPrompt: string){
+	const deploymentName = process.env.AZURE_OPENAI_DEPLOYMENT_MODEL_NAME;
+	const messages = [
+		{ role: "system", content: systemPrompt },
+		{ role: "user", content: userPrompt },
+	];
+
+	const response = await openAiClient.getChatCompletions(deploymentName, messages);
+	const responseContent = response.choices[0].message.content;
+	console.log(responseContent);
+	return responseContent;
+}
+
+async function getChatGptResponse(speechInput: string){
+	return await getChatCompletions(answerPromptSystemTemplate, speechInput);
+}
+
+app.post("/api/incomingCall", async (req: any, res:any)=>{
+	console.log(`Received incoming call event - data --> ${JSON.stringify(req.body)} `);
+	const event = req.body[0];
+	try{
+		const eventData = event.data;
+		if (event.eventType === "Microsoft.EventGrid.SubscriptionValidationEvent") {
+			console.log("Received SubscriptionValidation event");
+			res.status(200).json({
+				validationResponse: eventData.validationCode,
+			});
+
+			return;
+		}
+
+		callerId = eventData.from.rawId;
+		const uuid = uuidv4();
+		const callbackUri = `${process.env.CALLBACK_URI}/api/callbacks/${uuid}?callerId=${callerId}`;
+		const incomingCallContext = eventData.incomingCallContext;
+		console.log(`Cognitive service endpoint:  ${process.env.COGNITIVE_SERVICE_ENDPOINT.trim()}`);
+		const callIntelligenceOptions: CallIntelligenceOptions = { cognitiveServicesEndpoint: process.env.COGNITIVE_SERVICE_ENDPOINT };
+		const answerCallOptions: AnswerCallOptions = { callIntelligenceOptions: callIntelligenceOptions }; 
+		answerCallResult = await acsClient.answerCall(incomingCallContext, callbackUri, answerCallOptions);
+		callConnection = answerCallResult.callConnection;
+		callMedia = callConnection.getCallMedia();
+	}
+	catch(error){
+		console.error("Error during the incoming call event.", error);
+	}
+});
+
+app.post('/api/callbacks/:contextId', async (req:any, res:any) => {
+	const contextId = req.params.contextId;
+  	const event = req.body[0];
+	const eventData = event.data;
+	console.log(`Received callback event - data --> ${JSON.stringify(req.body)} `);
+	console.log(`event type match ${event.type === "Microsoft.Communication.CallConnected"}`);
+	if(event.type === "Microsoft.Communication.CallConnected"){
+		console.log("Received CallConnected event");
+		startRecognizing(callMedia, callerId, helloPrompt, 'GetFreeFormText');
+	}
+	else if(event.type === "Microsoft.Communication.PlayCompleted"){
+		console.log("Received PlayCompleted event");
+
+		if(eventData.operationContext && ( eventData.operationContext === transferFailedContext 
+			 || eventData.operationContext === goodbyeContext )) {
+				console.log("Disconnecting the call");
+				hangUpCall();
+		}
+		else if(eventData.operationContext === connectAgentContext) {
+			if(!agentPhonenumber){
+				console.log("Agent phone number is empty.");
+				handlePlay(callMedia, agentPhoneNumberEmptyPrompt, transferFailedContext);
+			}
+			else{
+				console.log("Initiating the call transfer.");
+				const phoneNumberIdentifier: PhoneNumberIdentifier = { phoneNumber: agentPhonenumber };
+				const result = await callConnection.transferCallToParticipant(phoneNumberIdentifier);
+				console.log("Transfer call initiated");
+			}
+		}
+	}
+	else if(event.type === "Microsoft.Communication.playFailed"){
+		console.log("Received PlayFailed event");
+		hangUpCall();
+	}
+	else if(event.type === "Microsoft.Communication.callTransferAccepted"){
+		console.log("Call transfer accepted event received");
+	}
+	else if(event.type === "Microsoft.Communication.callTransferFailed"){
+		console.log("Call transfer failed event received");
+		var resultInformation = eventData.resultInformation;
+		console.log("Encountered error during call transfer, message=%s, code=%s, subCode=%s", resultInformation?.message, resultInformation?.code, resultInformation?.subCode);
+		handlePlay(callMedia, callTransferFailurePrompt, transferFailedContext);
+	}
+	else if(event.type === "Microsoft.Communication.RecognizeCompleted"){
+		if(eventData.recognitionType === "speech"){
+			const speechText = eventData.speechResult.speech;
+			if(speechText !== ''){
+				console.log(`Recognized speech ${speechText}`);
+				if(await detectEscalateToAgentIntent(speechText)){
+					handlePlay(callMedia, EndCallPhraseToConnectAgent, connectAgentContext);
+				}
+				else{
+					const chatGptResponse = await getChatGptResponse(speechText);
+					const match = chatGptResponse.match(chatResponseExtractPattern);
+					console.log(match);
+					if(match){
+						console.log("Chat GPT Answer=%s, Sentiment Rating=%s, Intent=%s, Category=%s",
+						match[0], match[1], match[2], match[3]);
+						const score = getSentimentScore(match[1].trim());
+						console.log("score=%s", score)
+						if(score > -1 && score < 5){
+							handlePlay(callMedia, connectAgentPrompt, connectAgentContext);
+						}
+						else{
+							startRecognizing(callMedia, callerId, match[0], 'OpenAISample')
+						}
+					}
+					else{
+						console.log("No match found");
+						startRecognizing(callMedia, callerId, chatGptResponse, 'OpenAISample')
+					}
+				}
+			}
+		}
+	}
+	else if(event.type === "Microsoft.Communication.RecognizeFailed"){
+		const resultInformation = eventData.resultInformation
+		var code = resultInformation.subCode;
+		if(code === 8510 && maxTimeout > 0){
+			maxTimeout--;
+			startRecognizing(callMedia, callerId, timeoutSilencePrompt, 'GetFreeFormText');
+		}
+		else{
+			handlePlay(callMedia, goodbyePrompt, goodbyeContext);
+		}
+	}
+	else if(event.type === "Microsoft.Communication.CallDisconnected"){
+		console.log("Received CallDisconnected event");
+	}
+  });
+
+app.get('/', (req, res) => {
+	res.send('Hello ACS CallAutomation!');
+  });
+
+// Start the server
+app.listen(PORT, async () => {
+	console.log(`Server is listening on port ${PORT}`);
+	await createAcsClient();
+	await createOpenAiClient();
+});

callautomation-openai-sample/tsconfig.json

@@ -0,0 +1,13 @@
+{
+  "compilerOptions": {
+      "target": "ES2015",
+      "module": "commonjs",
+      "outDir": "./dist",
+      "rootDir": "./src",
+      "moduleResolution": "node",
+      "esModuleInterop": true,
+      "skipLibCheck": true,
+      "forceConsistentCasingInFileNames": true
+  },
+  "include": ["./src"]
+}