How to Build and Deploy an AI Voice Agent using Pipecat

Hi everyone. We're going to show you how to build, package, and deploy an AI agent to Pipecat Cloud. Here's a demo of the app we'll be creating. Hello world. I am your virtual assistant here to help with whatever you need. Just let me know how I can assist you today. Can you tell me what a AI voice agent is? Sure thing. An AI voice agent is a tool powered by artificial intelligence that can understand and respond to spoken language. Pipcat is an open source

framework for creating and running conversational AI and is maintained by daily. co, a global web RTC infrastructure company. This is a Python application where Pipcat is the orchestrating framework. Assembly AI will handle the ST or speechtoext portion. OpenAI is the general reasoning LLM, the larger language model and Cartisia is the TTS or texttospech component. We'll be using UV today to manage the virtual environment and dependencies. DockerHub will be used to host a Docker image of our completed agent and daily. co will provide the infrastructure and guey for the demo. These are the steps we'll be walking through. Installing and using Pipcat Cloud CLI, importing dependencies, adding API keys, integrating assembly AI, testing locally, and packaging and deploying to DockerHub and Pipcat Cloud.

Starting in our terminal here, let's create an empty project folder and jump in. Next, install the Pipcat Cloud CLI by running uv tool install Pipcat Cloud. Using this CLI, we'll log into our Pipcat Cloud account by running PCC O login. Now we can run PCC enit to prepopulate our project.

We can now hop into code. If you have cursor installed, you can run cursor space period to open the project. Select the pre-generated requirements. txt file and update the pipecat-ai line. We're going to add in entries for assembly AI and web RTC. Next, we'll open up a terminal interface right from cursor to spin up our virtual environment with uv space venv followed by uv space init. Now we can install all the dependencies from the requirements. txt file by running uvp pip install-r requirements. txt. For this application to run, we'll need

API keys for a few services. In the env. example file, you'll notice there are three keys listed. Cartisia, OpenAI, and Daily. Add in an additional var for assembly AI named assembly AAI_I_key. Save. Then copy the example file to a newv file before adding in your actual API keys. To get your Assembly AI API key, log in to the Assembly AI dashboard. Select the API keys sidebar menu option. Then copy or create a new API key. An OpenAI API key can be found through the OpenAI dashboard. Select the settings icon in the upper right. Then select the API keys option in the lefth hand sidebar. Keys are only displayed once, so reuse a prior key or create a new one, making sure to copy this somewhere secure. Cartisia's API key can be found on their dashboard, also in the lefth hand sidebar under the platform section, API keys. Use the plus new button in the upper right to create a new key with the resulting dialogue boxes. Save securely as this will not be displayed again as well. The daily API key can be found in your Pipcat cloud account dashboard. Go to the settings tab, then select the daily web RTC option, not the API keys option. At the bottom of the resulting page is a section for the daily API key. Use this value

to add speech to text to our app. We'll modify the bot. py file. Start by adding an import statement for the Assembly AI ST service and Assembly AI connection params. Scroll down and initialize an instance of the Assembly AI ST service with an instance of Assembly AI connection params and assign it to a var named ST. Next, we're going to add it to the pipeline after the context aggregator user and before the LLM. Assembly AI will now be used to transcribe the voicetovoice interaction between a user and the agent. To monitor the transcription output, import the transcript processor. Initialize an instance of the transcript processor before creating the pipeline. In the pipeline in it, add the transcript user after the ST line and the transcript assistant after the transport output line. Last add an event handler to process the on transanscript update call back. Here we are just printing them to console. When we run the app again, we should now see transcripts of the conversation printout. The initial code has already been configured for local running when a local_run flag is added to the environment. So we'll test locally using

that flag. Hello world. I am an advanced language model here to assist, inform, and entertain you. How can I assist you today? Can you tell me about Pipecat by Daily. co? Of course. Pipedream is a company known for creating innovative solat pipe c by daily. co. My apologies for the mixup. Pipecat byaily. co is a tool that complements daily, a leading webtc video and audio platform.

Now that we know the app runs, we can package it for cloud deployment. Now this process requires two steps. Building and hosting an image to dockerhub followed by deploying our configuration to pipecat cloud. First update the pCC-deploy. toml file that was autocreated for us earlier. Give the agent any name. In our case, we will just keep the default sample name. Next, set the image value to a combination of your DockerHub username and the agent name that you used above. Finally, assign a name to the secret set. This will be the name used to reference uploaded secrets for use with this image. For simplicity, just append the agent name with secrets as the value. Next, we'll need to build and push up a new Docker image to DockerHub. To do this, we'll run four commands. Docker login, Docker build, Docker tag, and then Docker push. A successful push can be verified by going to your Docker to hub dashboard. Click on the repositories and your new image should be listed there. Now that the image is available on DockerHub, we can deploy the agent to Pipcat Cloud. Start by uploading the secrets from the ENV file by using the PCC secrets set command. Be sure to use the same secret set value used in the PCC deploy file. Once completed, we can now run PCC deploy. This will use the information from the tommo file by default. A success message should return. To check again later or on prior agents, run the PCC agent status command. To interact with this agent, start it again, but add the use daily and API key flags. To find this API key, go to your Pipcat Cloud dashboard. Click on settings, API keys, public, and then copy the key you want to use. Paste it after the API key flag and then run. If started successfully, you will get a URL link for a web-based agent interface. And that's it for this video. Check out the description for a link to the repo with the final code demonstrated. Don't forget to check out our YouTube channel for other helpful tutorials. Like and subscribe to get notified when we release new content on AI and building voice agents.