Best Practices for Building High-Performance Voice Agents with AssemblyAI

Hi. Building voice agents can sometimes feel like a balancing act between latency and accuracy. But with Assembly AI streaming API, you don't have to sacrifice either. Today, I'll be giving you four best practice tips for building performant voice agents. I'll walk you through the code and then at the end we can have a demo to see what we've built. So, let's jump right in. So, in this code example, which is just a normal example code that you can find on our docs on how to build a voice agent with LifeKit, we're going to scroll on down here to the assembly AI section. And right away, the first thing that we're going to do is disable format terms. So, what this is going to do is decrease the latency of your voice agent while still maintaining its accuracy. For context, LLMs can understand unformatted text well, so there's no need really to use formatted text in the context of a voice agent. If you're using an orchestrator like Pipecat or LiveKit, odds are that this is probably disabled by default, but it's always good to check to make sure that you haven't enabled it by accident. Next, we'll want to set the acoustic turn detection parameters. So, this allows you basically to configure the end of turn acoustically, which means after a set period of silence. This puts a hard cap on how long you want assembly to wait before ending a turn. And it can decrease latency while still putting safety guard rails to prevent overly aggressive turn detection. So how you do this is basically with the minimum end of turn silence confidence parameter and with the max turn silence. So both of these parameters are in milliseconds. What we're basically saying here is when you're confident that the turn is over, the minimum amount of silence before ending the turn should be 160 milliseconds. And whenever the amount of silence is greater than 800 milliseconds, go ahead and end the turn. Next, you'll want to use semantic end of turn detection to basically only end the turn when the model thinks that the speaker has finished speaking. Semantic turn detection is basically only ending the turn once the model believes the user has stopped speaking semantically. So, if the user, you know, blanks on their order, like they say, can I have a hamburger, please? the model will wait for them to say what they want before ending that turn. And this improves performance because the model's actually waiting for the user to finish speaking and they don't feel interrupted. The last piece of advice is to use key terms prompt. So key terms prompt lets you specify terms that you think are likely to appear in the session and this increases the accuracy of your transcription. So in our example, we can add Wendy's, Baconator, Coca-Cola. In the restaurant use case, you'll want to add other items that are on the menu. But for example, if you happen to know the name of the person on the phone, uh it's also good to add their name to this list as well. So now that we've made all these improvements, why don't we go and test our voice agent to see how it does? — Hi there. Welcome to Wendy's. How can I help you today? — Hi, can I get a bacon nater and a side of Coke, please? Awesome choice. Would you like to make that combo with fries or just the Baconator and Coke? — Yeah. Can I get the combo with fries, please? And also, do you have ice cream? — So, you've got a Baconator combo with fries and a Coke. — So, I hope you found this tutorial on how to optimize your Assembly AI voice agents helpful. And if you have any questions, feel free to reach out to our team. We'd be more than happy to help. Bye.