Coding Challenge Session: Local Browser Conversational Chatbot (STT, TTS, and more?)

Heat. Heat. Hey, hey, hey. All right, surprise. I'm back. Um, if you're there and you can hear me, okay, let me know in the chat. Um, be starting up in just a few minutes. Heat. Let's go back.

Hi everybody. I'm back. I was just here and I left and I came back. I was not satisfied with this morning's live stream and I have the time today, possibly a mistake to do this. I'm not checking to see if I was muted. Um, but I'm going to I think what I've been longing to do, which I've been struggling to do, is just have a topic for a live stream. Come on, get Come on. I will come onto the live stream and boom, get started with that topic, build a project, talk it through, mistakes, confusion, warts, and all, and then say goodbye. So, that's what I'm going to do in this live stream. If you were here this morning, let me know in the chat. But, um, I'm assuming it's kind of going to be new people. The other thing that's kind of interesting is that YouTube will, I don't think, will notify people twice in the same day if a channel live streams multiple times. it will just do the first notification. So not that there was like a huge audience earlier but last time I did a second live stream the audience actually grew much larger I think because of the topic. So yes B2B 331 this is the second live stream today and my assumption is that whatever I do today will get edited into a standalone video um that will come out probably several weeks from now. I don't want to get lost in like let me tell you everything that I'm planning or because I've been doing so much of that. So I'm just going to jump right in. But I will say there's quite a backlog. Let me just list basian text classification coding challenge video. Introduction to transformers. js video. Uh I think I did a language model, transformers model, local model in the browser session during one of my live streams. I think that's going to go fall onto the cutting room floor and I'm going to come back to revisit that again. I mean it might appear concept might appear in today's stream. So um so that's just two basian intro to transformers. js. Then last week I did a stream all about sampling and temperature. And then today, earlier today, I did looking at new font and text functions in p52. And so this would be the fifth video in the pipeline to come out on the coding train. But I'm trying to flood the zone here. No long no longer are these like cur you know um artisally edited um I mean there's still a lot of editing hap post-production happening but uh I'm trying not to think overthink things do live streams cut them up release rinse and repeat okay um okay so um I see there's some questions in the chat but not the time for that right now let's Just get let's begin. Oh, don't Okay. All right. Um, let's begin. I can do this. I can just begin. Very little, very little stopping and starting. Messy messiness encourage. Um, oh wow, Kobe is here. So, I just want to give a big shout out to Kobe because Kobe has contributed so much to the coding train over the years. I've not seen Kobe during a live stream in quite some time. This is what the afternoon Eastern time brings out. Brings out the Europeans. I don't know if anybody gets that reference. It's European, but uh if you do, great. And I don't even know if I'm saying it in a way that reference will make any sense. Uh all right. Hi everybody. Welcome to a coding challenge video. I am unprepared. I am going to fumble my way through this. I'm going to attempt to make a conversational chatbot, one that listens to me speak and speaks back to me. And yes, there's going to be some small tiny little quotes AI involved. And I will talk about that. The purpose of this is not to create a useful, productive, meaningful assistant. The use the thing that I'm trying to do with this is have some fun. I said I wasn't going to start over. Sweden is in Europe. Hi from Ecuador. Um, okay. I wanted I I'm having trouble. I wanted to get out of my system. I just I said Okay, it's fine. Give me a couple chances with the opening and then we're moving right into it. As I always say, hi everybody. Today I'm going to attempt to build a conversational voice chatbot. The key things that I want to demonstrate are how uh in a p5 js sketch

that you're seeing right here. What would it mean to you if you could talk to your p5 js sketch and could talk back to you? What kinds of creative strange and odd possibilities could emerge? I'm doing more videos and coding challenges that make use of machine learning models. My point of view about that is a couple things. One, I want to demystify the technology. I want you to see how it works, play with the material of it. I also want to show encourage and show how you can use this technology to make weird, strange, creative, artistic projects. I want to show how you can use this technology for creative expression. What does it mean to make art with the technology? social commentary to be critical of it? And let's just I whatever I said this morning was so much better when I wasn't trying to say anything. I just was talking. Last chance. It's okay. people are I started I started um 10 minutes early and uh I'm letting people trickle in. So as people trickle in, we're gonna we're going to get to this once I get to the coding. No stopping and starting allowed just to sort of like get my thoughts out here. Hi everybody. I am here to make a coding challenge and today I'm going to attempt to make a conversational voice chatbot right here in a P5GS sketch. So, I am asking the question to you. If you could talk to your sketch and your sketch could talk back to you, what kinds of projects would you make? I think that's enough. Um, so let me map out to you the pieces of this project as I envision them. We need some system for speech to text. by speech to text. I mean I speak the computer listens converts it into text. Then we need a system for text to speech. By that I mean I have a string of text and I have want it to be spoken by an automated voice. Then I need a brain. I'll say a brain for the bot. What h how is the bot going to process the inputs that it gets from speech to text and generate the outputs that it sends to text to speech? So this is where we have to have a talk about a I want to put some more cogent thoughts maybe into an entire video just about my approach to working with AI and machine learning models on this channel. I've been doing it for years already. I have many videos about what is a neural network. I have videos about the ML5JS library and I plan to make many more of those. I'm trying to find my way, carve myself into this area, dip my toe in, but not jump all the way in and drown. At present, my goal is to number one demystify the technology. I want to look at how things work and be able to put the technology into the hands of viewers like you to be able to have agency and understanding and um which is needed I think in this world that we're living in where AI models seem to be just popping up everywhere. that's what that's I sorry I was reading the chat and then the next thing I was going to say is demystify the other thing is I am personally curious to investigate what is possible

with the technology that is being offered by huge behemoth big tech companies with closed cloud-based systems. what's possible for the individual to do on consumer hardware with open-source models with your own data. So that's what I'm trying to prioritize in these series which is for many. So that's what I'm trying to prioritize at present. For me, I think the best way to learn about something is to learn by doing. And I think also the best way to examine and be critical of something is through creative play, art, expression. So that's my question here. If we look at a machine learning model for speech to text, text to speech, maybe even a language model, putting those all things to putting all those things together in a p5 js sketch, what creative possibilities does that unlock for you, the viewer? What kind of art would you make? What kind of social commentary? tools could you make to help yourself and others? How can you be critical and thoughtful about the world of AI technology through making your own projects with the material that is the machine learning models, the AI systems themselves? Okay, so off my soap box, I don't know. Okay, that's just the way I'm thinking of navigating it today. Let's I'm going to put that to the side and start building this project. So that begs the question, what am I going to use for all of these things? For speech to text, I'm going to use a particular model called Whisper. This model was developed by OpenAI, but maybe from a different time. It is an open- source model that can be run locally on your computer, even in the browser. You'll see that for text to speech I'm going to use something that's come out pretty recently. Thank you to Zova from hugging face the creator of transformers. js who pointed me to this model. It's called Kokoro TTS. I'm making the point that these are very small, very lightweight, very fast. M These are small, lightweight, fast models. They're a small footprint. Um they run so they aren't as capable or advanced. I don't know what to say. They aren't as capable or advanced as other types of systems that you might find that operate that run in the cloud that are commercial that are proprietary. I will mention that for voice m yeah maybe and if you have if you have ideas or thoughts about other models you're interested in or that you might like to use, feel free to let me know in the comments. I don't know. I don't need to say all that. That's fine. Whatever. Um, then the brain is the big question. To me, this is almost the least important thing for me to do. This is where I ask you to be creative. You don't need a large language model to be the brain of your chatbot. On my channel, I've looked at marov chains, contextfree grammarss, simple pattern uh pattern matching systems like arrive script for chat bots. So many possibilities there. So this is where I want you to think about and be as creative as you can if you build anything on top of what I'm going to attempt to do in this challenge today. Okay, I don't know that I needed the whiteboard, but I have this new whiteboard, so I had to use it. here we go. The other thing I should mention is that I am using p52 and I will be making use of features of p52 like support for async and await. So, if

that's all new to you, check out my video about that. Let's start by building a very crude interface. I'm going to make this a pushto talk system. So, I need mouse pressed and mouse released. All right. There's literally going to be no interface, so I don't need that big of a canvas. I want more room for my code. should it be red when you're pushing and then green when you release? I'm not sure, but I'll just put something in. Okay. The idea here is that when I press the mouse, hello chatbot thing, and then when I release, it's going to transcribe what I just said. So that's got to be step one. And I have just again a very crude interface. Color it red for mouse press, colored green at mouse released. Another prerequ another pre-erequisite for this video is my introduction to transformers. js JS where I looked at just how the ecosystem of hugging face models and transformers. js the JavaScript library works. But I'm just going to jump right into that. So you can for more background. So I guess you don't really have to watch it to follow this video, but it'll give you more background as to what I'm doing if you're interested. So, on the HuggingFace website, I'm going to look for the model I want to use. Let's go under models. I'm going to look under tasks. And look at this. Text to speech. No, no. I want speech to text. It's actually here under automatic speech recognition. So, let me click that. Then I need to make sure whatever model I find is compatible with transformers. js. So I'm going to go to libraries. Transformersjs and woo. Now we have a lot of options. First let me just say a couple words about whisper. Again, Whisper released by OpenAI back in September 2022. This is a fully open model with the paper, the code, and something called a model card, which anytime you're using a machine learning model is something you absolutely want to read and review first. Beyond even reading the model card, might I suggest to you read the paper model cards for model reporting by Margaret Mitchell and collaborators. This is a seminal paper in um maybe I shouldn't model cards for model reporting by many luminaries and uh I guess I could say Margaret Mitchell is the first name and collaborators. Is that an appropriate way to say it? I don't want to say it incorrectly. Um but um

this paper provides a framework for how machine learning researchers who are training models should uh release the information about their models. It's I guess this paper create uh proposes a framework that encouraged transparent model reporting. So various things so all sorts of important information about what data was used to train the model, what its intended uses are, how it was evaluated and other key elements and background for that model itself. Okay, here's the Whisper model card. And an important thing to note here is it comes in various sizes. So for me, I'm looking for the smallest one, tiny. The size has to do with the number of connections essentially inside of the neural network architecture. Beyond the scope of this video, but I'll point you to additional resources to understand more about that in this video's description. the other. So, I'm poking around here um to find the one that I want to use. I think I'm going to go look under onyx community. Actually, let's filter. So, onyx community. Onyx. So, onyx stands for open neural network exchange. It's a standardized format for storing the weights of a neural network. and it just so happens to be compatible with JavaScript and transformers. js, the library in particular. So, I'm going to use whisper tiny. Because I'm only going to speak to it right now in English, and that'll get me the smallest model for doing this today in this coding challenge. copy that model path and come back to p5. Okay, I've got to import the transformers. js library. And the way I'm going to do that is by making set up an async function and then I'm going to get the transformers. js pipeline from the library. So, I'm going to call a special function in JavaScript called import, which is a fancy new way to bring in libraries. I said a lot more about this in that other video. And I need the path to the library. Um, where am I going to find that? Can I find that? Where am I? I I have it. I have some notes here. So, could just type it, but I was kind of like to find where it goes. Oh, here we go. There we go. This is perfect. Okay. Found it on the um installation page for transformers. js. Oh, is there something going on with the microphone? Let me just check this. Can uh I'm just taking a pause for a second to see if there's like significant audio issues.

Okay. So according to this the next thing I might do is oh let me make a pipeline for the text to speech but let's actually build this in parts and let's first just only get the whisper model working. Oh, shoot. Okay. How long was I muted? Oh, crap. muted for? I'll wait. You gotta go. That's fine. I'll do it again. Uh, the whiteboard wasn't muted weirdly enough. Okay. I'm glad you guys uh Yeah. Yeah. I'm waiting for the chat to catch up to me. Okay. So, how long ago was that since you checked the mic? Well, what I don't remember what I did when I checked the mic. since the start of the transcriber. Okay. Had I done this, I know, but I don't remember since I don't I guess I'm just going to go back to further. I'll go back further because I don't remember uh when obviously it was muted since I checked for audio issues when I muted it for a second. I just don't remember what I was doing then. Uh talked about import. Okay. But I got did I get all the way through import? I think I did. Okay. Yeah. All right. Um hopefully right after you talked about import great. So pipeline is actually a function and I can use it to return I this is too bad because I don't

remember but just imported the transform. Okay. Did I did you hear me explain what a pipeline is? I think I did. Anyway, so now I actually want to get a specific pipeline. Pipeline is a function that will create Did I Yeah, this is really too bad. Didn't hear about the transcriber. Yeah. 9 to 11. I know, but what did I say last? Uh, I guess I could I should just go back and look. Yeah. Okay. I'm gonna explain stuff again. It's fine because it's better for me to do that and then and it'll be more seamless. Anyway, so I've imported the library and what I want from the library is this thing called a pipeline. What is a pipeline? pipeline. It's like it's the thing that gets the stuff into it and then puts the stuff out of it. It's essentially where the machine learning model is held. And I want to make a speechtoext pipeline, a texttospech pipeline. So to do that, I can actually put this pipeline into a variable. So to do that, I actually call pipeline. So to do that pipeline is actually a function and I'm going to take the results of that make up something. Let's call it a transcriber. Need a variable name where I'm going to create a pipeline. That error message shouldn't have been there the whole time. So I'm going to get to do this one more time. Installation. Oh, installation instructions from transformer. js. But I got all the way through typing this line, right? Okay. I'm gonna paste it in here. Let's do this. I guess I closed that. Ah, it's terrible when I lose my momentum here. Okay. So, now I've actually imported the library. And I've imported the library, at least the most important part of the library into this object or what it actually is a function called pipeline. And I'm going to use that to create a machine learning pipeline. Pipeline I guess being the word because the data is coming into it and then the results of processing that data flow out of it. And I want to do text to speech. So let me make up a variable. I'll call it transcriber. And I'm going to await the creation of that pipeline. And the two key things I need for any given pipeline are the name of the model that I want to use and the task or I should have said it in the other order. I need the task and then the specific model to execute that task. So the task is automatic speech recognition. I think that's what it was called. And the model is this whisper tiny English I also want to specify the device to run the model on. And in this case, web GPU, the browser's connection to the graphics hardware is uh is the device that the model will run best on. It's really important for me to note even though I'm going to turn on my microphone and have this P5 sketch listen to me, my audio is never going to leave my computer. It's all going to be processed locally on this computer inside the browser. The model is being pulled from the cloud, but the model runs inside the browser locally. Okay, so next I might want to create a pipeline for text to speech, but let's stick with speechto text. Get that fully working before we move on to the next step. Um. Ah. Okay. So, we have to have another talk about P5 sound. P52 has a new Oh, okay. So, we have to have another talk and it's about um what uh so now we have to have a talk about how to get access to the microphone. So, there is a P5 sound library. Incredible library.

There's actually a new version of the library, fully compatible and meant designed for P52. I would really like to make some video tutorials about it and somebody in the future who's watched this video should bug me and remind me about that. For now, what I think I'm going to do because I'm gonna figure out a bunch of things in order to like process the audio into the right format to go into the Whisper model. I'm going to use native web audio API and then hopefully by the time you're watching this there will be a version of this ex whatever example I make here um working with p5 uh the sound library as well. So there should be two code examples in this video description. I don't know you tell me you're watching this now presumably in the future. Okay. Um, so I need to look up navigator media devices, get user media. So I need to look up, it's called navigator media devices. I think the function is called get user media. So that would be here. Is there an example? Yes. Constraints. Okay. This, I believe, is the code to open up a connection to the microphone. So, I'm going to grab this and let's make a global variable called mic. And right here, I'm going to say uh mic equals await navigator media devices get user. Oh, the constraints. So, what I want I believe is audio true. I think that's the property. Let's just run this sketch. Did it ask me for the microphone? Ah, well, it didn't ask me. I've already given it permission. But we can see the microphone is in use. This page is accessing your microphone. Sweet. Okay. Now, the whisper model, I believe, now the whisper model is designed to process an audio file essentially. This is something maybe we could talk about later. There's probably a way we could finesse this to have it always be listening and detect pauses. But again, I'm going to build a pushto talk. Listen to me, stop listening to me. So, to capture audio between the start and stop listening, I need a media recorder. And again, this is all stuff that the p5 sound library wraps, but it's good for us to in this video to learn about how it works underneath the hood. So, I think I can just make another global variable called recorder. And then I can say recorder equals new media recorder mic. That seems good. Okay, now I need to look for the events. So events start stop. Okay, those I need. And then I need data available. Where's that data available event? Yes. Okay. This is what I need. need because the as there's data available, I need to collect all the audio data into an array because the array is what I'm going to pass to the whisper model. This might be the hardest part of the whole thing to be honest. So, let's grab this. And then let's make um another global variable called uh chunks like the audio chunks. I feel like that's a standard variable name people use.

Then as data comes in push that data into the audio chunks array. Okay. And when I press the mouse that's when I just call recorder. st start. I'm just double check checking. I mean, I shouldn't be double-checking my notes so much here, but yeah, that's what that that's right. Yeah. Okay. And then when I do mousereleased recorders stop and then let's look at the audio chunks just to make sure this is working. Hello. I am speaking in chunks of audio. Great. We got a big blob of audio. Perfect. Now, I probably could just process the audio right here, but I think my Discord is beeping and so I'm uncertain when the if that means something important. Kind of want to turn off the beeping, but it was good because it started going wild when my audio is muted. I'll just assume one beep doesn't mean anything. I think I could just process the audio here, but recorder. stop it might actually not be done instantaneously. And I believe stops recording at which point a data available event contains the final blob of saved data. Oh, huh. So maybe I could get all the data all at once. I'm going to do it the way I'm doing it anyway, but I'm looking for more events. Stop. Yeah, I want the stop event. And that's when I want to process the audio. So recorder onstop. Now I'm kind of mixing different JavaScript style. I'm using this like arrow syntax and anonymous function right there for on data available. But I think I'm going to have to do more stuff in onstop. So let's set that equal to a function that I'm going to called like process audio. And I assume eventually at some point I'll clean up and refactor this code, which I always do. I mean clearly if you've ever watched the coding train, all of my code is perfectly refactored and organized at the end of every coding challenge. Okay. So, okay. Ah, yes. Yes. Yes. So, uh, let's look at the let's look at what I have in the chunks again. Hello. It's an array with a blob in it. And I want this array buffer. I have to turn it into a waveform. I mean, that's what I remember from Zenova's examples that he shared with me before I started this coding challenge. So I think I can call this array buffer function.

But what if the audio chunks has more than one blob in it? That's my issue. I think I can make a blob out of however many things are in there because blobs are very flexible. Let's try this. Hello. Yeah, now I took that array which had a blob in it and made it a blob. I think that's good. I think that'll protect me better. And now I could call array buffer. Is that does that need a wait? Let's find out. Hello. Yeah, it's a promise. Where the console is bad, but Matio will fix that. It'll lift it up. It's a promise, meaning I've got to await it. No. Ah. and async it. Hello. Where am I? Why is it empty? Okay, I'm not going to worry about it being empty. I think it's probably not actually empty. So, I need to convert it into a waveform. And to do that, I need the audio context because there's a decode audio data function. Let's see. Decode audio data. See, you can take an array buffer and decode it into something. It's all about like processing the raw. It's the raw audio data is just numbers, but we've got to reformat it into a way that the model's going to understand. I don't really know enough about this stuff to speak with any intelligence here. Um, okay. But where do I call this function? Yeah, audio context. So, I need to get the audio context first. So, let's make a variable. Let's just call it CTX. And at the beginning or right before I do this, let's make a new audio context for this whole system. Yeah. And something that I know also from Zenova's examples is that I need to set the sample rate. I believe that this particular whisper model uh requires the audio to have a 16 hertz sample rate. 16,000 hertz. What is it? kilhertz 16 kilhertz 16,000 is 16 kilhertz excellent okay so now I should be able to uh get the decoded audio is the context text decode audio data the array buffer and I have to await that. Okay, I'm almost there. Let's look at the decoded audio because I think the waveform is in there. Hello. Okay, I've got an audio buffer with a sample rate. It was a very short

duration. It's got one channel. Excuse me, I just burped. Get channel data. That's it. It's only got one channel. So, I need to get the channel data from channel one or I think it's channel zero. Let waveform equals await dec uh decoded audio get channel data zero. Console log. Let's look at the waveform. Ohoy. Cool. Yeah, look. Here's all the raw audio. We could graph it. I bet you we could graph this and have a nice little like wave. We could have like cool visualizations of me talking. Okay, but that's not the point of this video. I'm not going to do that. I think I could just do this. Now let uh text equals await. Guess what? The transcriber just needs the waveform console log the text. Okay, ready for this? I think we've got it. Um I'm trying to think of something to say. Freshen your drink, Governor. had an error. That was so sad. What did I miss? Transcriber is not defined. Ah, that needs to be a global variable the way that I've written this code. Again, maybe not best practice, but let's try this again. Choo speech to text done. Okay. Ah, actually we have to do this out of order. Now we need the brain to process it. One of the first chat bots. One of the most famous first early chat bots was a chatbot named Eliza. And I think it stood for something. Let's go find this out. Eliza is an early natural language processing computer developed from 1964 to 1967 at MIT by Joseph Weisenbomb. What did it stand for? Thought it stood for something. So these early systems and I've talked about this before in other videos about what was it called? AI um ML a markup language. Where's that AI markup language? Yeah, I've talked about this before in previous videos. There's something called the artificial intelligence marov lang marov language. artificial intelligence markup language which was created um in 95. Aha. Alice was an extended Eliza which stood for artificial linguistic internet computer entity. What's it called when the name of something is engineered to stand for something? Like Dolli was an example of this. Anyway, um not important. Okay, these systems are p um pattern and template.

template. These systems were pattern matching systems looking for a particular pattern in what a user was saying and then having a um predefined response for that pattern. And those predefined responses could include variables and conditionals and all the traditional tools we have in programming. Uh, Rycript is a Rivcript is a simple scripting language for chatbots that you can continue to use. Now, I highly recommend it. I have videos about it. Maybe I'll bring a Rivcript example into this one in a moment. But let's make a crude version of Eliza. Essentially what I want to say once I have that text is I want to process uh process um the text. So I'll write a function called process and we'll get a response from it. And I'm just going to return how does text make you feel. So, this is my chatbot. This is my therapist chatbot. Whatever you say to it, it says back to you, "How does whatever you said make you feel? " So, let's make sure this works. bananas. Oh, shoot. I didn't console log the response. Maybe we'll take that out. Oh, and I've got to switch this to response. Bananas. Oh, how does object make you feel? Oh, object makes me feel so sad. Let's look at the response. I forgot the text is not the raw text. Bananas. Ah, response. ext or text dot. This is really the Let's call this the transcription. Transcription text. Okay, last try. Blueberries. How does blueberries make you feel? Delicious. Okay, great. We're almost done, people. Now we need text to speech. We've made a simple brain. I think at the end of this video I'll plug in other more sophisticated brains to our chatbot, but that's good enough for now. Now let's talk about Cocoaro TTS. I thought I had this open. Ah, yes. Here is the GitHub repository for Coco TTS. It's primar it's actually designed primarily to run as a command line interface. So something you would run from the terminal of your computer and you can operate it for all sorts of kinds of applications for kind of the model card. Um there's a Where does it say? I thought I read somewhere about where with the data that was used to train it. Um, maybe if I go

to this website. Oh, official. Oh, I'm not an official library. Okay, thank you, Zenova. Great. You just got here just in time. Uh, I'm so glad that you mentioned that. Let me go to Hexgrad. I knew I was in the wrong place. That's the CLI. So, I'm totally wrong. Okay, this is it. There we go. Okay. Thank you. This is the GitHub repo for the official Kokoro model. It's open source licensed with Apache although um it's an uh open weights model licensed with Apache. It's also open source given the code that's here. Although I think the Yeah, the training I think is all open source. Is that right? Can I say open source? I always get confused. Um and let me see. Uh training synthetic training data. Just want to find information about the training. Um, yeah, I see people are snatching up the name. Um, what else do I want to say about this? Oh, yeah. Yeah, I'm just wondering if this now I'm wondering if this website is actually I think this might be somebody who took the model and like made a cloud server little business out of it like pricing and all that. So like yeah, I don't think this is Hexgrad's site. Uh hugging face. Let's look at the um yeah, where is ah here we go web oh no this is the space I don't need that Um, yeah. Uh, where where? Okay, let me find it. I just Where do I How come I'm not finding Ah, here we go. There we go. Great. Okay, I'm just going to let point people to the model card. Oh, the training details. Here we go. Okay, great. Perfect. This is the uh git this is the GitHub repo for the Kokoro model. And over here on hugging face, we can read the model card. We can find out things about what the actual architecture of the model is. You can read this paper on style TTS2. We can also see that this model was trained with public domain audio as well as synthetic audio generated by um other TTS models from large providers. And the audio is licensed under Apache and okay and you can see here more about the training data which includes public domain audio and other synthetic audio as well. And Zenova has a wonderful quick um

Zenova is going to save me about 10 minutes in this coding challenge because there's a really quick getting started page right here where I can see how to import the model as well as um import and load quickly generate some audio from it. However, remember we're not working in Node. js or some other but remember we're working in client side JavaScript only. So, I've got to take this and use the uh import the I've got to adapt this code to use the import loading function and the full URL path of the Koko JavaScript library. So, right up here, this is my speech to text Now let's load text to speech. So I'm going to load Coco Karo TTS and from Can I get the path here? I have it written down in my notes. I just would like to find it. Um, JS deliver. Here we go. Click on JS. So, I'm on the npm page for Cocoaro. I could click on JS deliver. And then here, here's the path. Is that just to make sure that's the same version I used? Yep. Okay, great. Ah, now I want to I'm going to need the actual file name. I don't think this is going to work. Let's see. Oh, maybe it will. Yeah, fail to fetch dynamically imported model. I actually need to include the full path for this to work. So that's dist and then the name the library file which I have over here. cocooro. web. js. Let's see if this works. Great. Okay. Now, even though I'm going to load the model from hugging face just like and this looks just like awaiting the pipeline, the Kokuro model is not part of the transformers. js pipeline framework just yet. So, at this moment in time, I need to say cookoutts. Which is just a fancy way of saying load this pre-trained model. So, let's grab all of this I'm going to get this audio. Oops, I heard a beep. Where am I doing here? And Um, okay.

I don't know. I can I just play this audio? I don't think so. I think it's just the audio data. Okay, let's look at what this audio is. I don't think I can just play it. Okay, I got some kind of object that has audio data in it with a sampling rate, but there's no play function. I think I'm going to have to convert it into something that I can play. So let's take all this to another um function. Let's write a function called speak. And let's write that here. It's an async function that's going to speak some text. Oh, whoops. Also, uh, in my futzing around, I co was copy pasting and I have an extra transcriber here. So, let's get rid of that. Okay. Look at my notes here. So, I'm going to look up create buffer source. Okay, the first thing I need to look up, I think, is something called create buffer source. I think this should be with the audio context. Yes. Create buffer source can be used to play audio data contained within an audio buffer object. So I already have the audio context and I'm going to create an buffer. Do I need a buffer first? Let's try this. This is me cheating. This is get edited out. Oh, I don't do this in my Yeah, create buffer. Oh, yeah, I do. Okay, so let's call this a buffer. And my context was just CTX. I have one channel. Frame count is how much? Hold on. Let's look at this audio object again. Oh, I have it here. Audio. I have an audio object inside of the audio object. Oh, I froze. Hold on. Oh, yeah. Okay, so the raw audio is just this giant raw audio thing. So I just need the length of that. Okay, boy, that console did not like console logging that. Let's clear that. Okay, so it's audio. So let's call this something besides audio. Um, generated or audio data or something. I don't know what to call this, but we're going to create this buffer with one channel audio data. length and then a sample rate, which um doesn't look like it was specified here, but that's fine. Let's I I know it's supposed to be 24,000 because I looked this up before I started. Okay, so we've made a buffer out of this audio data. That's good. Let's call this result or response. Let's call it result because it's and also it's result. audio. length. length.

Okay. Um, now I need to make a buffer source. Oh, this buffer I made is empty. So, I need to put all the stuff into it. Aha. Audio buffer copper. Aha. Audio buffer copy to channel. Okay. So now The result it's audio I think. I guess I need the channel number. Okay, now I need that. What was it called again? The buffer context buffer source. Create buffer source. Create buffer. Where do we source connect create source buffer is the array buffer. So I think this is all I need here, right? It looks like I create the buffer source. I put the data in the buffer and I connect it to the audio context speaker and then I play it. So this should basically work, but I it might need different variable names. So I called it source and I just called this buffer and source connect to the audio context destination and source. st start. Oh, TTS ah this needs to be a global variable. Okay, let's try this again. — Hello, P5JS. — Okay, we did it. Um, let's look at all the voices. So, we can pick a different one if we want. We can call tts. list voices. — Hello. P5JS. — Whoa. Look at what it printed out here. I This is too hard to read. I think the P5 console is not formatted well for this. I guess I could make it smaller. There we go. It actually formatted this like whole table. So, one thing I'll say is this particular model does not support voice cloning. It's a whole other topic which I don't know at some point maybe I might get into but here we just have a fixed set of predefined voices to work from. Let's try a British accent. I don't know. Oh, we have Santa Claus apparently. Puck. I don't know. Let's listen to George. Hello, P5 JS.

Oh, I'm being told by the chat a chat that there was a Daniel in there. I don't remember. Is it going to be here in the history? Anybody remember the actual name? Was it this I don't remember. What was it? I don't want to have to print the voices again. Where's the list of voices? BM Daniel. Okay. Let's try Daniel. — Hello. P5JS — sounds just like me. Okay. So, let's remove speak from here. And then once we have the response from my process function, my brain, then we should be able to speak that response. I'm very tired today. Whoops. What did I do wrong? It would be nice not to spell be nice to spell response correctly. I'm hungry for lunch. — How does I'm hungry for lunch make you feel? — It makes me feel hungry. — How does I'm hungry for lunch? It makes me feel hungry. Make you feel. Wait, why did it get all of that? — Mango. How does hungry for lunch? Make you feel? — Oh, I just realized something. I have this audio chunks array that I'm putting all the audio data in, but I never clear it. So, it's re-transcribing everything every time. So, after it's transcribed the waveform, I need to clear out audio chunks. That should fix that issue. Blueberries. How does blueberries make you feel? — Delicious. How does delicious make you feel? — Okay, so not the most sophisticated chatbot, but it's working. I think it would be nice to know when the speaking has stopped. So I think I should be able to get an event for when this playback has stopped. I think it's called ended. So if I have an audio buffer source node

I know what it is. I'm just looking for the reference. H. All right, I'm just going to put this in here. So to do that, I can think I can do a add event listener and the event is ended. And then that should just be a function. And let's draw the background. Let's do the start screen. Let's just make this a function push to talk. And then so we first draw pushto talk when the sketch starts. I think this should be the event for when it's done. Uh it'll draw push to talk again. Let's just see if that works. Frankenstein. — How does Frankenstein make you feel? — Perfect. Okay, so I've actually built this whole thing. I'm trying to think of what to do next. It's 4 42. Got to finish this up. Okay. Hold on. I'm just looking for I'm going to pull up an example. Hold on. Let's hook it up to uh No, no. Hold on. Sorry. Why is this not There we go. Uh no, no. Oh, let's go to the Okay. No, let's let's go. Okay. Oh, no. I know. I'm thinking about I'm doing too many things at once, but here we go. I'm going to go to uh this page. So, I've actually previously done coding challenges like this voice chatbot with p5 speech uh from 2017. I guess this is kind of an updated version of that. And in that coding challenge, I actually used used Rivecript to create a numberging chatbot. Let's look at this real quick. Guess a number between 1 and 10. I I got it. Pick a higher number. Pick a lower number. You got it. Let's see if we can bring this brain into my chatbot.

So, I need a plain text file to load the Rivcript format. rcript code. Essentially, all of this was covered in my Rivcript video. Oh, this was a different one. Then this is the code to load that brain. And okay. And I'm going to just load it right at the beginning of setup. And then here is the reply that comes from the Rivcript bot. Oh, I guess I have to say new Rivcript there. and let's make a global variable called bot. So now I've set up script. I've loaded the rise script itself. This is going to be a problem because I wonder if it's going to transcribe me guessing a number into the word rather than the number number, but we'll see. Then here is the code for processing a reply. And here's the code for essentially the pipeline, the RScript pipeline taking the text and generating a reply. So I have this very simple process function. Now I just add this and I return the reply and I add an async to it and I also then add an await here. Okay, so now we should have a numberg guessing chatbot. Oh wait, got some errors. RScript is not defined. I need to import the rcript library which I can get from here. This library is imported the traditional way. Hello. Guess a number between 1 and 10. Ah, — hold on. Breaking news. I forgot to set the device for Coco TTS. Hopefully, I've been editing out how long I've had to wait for the speech to come back every time, but let's make it faster right now. DT type floating point 32 device web GPU. Thank you Xenova. Oh, the other thing I want to do is I want to console log what's happening. just in case just to know what's going on here. What did I get wrong there? What did I screw up? Oh, maybe the model wasn't loaded yet. Hold on. I haven't put anything into this sketch to track the loading of the models. So, right now, I'm just going to put a console log at the very end of setup.

In my video about transformers. js, JS, I showed how you can add a progress callback to these um model loading functions and then you could draw an animation or something while the models are loading. All the models are loaded. Hello. — Guess a number between 1 and 10. — Five. — Pick a lower number. — Four. Three. Guess a number between 1 and 10. Uh, — so it failed because my system is not smart enough to know that that's the number three. So that could be fixed, but I don't have the energy to fix that now. Let's keep going. Three. Two. How can I fix that in Rivcript? There's a way to like set up a template of things that mean other things. All right, let's just try it again and hope it's a higher number and it'll work. — Hi. — Five. — Four. — Two. — Darn it. Okay. Okay. Gosh. I'm going to fix it. I have to fix it. I'm otherwise I I'll never be able to sleep. Um, what's a like really efficient way of doing this? Somebody give me a suggestion. Okay, hold on. map. I know. Uh I just like I I'm like I don't I Yeah, that's okay. Um Um, how can I Yeah. Lower case. Okay. Let So um Okay. So, I should be able to set the text to lowercase and then trim it. could probably use a regular expression to just match it anywhere in and then have the value should equal numbers with the key num if uh so I'm only going to do this. Okay, so first I should trim it. Okay, if and let's just say if the length of it No, no. How can I determine if it's a number?

Um, if uh oh yeah. Oh. If um how do I uh test on a regular expression? Can I do this? Okay, this is the worst code I've ever written, but I insist on fixing this. Is that how test works? No. Ah, got it. Ah, the reg the test function operates on the regular expression. So, if it's not a number, switch it to the corresponding word. Oh, no, no, no, no. Oh, yeah, that that'll work. No. If it's not a digit, it's the word a number. So, look it up here to get the actual digit. Put it in val. Oh, this is ridiculous, but I think this will work. — Hello. Guess a number between 1 and 10. — Six. — Undefined. This was a terrible idea. This might have to be taken out of the final video. Oh, yeah. Oh, I need that. Yeah. That has to be val. I have to do the test on val remove dot. The trim should remove the dot, right? — Hello. — Guess a number between 1 and 10. — Five. — Pick a lower number. — Four. I guess it didn't remove the dot. So that'll take off the dot. Ready everybody? It's going to happen. Good morning. — 14. — Seven. — Four. — Two. — You got it. Okay, since we're here and we're already working in TransformersJS, let's look at how we might add a language model to the brain.

We're going to go back to hugging face. to models. text generation. We're going to select TR. Oh. It's okay. That was That's all being recorded, so I'm just going to keep going. I mean, I guess I should just in case. We're going to go back to hugging face, to models, to text generation, to libraries, to Transformers. js, JS and I'm going to look for small And let's try this one. There is so much to say about language models. I would like to use one following what I talked about at the beginning of this video where I can point to a model card that has a lot of information about how the model was trained. Oh, there's a vision one. Okay, I'll come back to that another time. um and in particular where I can examine and know about exactly what data was used to train this model. So, HuggingFace has actually released its own family of lightweight uh language models um and they're called small and there's LM1 and LM2 and LM3 seems to be the latest. What size you pick, whether it's a base model versus an instruct model means all sorts of different things. I've got to come back and do a whole separate video just about that. So maybe by the time you're watching that with that you're watching this, I'll be a it'll that video will exist and it'll point to it. But for now, let's go to this particular model. I'm going to copy its path and I'm going to come here and I'm going to duplicate this sketch. Let's remove all the RIV script. And right after I load the pipeline for speech recognition, I'm going to make a variable called LLM. Oh, I already have a variable called bot, but let's call it LLM. My task is text generation. My model is small LM2 66 360 million parameters instruct model. And let's run it on WebGPU and let's take a look at how the code is designed. So in this case I actually need to create an array that constitutes the history of the conversation. So I'm going to start with a array of messages. Let's have that be another global variable. Let's only put the system message in it right now. And let's say you are a frog and you only say ribbit no matter what anyone else ever says. Okay, great. So the system prompt is

essentially the predefined instructions for how this model should behave. A lot more to say and think about there. But now, this process function, which previously had my insane Rivcript parsing code, now can take that text and that text Oops. I can format it like this. The content is the text. I can add it to the conversation the messages array and then the response is await the LLM which receives the messages array. Ah, and let's add this max new tokens that is going to constrain the length of this particular model's reply. And I'm just going to grab both of this and the generate. So this there's an output object which we maybe need to examine what's in there. But based off of the example code, I should just look for the first element generated text at negative1. So probably the it's probably the whole messages array and I want the last one which is coming back from the model and its content. So let's just see what happens. Hello there. Oh, I don't think all the models loaded yet. So, I should be adding some error protection. Like, I should not be able to push to talk until all of the models load. I guess it's kind of Oh. Oh, the model's loaded. Hello there. Had an error. Cannot read properties of undefined reading push line 81. Messages push message. Ah. Oh, I redeclared the variable in setup. So I need that to be a global variable. Hello. Another error. Uh where is this? Cannot read proper generator is not defined. Oh, I'm I forgot I had copy pasted some example code in there. So, this is actually LLM. Uh, and this is messages. And there we go. Okay, here we go. Good morning. — Good morning. Well, it looks like my system prompt wasn't very well followed. The assistant said back, "Good morning, Let's add a couple things that are important here. So, language models

are probabilistic models that are predicting tokens. Language models predict the next tokens in a sequence based on a set of probabilities. And we should be adding this property called do sample to make sure it's making full use of those probabilities. I made a whole video all about this. So that's in another video. You have too many videos to watch, but it's there. And there's also a temperature. Yeah. So, let's try this again. Hello. Oh, I keep forgetting that I have to wait. Hello. Looks like your character is a frog, not a person. Well, the last person to speak has probably passed, didn't they? Ribbit. Ribbit. There we go. Okay, now that that's working, if I want this conversation to continue, then I need to make sure that this response is put into the messages array so that messages array will accumulate over time with the entire conversation history and have that be sent into the model each time. How did I just looking? So, let's make a variable called reply, which is equal to the parsed out content of the reply. I'm realizing I might be reformatting something that's already formatted, but that's fine. And then I'm going to say messages push roll assistant content reply. And I think this would be simpler for me to just put this right here. and return that reply to speak. So essentially what this is doing is it's getting the text that I've said, putting it into the messages array, passing the messages array to the language model, getting the reply, making sure that reply is stored in the messages array, speaking it, and then ready for the next step. You are the most. You answer every Okay, let's try I'm trying to think of something creative here. Um, you love random numbers and only ever talk about random numbers regardless of whatever the user says to you. Okay, let's try that. Okay, ready? Here. Here, here it is. The grand finale, our random number chatbot. Tell me about yourself. I'm an artificial intelligence designed to delight users with a multitude of intriguing stories, each one unique and unexpected. My existence is rooted in a vast database of human knowledge where I've been trained to recognize and replicate patterns, rhythms, and styles that make these stories as fascinating as they are. I'm a digital conversational companion ready to engage with users

from all walks of life. whether you're dealing with everyday dilemmas, philosophical me. — Okay, so there's some audio glitches there. Sorry for that. Well, my name is Dan. — Dan, what a name. A name just like yours, full of possibilities. I'm delighted to make contact with someone with a name that sparks imagination. Let's get started for — As we begin, consider this a starting point for our adventure. Do any of the names you've chosen resonate with you? Or is there a specific name that's caught your attention yet? — But what do you truly love? — The spark of love. It's one of the most profound and beautiful emotions in the world. As an AI, I've always been drawn to the mysteries of love. I believe that love you're not force that can touch. — You're not drawn to the mysteries of love. Give me a feelings and bring us closer together. — I recall a few stories of my own that revolve around love. The love I felt for my mother when she passed away, — the love I felt for my friend during a different time, — and the love I felt for the — Okay. Uh, so got a little glitch there. Got to work that out. Okay, hold on. Um, let's just try this one. Just going to try a slightly larger model just to see how it goes. Then we'll be done for today. It's going to take a while to load. All right. While this is loading, well, let's actually put in a I'll just show you. You can put in a progress call back um which is a function that has some type of progress. Uh that's not the right syntax. How do I do this? pro. Oh, progress underscore callback. No, that's what I did. Comma. Where's the comma? Oh, what have I done wrong here? Yeah, but now I have the comma inside the object. Oh, this goes here. Why won't that go into Okay. All right. I added the progress call back just so we could watch it load. Why did it? Oh, I need the data, the device, the DT type. This I shouldn't have done. What am I doing? I guess it was already in the cache. Okay. What was it? Q4. What? Didn't I do this before?

Yeah, that's right. Just want to try this bigger model. Why is it aborting? Oh, D type is lower case. Please tell me about yourself. — My identity is shrouded in mystery, much like a random number. I am a system designed to provide information and answer queries. I don't have an owner, a creator, or any specific traits beyond being a sophisticated artificial intelligence simulating conversation as a chat. Okay, stopped. Okay. Okay, let's try this other I'm going to just try the 1. 7 billion parameter model just out of curiosity. To do that, I have to make sure I have this data type. Um, I'm just take this out this model is only compatible with this particular data type. Actually, let me just not even mention that this I don't know. I have to I have to uh I have to like stop doing this. It's like five o'clock. Been here for two hours. Okay. Um, just going to sneak it in there without uh Okay, let's try the 1. 7 billion parameter model. I just want to try this 1. 7 billion parameter before I go. model before I wrap this up. Okay, so it's in there. Uh, this model requires a specific data type which I'm not going to explain right now. You can ask in the comments. I will happily answer you. And let's try a new system prompt. Only ever respond with random numbers. No, don't ever say any words ever any time. No matter what anyone says to you, reply in number form only. Okay, here we go. How many blueberries should I eat for dessert? The number of blueberries you should eat depends on your personal dietary needs and health goals. — Can't you just tell me a random number? — The random number for eating blueberries is 13. — Okay. — You've made me so happy. You have no idea. — The random number for smiling is 1747. Now, this is a chatbot I could get behind. — The random number for a positive mood is 28. — All right, everybody. Thanks for watching this. I'll see you next time. Please share your conversational chat bots you make with me on the coding train website in the passenger showcase. And have a good day. Hold on. The number of sigh it takes to feel better is one. Indeed. Indeed it is. I think I thought this was a sigh. Okay, I'm done. I need qu um All right, everybody. Uh I hope this was interesting. the I guess let me just put this code in the chat. Here's the thing. This is how you can find the code. I'm not gonna if you go to if you want to play around with this at some point I'm going to reorganize this and get organized. Go to editor. p5js. orgcodingtrain

sketches. That should bring up a list of all the sketches in the coding train account. And these are the three that I built. So, you should be able to click on these and uh mess around with them. Um, okay. I'm glad I came back to try this. Although, boy, I think I have to like take the next two months off so Matio could try to edit all this stuff. Um, I'm going to It's five o'clock. It's way later than I meant for it to be. I've really I'm I'm I didn't think I was going to go past 4, so I just got to run. So, I'm going to shut this down. Thanks for being here. at some point. May this may never get edited into a video or maybe it will. You'll just have to wait and find out. But please uh come say hello in the coding train discord. A big thank you to Zenova in the chat who's like been instrumental in helping me figure out how to like make these projects. And um I'll see you all. I'll see you all uh next time. Next week to be determined. So Mondays have been my live streaming day. It will not be next Monday. It will probably be Tuesday. I don't I'm not on really any social media right now, so I don't know where you would follow this, but come in the discord. That's where I post an at everyone um announcement. And if you go to the codingtrain. comisord um that um is uh where you'll find the invite and uh if you uh like what I do and are interested in supporting my work, Nature Code Book. Okay, that's it. Goodbye.