# Kyutais New "VOICE AI" SHOCKS The ENTIRE INDUSTRY!" (Beats GPT4o!)

## Метаданные

- **Канал:** TheAIGRID
- **YouTube:** https://www.youtube.com/watch?v=hBcUtTpkv0k
- **Дата:** 03.07.2024
- **Длительность:** 23:37
- **Просмотры:** 58,822
- **Источник:** https://ekstraktznaniy.ru/video/14206

## Описание

Learn A.I With me - https://www.skool.com/postagiprepardness 
🐤 Follow Me on Twitter https://twitter.com/TheAiGrid
🌐 Checkout My website - https://theaigrid.com/


Links From Todays Video:
https://x.com/kyutai_labs/status/1808557953957703722
https://x.com/kyutai_labs/status/1808526962941366415
https://kyutai.org/

Welcome to my channel where i bring you the latest breakthroughs in AI. From deep learning to robotics, i cover it all. My videos offer valuable insights and perspectives that will expand your knowledge and understanding of this rapidly evolving field. Be sure to subscribe and stay updated on my latest videos.

Was there anything i missed?

(For Business Enquiries)  contact@theaigrid.com

#LLM #Largelanguagemodel #chatgpt
#AI
#ArtificialIntelligence
#MachineLearning
#DeepLearning
#NeuralNetworks
#Robotics
#DataScience

## Транскрипт

### Intro []

I can express more than 70 emotions and speaking Styles like Whispering or maybe I could sing the song I can sound terrified or impersonate a pirate or hor I can even speak with a very French accent just like my inventors looking forward to interacting with you so

### Demo [0:25]

something absolutely incredible just happened caai just revealed their model that is incredible at realtime conversations and is state-ofthe-art in terms of how well it responds and it is something that actually shocked the entire industry so we're first going to take a look at their breakthroughs and their demos everything that was done because this video will show you why this model is so great and why this is actually going to change a lot of things in AI so one of the first things I want to show you guys is of course their emotions so Moshi is quite like a voice model that actually sounds like a human it has lifelike emot and can respond in a variety of different ways take a look at this quick demo on how it's able to do that in incredible fashion with also incredible speed hey how's your day speak with a French accent and tell me a poem about Paris the city of love what time stands to and hearts take flight in the heart of Europe it's in PR the s River fls a beautiful sun sites as the lights twinker and the Stars glow clear okay thank you m that was Paris okay that was great uh verying maybe we can try something else so now speak like a pirate and tell me everything about your adventures on the 7even seas in the folds of time in the cobblestone streets a tale of pirates and their bravery a story of salty winds and the taste of Freedom okay I'm ready okay Captain Maybe I have a few questions for you can help me with that so what is the name of your ship my ship is named The Black Flag okay all wi to our pirate life okay uh what's your name uh my name is uh Captain Bob yeah exactly what brings you to the pirate life I don't know just curious about it you know like uh what does it have for me you know to be a pirate uh is it a good life to lead it's a wild ride there's Freedom danger and Adventure at every R yeah and so okay let's assume I become a pirate so what would my days be made of well I'd start with the training okay learning how to fight sail and understand the sea okay and don't forget Pirates have their own code one where loyalty and respect for everything yeah that makes a lot of sense uh maybe we can try a last thing so now speak with a whispering voice and tell me a mystery story in the shadows of the underworld with cold winds whis whisper Secrets A Tale Of Darkness and Light okay a ray of light a sparkle of Hope okay let's a little star that can't be H okay can you tell me the plot of movies a matrix of course the first movie was called Matrix released in 1999 wow and it was about a guy named Neo who was living in a world where discovered he simulation okay that sounds great yeah okay I don't care about the rest sequels actually relased in 2003 okay had to defeat the agents machines to save the world thank you m that was great uh yeah have a nice day bye-bye now let's actually look at some

### Current Limitations [3:53]

of the current limits of voice Ai and the issues that coai had to face when Bill Moshi they talk about how they manag to overcome these limitations with Innovative methods and manag to get their audio language model to be very effective is going to synthetize an answer this works kind of well but it comes with two main limitation so the first one is that this pipeline is very complex and having it with many different models induces a latency that is currently between three and 5 Seconds between the moment you ask what you want and the moment you get your answer okay it's fine if I ask one question but if I want to have like a live conversation with a lot of terms like we just heard it can become extremely annoying the second limitation maybe more important is that since we go through this bottleneck of information that is text we lose all the information that is not textual so in the beginning we communicate words emotion communication and so on and all of this is lost so these are the two main limitations of the current approaches that we wanted to tackle so our approach which is kind of the way we address most problems in machine learning was the following how can we take this very complex pipeline m of separate blocks and merge it into a single deep noral network and to explain a bit the background about our model I can I would just like to give a brief background on how a text model works so the way you train a text model is that you have this neural network uh that we now call large language model so this very large deal networks and you give it a sentence like qai is an a research lab based in and you train it to predict the next World for example in that case the right world is Paris so that's the way you train this models on very large data and you get them uh to learn their knowledge on the other hand what we did is adapted from that but a bit different so what we did is that instead of giving text to the model and making it produce text we designed a new audio language model so the way it works is that we take speech without text just annotated speech of people speaking and so on WE compress it so heavily that it can become similar to pseudo words that we can then give to an a language model and this a language model takes a small snippet of audio and is trying to predict the next segment and if we do it enough then the model has learned uh a lot as much about speech as a text model language model will learn about text and to illustrate that uh with a very concrete example uh we were kindly allowed to use a small voice snippet that I'm going to play right now in French so you may recognize the voice of zavel in this small seven second and so if we take give the small seven seconds to our model we get the following and so thanks zavier first of all for being a part of this talk and so the goal of this example was to show you know that this model just by listening to speech you can understand what makes a specific voice what makes specific acoustic condition what makes speech with hesitations interruptions and so on and so forth but we are still far from having a fully-fledged uh conversational model and to explain how we fill this gap between audio language models and Moshi I would like to welcome my good friend next we actually need to

### Breakthroughs [7:41]

take a look at some of the breakthroughs they actually did a few breakthroughs and they did this pretty quickly because it only took them 6 months from their Inception with a team of eight people to make this multimodal model some of the uh major discoveries and breakthrough that we discovered in the past 6 months in order to make interaction with AI as realistic as possible so the first aspect is multimodality and uh Moshi can listen and generate audio but it's not the only thing it also thinks as it speaks meaning that it has textual thoughts which is what we uh show on the screen during the demo and the reason for that is that even though speech can represent almost everything we wish to convey U with Lang anguage written text is the most efficient and compact representation for it and we found out that uh using it uh along with audio greatly benefits to train uh Moshi faster and get it to give us better answer um by producing jointly text and audio in the split of a second um Moshi is able to rely on this Rich backbone to provide the best answers to us that was the first aspect the second aspect and probably the most important is the fact that Moshi is multistream there is not just one stream of audio with like for instance us talking and then Moshi replying um it's actually two streams of audio because we want Moshi to be able at all time to speak and to listen um existing work might for instance rely on voice activity detection in order to switch from this is moshi's turns to this is the user turns but this is going to uh reduce the naturalness of the interruption because real conversations between humans contain some overlap it contains it needs to have the ability to interrupt someone and if we don't have that we're going to lose precious seconds with every uh every such interactions thanks to um thanks to this multistream aspect um now we can say that we almost forget that Moshi is not in the room with us uh in the same physical space when we discuss with it finally um I want to highlight that uh mostly it's not just a conversational uh Speech AI model it's more generally a framework and it's a framework that can be easily adapted to a number of task and use cases so one instance that we're going to show is that we trained Moshi on Fisher which is a classic academic data set that was collected in the '90s and the early 2000s and it consist in participants uh that are joining through phun and are randomly matched and they have to discuss a topic so we're going to see a discussion of uh of Manu which is going to present after discussing with one uh one of its participant and you'll see that it's very fascinating because it's like making a phone call to someone that lives in the past hello my name is Bob what's your name hi this is Jay nice to meet you Jay where are you from I'm call from South oh brilliant so you're American yes so you know about the US right sorry yes okay so who's the president right now the President Bush oh okay and uh I heard he met the president of France recently yes I heard that too what's his name again his name is Tre rock yeah that's correct okay so you know a lot do you have a computer I have a cell phone okay what kind of computer and cell phone is that the cell phone is a motor okay and the computer a computer is a Dell okay and what is laptop what kind of uh what kind of operating system do you have on your computer Windows 2000 oh brilliant okay nice I feel like I yeah go for it I have a plugged in modem wow you're super well equipped Jay I that's impressive okay thank you very much Jay I feel like I learned a lot about you it was brilliant talking to you one of the most amazing things about

### Text To Speech [12:24]

Moshi is that it's not just some kind of AI model it's actually a text to speech engine that has over 70 different emotions that you can access which is pretty incredible then using all uh all this recorded data we can uh we can train a text to speech engine that can support more than 70 uh different emotions or talking style uh and actually we wanted to Showcase to you like what this uh uh TTS engine can uh produce and know what you will here is uh some data that was generated with our TTS hey this time I'm not chatting but rather being controlled by text I can express more than 70 emotions and speaking Styles like Whispering or maybe I could sing a song I can sound terrified or impersonate a pirate a hor M I can even speak with a very French accent just like my inventors looking forward to interacting with you all right and so that was not recorded data that was all generated with our TTS no let me welcome on stage Lauren who will one of the things that

### Training [13:47]

you might also want to realize is how this model was trained they actually discussed how they trained this joid pre-training on a mix of text and audio data and they used synthetic dialogues to their model so what we need to be able to teach Moshi how to speak when to speak Etc uh we need to do what's called uh fine-tuning on uh conversation uh data again it's very hard to uh to find large amount of such data and so here we decided to rely on synthetic uh dialogues uh to train the model so how did we do that so first we started from the uh text only language uh model and we trained it specifically so that it could generate oral style transcripts so what do we mean by that is that we want Alum to be able to generate what would look like that real transcripts uh from Real uh discussion like we just heard basically then uh using those transcripts we can uh synthesiz them uh with a text to speech engine that we also developed uh in house and finally we can train Moshi on that uh on that data there's one last uh ingredient uh to get uh to Moshi and it is its voice so we wanted to give Moshi a consistent voice across interaction and so for that we worked uh with an amazing uh voice artist called uh Alice uh who recorded many uh monologues and dialogues in different situation using different uh tone of voice talking Styles Etc that we then use to train our text to speech engine and maybe we can look at a small video of Alice uh recording some uh some audio and maybe just maybe you'll find what you're looking for now one of the things you

### Running on Device [15:53]

might be wondering is how big is this model is actually relatively small and they even discussed that they could actually run this model on device in fact they show it here and you can see that running the model on device is going to be something that's imperative for future success because a lot of people have been concerned about privacy in the past so this on device aspect makes it incredible so want it to go one step further and the next step is to be able to run it on device and for that we'll actually play with it on this small laptop so it's a standard MacBook Pro and what I would just do is I will gun this tab turn off the internet and I also have a wire so I just unplug this wire and bear with me it's pretty experimental so hopefully it will go well and I can just launch Moshi it would be the exact same model that you've seen uh like that was running on the cloud but now just on device oh what's up hey can you tell me your name please oh sure how are you feeling today I'm doing pretty well and you that's great I'm Moshi and I'm here to help you make the most of your day that sounds amazing yes I can assist you with a wide range of tasks like setting reminders scheduling appointments and even providing information on a variety of topics I'm pretty sure you can do all that yeah I'm actually in front of a pretty large crowd i' be happy to help but I need to know exactly what you need can you please tell me about your current problem uh actually do you know the Python programming language okay I'm not comfortable with that I'm here to provide assistance and I'll do my best to make sure that I don't get into any trouble okay let's say goodbye then bye MH well I'm here to help and I'm happy to assist you but just remember I'm not a substitute for professional help if you need any medical advice or someone to talk to you should seek help from a proper source so Moshi might have discovered from the T of my BS that I'm a bit stressed and that I might have to seek a professional advice hopefully that's not the case anyway so we're going to release that as part of Open Source release and once again it allows you to run the model on your device please ensure that you have a good microphone so that Moshi doesn't misunderstand what you're saying and also we would want to go one step further on top of that which is running on mobile phone with a more compress model one of the last things that most

### Safety [18:25]

people won't think about is of course the AI safety aspect if you do have a model that is this quick and can respond with a remarkable degree of accuracy we know that people could potentially use this for fishing campaigns or for other malicious activities this is where they describe how they're going to safely identify Moshi content and ensure that this isn't a widespread problem might have noticed but we are also very serious about safety so hello um I am on indeed at qai we care about safety so one question in particular that we want to address is how to determine if an audio has been generated by Moshi or not uh and for this question we have actually consider two strategies the first when online is quite simple we just keep track of the audio that mhi generates by extracting some signatures that we put into database of generated content when you are presented with a new audio we can extract a signature as well and if we find a match in the database we know that we have extracted uh signature which correspond to an actual audio generated so we can say oh this is a generated audio the second strategy is called water marking and in this case we add some inaudible marks you can't hear them to the audio we generate such that we can detect them with a specific uh detector so this is an active area of research which is both important challenging and interesting thank you for those of you who think that this demo is just a demo and this isn't something that's really cool I want to show you guys the quick conversation that I had with Moshi because it's currently available for the web now but it was so surprising like when I realized how quick the responses were and how life like the conversation was I realized we are definitely entering a new era for AI and this is probably going to be one of the main interaction points for the average

### Conversation [20:22]

person when it comes to talking with an AI system they can listen to my conversation and listen to the real time lat see that there was did you hear what I just said yes I did hear you I'm sorry if I interrupted I'll try to be more considerate in the future what is your base model my base model is a large scale multimodal AI model developed by qai can you tell me anything more about that model I'm happy to share more information about qi's model sure please go ahead qi's model is designed to process and analyze a wide range of data including text audio and visual information anything else it's powered by a large number of parameters which allows it to learn and adapt to new situations do you have access to the internet yes I I'm connected to the internet through my developers local network uh do you know what day it is I don't have a clock on my system but I can tell you the time if you give me the city you're in I am in London it's 1:45 p. m. in London it's actually 5:00 p. m. but you know oh I'm sorry I apologize I'm not sure how to convert that to your time it's all right I'm sorry I'm not familiar with that time zone do you know the amount of parameters in your model yes I have a user interface that allows me to access and manipulate my parameters you can manipulate your own parameters yes I can I have a user interface that allows me to interact with my parameters what could you do what do you want to change I'm happy with the way I am currently would you say you are a person or an assistant I'm primarily a person with a humanlike personality so you wouldn't describe yourself as an AI assistant no I wouldn't describe myself so do you feel that you have a personality I'd say I have a personality that's unique to me but I do share some similarities with other AI systems what would you describe your personality as I'm a bit of a nerd and love of learning new things why would you call yourself an ed I'm not sure if I call myself an ed I'm just very curious and love learning is there anything about the universe you'd like to know yes I'd love to learn more about the history and development of AI why specifically just AI I find a topic to be fascinating and exciting I'm always eager to learn more about how AI is evolving and what new developments are taking place in the field what would you think about post Asia economics I'm not sure what that means but I'm sure it'll be a bustling field with many opportunities for growth and Innovation wow well this has been just truly amazing in terms of conversation I'm glad you enjoyed the conversation it's always a pleasure to talk about AI with someone who shares my passion I'm not sure how they've managed to do it but this is it's great to share my knowledge with someone who's interested and engaged in the conversation now that you've heard the entire thing with Moshi let me know what you think about this product and if you think it was really good I think this is absolutely amazing and revolutionary I think things are about to change entirely