[News] Facebook's Real-Time TTS system runs on CPUs only!

hi there check this out modern text-to-speech systems have come a long way in using neural networks to mimic the nuances of human voice to generate human-like audio one second of speech can require a TTS system to output as many as twenty four thousand samples sometimes even more the size and complexity of state-of-the-art models require massive computation which often needs to run on GPUs or other specialized Hardware so this is generated by a system that Facebook AI has built and we're going to look at it it's called a highly efficient real-time text-to-speech system develop load on CPUs there's a lot to unpack here but this is not a paper this is basically a technical blog post because I think they built this into their products and it's mainly explaining on a high level what they did so they have a real-time text-to-speech system which means that

you have a you know a text like the sentence here share on Facebook you give it to the system and the system comes up with a sound wave right a sound wave that says this sentence so if you listen to it you'll hear share on Facebook and it has to do that a in a you know let's say credible way such that it is a human-like voice because people like hearing that you know not these kind of robot old-school telephone robot voices where they just chunk together words it has to flow naturally so what you want to do for this is you want to have some sort of recurrent neural network or any sort of autoregressive Network that outputs basically these samples here one at a time so these points you're going to output one at a time and this for once they say for one second of audio it can require you to output two or 24,000 of these data points right so 24,000 for word propagations of your auto regressive model that's massive and if you want to do it in real time and that's why real time is so impressive is you have to do this in less than one second they have to do this 2,400 forward passes sorry 24,000 forward passes in less than a second okay and even more so this was already possible I think but it required you basically required a big data center with many many GPUs in it so you basically would send your text to this and it would stream back the audio and they can do this just on CPUs and in fact they can do this on a quad-core CPU in real time in they can generate this many samples in half a second pretty impressive so let's dive into how they do it and they say it's deployed in portal our video calling service and available for use across a range of other Facebook Applications from reading a support for the visually impaired to visual reality experiences okay

so first they show this graphic right here what are they doing so this is their entire system it is chunked in multiple parts if you're a deep learning practitioner you're very keen on taking this text and just like giant neural network and just run it through and generate audio and to end right this doesn't really work in this case first of all because it would be too much of a to too many parameters to evaluate this many times but also especially text and audio are such different modalities that you'll have to basically chunk this into individual parts and that's what they do so they have this linguistic front-end and the linguistic front and generates two different things it generates what needs to be said and how does it need to be said so what needs to be said they call this linguistic feature the linguistic features there things like phonemes and so on so that's sorry they don't even have a title for this one so the linguistic front-end converts the input text so this is probably it's a sentence by sentence thing so this is one sentence of text into a sequence of linguistic features such as phonemes and sentence type okay so these linguistic features are if it's like share on Facebook it would be like okay Shh is one and then a and then these are the phonemes of share right it would kind of chunk it into that so that is much closer to what we think of an audio signal this will make one sound this will make a sound and so we sort of chunk it into that and then here the how is it said this would be for example the fact that share on Facebook is sort of an instruction right and this is different from if this would be a question like if it said do you want to share this on Facebook then the what would output much the same phonemes right here except it's different words now but the how would output a this is a question so as you can see the information flow right here informs the later stages this would then cause at the end of the sentence the voice to go up because it's a question right so it is this linguistic front-end as you can see it still deals with text it deals with text and it outputs these how features and these linguistic these what features so the what features now go into an acoustic model and what does the acoustic model do the acoustic model is meant to generate a spectrum a spectrogram of the sound now we'll skip this for the moment and go to the neural vocoder is a kind of a standard thing in text in any speech producing it takes a spectrogram of the sound and turns it into actual audio so this here I think they achieve it with they say something similar like a wave RNN based on plus like a CNN so we'll look into that quickly so the spectrum the spectrogram of the sound is going to be a bit of an image and the image has time on the this axis and frequency on this axis and then there is a it's usually some somewhat like color-coded but it's just intensity right so whenever there is whenever a frequency at a given time is expressed strongly it will light up so it could be something like this so over time the mid frequency is always there but this frequency right here is not there and this is here at the beginning so there's a sort of a way to read this spectrogram but this represents maybe something like this represents the not even sure how much sound this represents but this can represent maybe something like 200 milliseconds of audio and you have to basically perform a Fourier transform to transform that into 200 milliseconds of actual WAV audio but since the audio has to be output at what is 24,000 samples per second what's coming in here is not that much the acoustic model outputs not 24,000 spectrums per second so first of all this has to be up sampled so this time dimension here has just too few samples so we use the CNN because this is basically an image in order to up sample this in order to make a long image out of this and be basically the CNN will have to impute it will have to this is learned right this is learned how this spectrogram would look if it were sampled much more densely and now it has basically the correct number of samples here and now this way Voronin can step through it and look at these slices and look at the last slices or actually also can spectrograms maybe I don't know they don't really say what the RNN exactly goes over but this is how I imagine it and the wave or Ihnen is based on a wave net architecture and that means wave net is sort of like if you look to produce this thing right here you can look back all the way to the beginning but you this would be too many connections right it will be too much memory so what you do is you can look back at the directions at the things right before you in very great detail but then as you go back further and further you basically lose detail see there's only two connections here within this long stretch where there is many connections here at the very beginning so right before you that means you look in more detail what you've produced recently but you also sort of look back in a more blurry way at where you've produced a long time ago so this is a wave net architecture and that's they say they use something like this in order to then actually produce the final audio from the spectrograms so actually a neural vocoder you can train this thing you can train it by itself right you simply feed spectrums and you make it go audio and you know there's a lot of audio on the Internet you can simply produce spectrograms from that and then train the vocoder to produce the audio so the good thing about this pipeline here is you can train a lot of these things independently from each other I don't actually know whether they do that or not but you can you see this box up here this prosody model and that will take in these how features so how does something need to be said how come on well this is an H how does something need to be said and it will transform it into features that the sort of neural network can understand these neural networks they need features like they need embeddings and as you can see here it also takes into account the speaker embedding which is not only how what the sentence is the fact that the sentence is a question you would also get the information that the speaker should be you know kind of calm sorry that would be the style the speaker should be maybe a woman voice and then the language should be it should have like an English sound or a German sound and so on so this model here will take in all of that and omit features that these neural networks here can understand so as you see here the neural vocoder not only does it transform spectrograms to audio but it takes into account how you want the audio to sound and so does this acoustic model so the acoustic model is sort of along with the prosody model is a bit of the heart of the thing here it takes in these linguistic features so what needs to be said right the acoustic model it takes into account the again the speaker embedding language embedding and so on it takes into account the output from the prosody model of the how you would like it to be said this includes what type of sentence this is and this it synthesizes at all in order to come up with the spectrum right here these spectrums so this is sort of the heart of the thing okay so about the porosity model the say we use style embeddings that allow us to create new voice styles including assistant soft fast project and formal using only a small amount of additional data with the existing data sets in we don't have to create a separate model for each style we need only 30 to 60 minutes of training data for each voice style so that happens because you take in too you take in actually an embedding of the feet of the speakers and you train these things independently so that means you can sort of generalize to a new style really quickly here the they describe in essence how their acoustic model works the fact that they output 13 dimensional MF CC features concatenated with the fundamental frequency and a five dimensional periodicity feature which is much easier for the acoustic model to generate so that's what the acoustic model generates okay and then here our conditional neural vocoders is the final part of the pipeline consists of two components a convolutional neural network that up samples or expands the input feature vectors from the framerate 200 predictions per second so that's not it's not 200 milliseconds it would be 200 spectrograms a second to the sample rate 24,000 predictions per second and this similar two-way voronin with synthesizes audio samples autoregressive Li that means one sample at a time at 24,000 samples per second crazy right so this all seems like it's a lot of

computation and now they describe how they get the this to run faster than real-time and they list their individual contributions here so if they just run this on one CPU core you see it takes 80 seconds to produce one second of audio then they do optimized inference operators which means they basically use a PI torch get along with some cities PI torch these PI torch it which is kind of where you can sort of compile your deep learning model to an optimized form that runs faster and they also customize this so they get to 20 seconds per one second then they do parameter reduction by sparsa fication and they are able to abuse the sparse matrix compute operator I think they implemented a custom one to do that so this here is very much like if you have a neural network and that's somehow connected connected you want to train it in such a way that it's sparse meaning that only very few of these connections have nonzero weights and because of that you don't have to store the nonzero weights and you don't also have to compute because something multiplied by zero is going to be zero so you don't have to compute that so there that they achieve sparsity 60 96 percent right and this basically you do by some sort of teacher teacher-student model or there are many ways to do this but you can sparsity regularize a neural network and basically force most of the connections to be sparse while still maintaining a good training error or a good generalization error so they supercharge this and with this sparsity they bring this down to v s/s and then they go further and do block wise specification and distillation so they distill it to an entirely smaller model that then is also clockwise sparse so they also enforce this block wise specification and that not only can you then have a better operator they implement this block sparse matrix compute operator that is specifically designed to multiply block sparse matrices they also in the text they describe that it also optimizes cache access so if you know about CPUs they have these level 1 level 2 level 3 Asha's and you can optimize your computations that's what libraries like la park and blas do you can optimize your computations such that your cache access is optimized and therefore you can speed up a lot your computations because the amount of times that you actually have to go to your RAM and retrieve something is minimized and that tends to be the slow part of the process when your cache misses so again they achieve 94% blocks parsecity and that almost gets them to real-time and it's still one CPU so now they paralyze the operators that are doing the heavy lifting to four CPU cores and that doesn't divide it by four of course because there's an overhead in parallelization and synchronization but that gets you to this one half a second needed to produce one second of audio so there you're at real-time and that is pretty impressive and they go on to describe so in detail how they did it but they also give some examples of what they can do and would like to achieve even better in the future so here is an example where they can adapt their model to a given style and here they have a British accent recently we successfully applied our new approach to create a British accent advice this is the first of more accents and languages to come and also you can adapt it they their idea is that you have sort of an assistant and this assistant will be able to adapt to let's say your mood we're also exploring features to make our voice respond intelligently with different styles of speaking based on the context for example when you're rushing at the door in the morning and need to know the time your assistant would match your hurried pace when you're in a quiet place and you are speaking softly your AI assistant would reply to you in a quiet voice and later when it gets noisy in the kitchen your assistant would switch to a projected voice so you can hear the call from your mom right so this ties in very much with sort of conversational AI so assistants and so on but it also ties into wearables I think so the fact that you are now smaller then real-time means you can run this potentially directly on your smartphone you could run this on your fridge or something on your stove in your car without having to stream basically so you'll get much more real-time on device assistance maybe even in your watch and I'm excited by this technology so far it seems you can get it in Facebook products but I'm sure this will come to places all right if you enjoyed this please consider subscribing thank you for listening and watching leave a like if you liked it and leave your comment if you have something to comment with that bye