What is Natural Language Processing?

Ellen Turing argued in the 1950s that when a machine is able to converse with a human being seamlessly, that would be the point to stop questioning its intelligence. Since then, computers have come a long way to achieve this goal. Mainly thanks to the advancements in the field of natural language processing or NLP for short. Today, we will learn about what NLP is, why it is a problem worth paying attention to, and what are the latest techniques in tackling it.

All right. So the first question is what is natural language processing? Well, NLP is basically the practice of trying to make sense out of naturally spoken language and many times it's in the form of text. Under the umbrella of natural language processing, we have many different techniques and some sub goals. Some examples are tokenization. Tokenization is a way of separating a piece of text into smaller units called tokens. Tokens can either be words, subwords, or even characters. Another example of a technique is stemming or lemitization. Both in stemming and leatization, our goal is to reduce a word to its root form. The only difference is they have a little bit of a different approach. Word sense disambiguation. This technique helps us understand in what meaning a word is used in a sentence using the context of the sentence. Named entity recognition. Named entity recognition helps us understand a unique entity that is mentioned in a given sentence or text. On top of these relatively simpler tasks, we also have more complex subgoals of NLP. Some of these are text classification. This technique is a bit of a higher level task that gets a piece of text and classifies it into the relevant group. Machine translation. As you can understand from the name, it is the goal of translating a piece of text from one language to another language. Content moderation. It is the goal of filtering out anything that is not appropriate for a certain group. For example, if you're moderating children's books, you might want to make sure that there are no swear words in there. Topic detection is understanding what topics are being mentioned in a given text. Sentiment analysis is the goal of trying to understand if a given sentence or piece of text has a positive or negative sentiment. Summarization is getting a longer piece of text and making a smaller one or two sentence summary of

it. Well, the second question is why are we so interested in natural language processing? The short answer is there is so much text and written format data in the world that we want to start using it on top of the structured data sources that we have. But on top of this, there are many other small reasons. For example, sometimes the structured data that you have is just not enough or you don't even have any structured information. You only have the chance of extracting the structured information from unstructured sources. For example, written text. And at the end of the day, talking is the easiest form of communication for humans, especially using it in a natural language way. So that's why we are trying to get the computers to understand and be able to respond using natural language to have our communication with the least

friction. And you might have seen there is a lot of research going into making this possible. And you might ask, why is that so hard? We certainly don't have a hard time communicating with other humans. It comes very naturally to us. Why are the computers having such a big problem dealing with natural language? Well, language comes easy to us because we're already used to it. But when you look at it closely, language has a lot of sarcasm, hidden meaning or idioms. For example, language evolves all the time. We come up with new phrases that did not exist before. And also there could be multiple meanings for one sentence or one word. And depending on the context that meaning is going to change. And these are all the things that would require a computer to know about to understand one language. And if you change the language all of these things start from scratch. And these are all the reasons why it's so hard for computers to understand and communicate with natural language. So to overcome these challenges and to make a model that can understand language or at least some part of it, researchers have been working on AI models for years now. Before the 2000s, the focus was on creating logic and linguistic based systems to decode and understand language. So researchers would write down rules of how language works and would hope that the computer would be able to use those rules to fully understand everything in the language. But it quickly turned out that language has way too many corner cases and exceptions to have a rule-based system to work alone. That's why in 2000s researchers started using machine learning based techniques and statistical techniques to understand and decode language. Although this was an improvement, machine learning based systems were still not able to fully understand language and work with it seamlessly. But over the last decade, we started seeing some serious leaps in terms of how machines can understand and utilize language. With the wide adoption of neural networks, we started seeing convolutional neural networks, recurrent neural networks, and their variations be used in the area of NLP. But arguably the biggest progress has been made using transformers architectures with the attention mechanisms over the last

couple of years. Of course, now you might be wondering how can I get started? How can I start using natural language processing in my projects? Well, building a accurate and custom natural language processing model from scratch would take a lot of expertise and a lot of time. That's why in the industry, many companies opt for fine-tuning alreadymade publicly available models or using APIs that offer the NLP service. Either way, if you want to build your own NLP project, there are many libraries that can help you. Some of them are NLTK, Spicy, Text Blob, Jensen, and Stanford Core NLP. If instead you would like to take a publicly available model and fine-tune it for your purposes, you can go and check out the hugging face library. If you would like to take NLP as a service, many big companies offer NLP capabilities as part of their cloud computing services. For example, IBM, Amazon or Google. Some other companies that give NLP services are Monkey Learn and Alien. These libraries and services mainly focus on text data. But if you instead have audio data that you would like to process, at Assembly AI, we offer audio intelligence capabilities such as topic detection, entity detection, auto summarization, content moderation, and more through our API.

All right, so this is where NLP stands for now. But where is it going to go? What does the future hold? One school of thought argues that combination of linguistics and deep learning techniques will be the future of NLP. But at the same time, we are seeing very big models developed year after year, surpassing what the previous ones can do. And it was predicted that this trend will continue and we will keep seeing bigger models every year for the foreseeable future. Are you interested in NLP? What do you think the future holds for natural language processing? Leave a comment and let us know your thoughts. Thanks for watching. I hope you enjoyed this video. If you like the video, don't forget to give it a like and maybe even subscribe. Have a great day and I will see you in the next