What is Q*? | FULL DEEP DIVE

a few days ago I asked what do you think about qar in relation to open ai's quest to AGI being artificial general intelligence in which an AI has the same intelligence and capacity or greater intelligence and capacity than a human would the answers were pretty divided well in a recent interview by diverge it seemed like Sam did confirm that qar is a thing he said no particular comment on that unfortunate leak that implies that there was something El he would have said something like the unfortunate rumors Elon Musk also tweeted Ilia had a good moral compass and does not seek power he would not take such drastic actions unless he felt was absolutely necessary in reference to drastically firing Sam alond so it does seem like the evidence points to the fact that qar must have spooked Ilia to some degree which contributed towards everything that happened at open AI although this is of course still speculation since nobody at AI has given any concrete answers but in any case in this video we're going to be focusing on qar what it could be why the fact that it being able to do grade school math is such a big deal and to the common person what does this all mean and why should you care this video is sponsored by zenes who is leading AI safety and customer experience check out the links in the video description to learn how big companies are considering ai's impact on their customers after Sam got fired and then reinstated 4 days later there was a lot of speculation about why he got fired in the first place maybe it was his issue with Helen toner who published This research paper that was slightly critical of chat gbt maybe it was because open AI agreed to buy $50 million of AI chips from a startup that was personally backed by Sam Alman maybe it was because he was also fundraising in the Middle East but what really caught everybody's attention was when this Reuters article initially came out saying that open AI researchers worn Board of AI break through ahead of CEO aler possible breakthrough in artificial intelligence and a staff letter warning the board about it the researchers letter raised concerns that the discovery could threaten Humanity yeah pretty dramatic there's honestly quite little details surrounding this Reuters said they were unable to actually review a copy of the letter and after being contacted by Reuters open AI which declined to comment naturally they declined to comment acknowledging an internal message to staffers about a project called qar and letter to the board before the weekend's events it's pretty confusing the message um sent by long-term executive Mir moradi alerted staff to certain media stories without commenting on their accuracy so basically it's like one huge question mark there was a couple more details given a vast Computing resources the new model was able to solve certain mathematical problems though only performing math in the level of grade school students acing such test made researchers very optimistic about Q star's future success you might be wondering why is performing math a grade school level something to be so excited about well the reason is because generative AI is not pretty good at language based stuff like writing and language translations but there is a lot of room for error because there isn't one single correct answer but concrete ability to do math where there's only one right answer implies AI would have greater reason and capabilities resembling human intelligence this would essentially open up a whole other branch of capabilities especially in felds like scientific research where logic is very important so even though it can only do math right now at a grade school level now that capacity is there actually optimizing it and refining it is just a matter of time and money it would be a huge step forward towards AGI obviously this is not super solid confirmation of anything of course there are prominent figures in the AI field like Yan leun he says please ignore the Deluge of complete nonsense about qstar it's not anything particularly new fras cholette who also does deep learning at Google and is the creator of Kira and every single month from here on there will be rumors of AI having been achieved internally just rumors never any actual papers product releases or anything of the sort so yeah take everything with a huge huge grain of salt but I did personally do a lot of digging into all the theories that people had about qar and I thought I would share with you guys a couple of the most believable theories at the very minimum give you some context and some insights into what some people are working on in the AI field how they're being applied now and what lications they can have for our future this Twitter post by cus alberty um is a pretty good summary of the two major theories that people have that seem like the most possible the first one is some optimization related to q-learning and the second one is some combination of the AAR algorithm and q-learning given its name we are going to unpack both of these now starting with Q learning I asked chat PT what is Q learning can you give an analogy to explain it and came up with a pretty good analogy the Maze and the treasure it says imagine a mouse in a maze the mouse's goal is to find a piece of cheese hidden somewhere in the Maze each time the mouse moves from one point in the Maze to another it learns something new about the maze the mouse doesn't know the layout of the maze initially nor does it know where the cheese is but as it explores it starts to learn which paths lead to the cheese and which don't Dolly came out with these cute illustrations and the first one here is the mouse taking a step into the maze as it takes it then starts learning a q value which is a score that tells the mouse how good each step is in relation to getting closer to the cheese the mouse keeps track of the Q values in a table called The Q table and over time it fills out this table with values based upon experiences after learning which steps are most likely to lead to the cheese as the mouse keeps exploring the values more and more the Q values in the Q table keep getting updated and they became more accurate representing being able to find the cheese more efficiently throughout the whole process the mouse always chooses the step with the highest Associated Q value which is the step that is most likely to lead to the cheese based upon its prior experiences exploring the maze so yeah this is how Q Learning Works the mouse learns to make decisions on where to step next in the maze in order to maximize his reward of finding the cheese based upon the Q values that represent its experience kind of extrapolating from this example the key idea here is that the agent like the mouse in this case learns from its own experiences by exploring the environment and it updates its decision based upon past rewards and actions Q learning is used in a lot of different fields like for example in robotics is used for pathfinding and avoiding different obstacles in trading q-learning alos can optimize stock trading strategies and it's used to learn Market data and maximize different returns it's also used for autonomous vehicles like self-driving cars uh supply chain Inventory management Healthcare personalized medicine list goes on okay so coming back to this tweet it sounds like it's related to Q learning for example Q start Den notes the optimum solution of the Bellman equation what he's referring to here as the bellman's equation the simplest explanation is that it's an equation that helps a model guide towards an optimal solution which is called the qar so the whole point of Q learning is get towards this optimal solution of qar which is why people are speculating that this AI called qar is somehow improving Q learning in some way to get closer to the optimal qar so I asked Chach to speculate a little bit and it says that it could be conversions to W qar in terms of accuracy and optimality because one of the main issues with q-learning now is that it does try to approximate towards the optimal but it doesn't always get to the optimal it could also be speed of conversions which is efficiency um qar right now can be very slow to converge depending on certain environments especially if it's very complex environments it can be better at learning things it can be more stable etc so a lot of different ways that you can improve Q learning that qar may be second plausible explanation is a combin comination of AAR algorithm and Q learning hence the name qar there's this really good article in LinkedIn um that explains how it could be combined together so I'm going to walk you through the example that the article gave full credit here to civa although I did ask Chach PT to draw some images just ignore the fact that it's not very good at spelling but anyways so in this example think about Q learning in the context of learning a new recipe imagine Q learning as learning to cook a complex dish without having a recipe so each ingredient and cooking step is a decision that you have to make first thing that you do is trial and error you try a lot of different combinations of ingredients and cooking techniques um some of it is going to taste yummy which is a reward and some of it is going to taste not very good which is going to be a penalty as you experiment you keep notes at which combinations and techniques work the best um and you put that all together into a notebook which is your Q table with each cooking attempt you keep refining your notes um learning about successful and unsuccessful outcomes aiming to perfect the this dish until you get the recipe for the perfect dish that has the best ingredients and the best techniques now let's talk about AAR search which is an algorithm that goes all the way back to the 60s it's usually used to find the shortest path between like two different things based upon some information um some experiences that it has so here's the analogy you can think about AAR as the process of planning a complex meal where you have to prepare multiple dishes efficiently so in the beginning you're evaluating different steps of the cooking process um you're looking at different options like which dishes to start with which ingredients to prepare next which cooking method to use for each option Asar then evaluates how choosing it will affect the overall meal preparation time so it's thinking like hm if I roast the vegetables now while I simmer a sauce then will everything come together more quickly or not so it keeps going through this process um choosing which seems to optimize the cooking process which doesn't try to make sure that all the dishes are prepared in the shortest combined time with the ultimate goal to find a sequence of all the cooking steps that lead to the most efficient preparation of this entire meal so it does need to have information about what the recipes are composed of can you roast vegetables while you're simmering to Sauce um how long does steaming take and with that information is able to optimize for the shortest amount of time and the least amount of effort in order to make sure that everything is done correctly so how does q learning and Asar search come together well imagine that you're organizing like a really big dinner party where you need to cook multiple complex dishes each with its own set of ingredients and its own cooking steps AAR search comes into play by planning the sequence of cooking steps for the entire meal it would consider factors like how long things take to cook um what are the preparation steps in order to figure out the most efficient way of preparing the dishes while also using Q learning to refine each Dish's preparations based upon its learnings about successes and failures and all of this coming together this dual approach makes sure that each individual dish is as tasty Y and yummy as possible but the entire meal is also coming together in the most efficient way possible that's an example of how Q learning plus AAR search could come together and make a very powerful algorithm called qstar okay so there's one more piece of the puzzle that we know it was reported that SS covers breakr within open AI allowed open AI to overcome limitations and obtaining enough high quality data to train models a major obstacle for developing Next Generation models the research involved using computer generated rather than real world data like text images pulled from the internet train new models this is what is called synthetic data so synthetic data is data that's generated from computers as opposed to real world data that most of these large language models are trained on by being able to use synthetic data efficiently this can be really powerful for example there are certain topics in certain industries in which there's not that much high quality human generated data in order to train algorithms on synthetic data would be able to solve that problem also human data is messy it's dirty it's biased so by being able to use synthetic data we're able to train the models minus the things that humans do that we wouldn't want AI to have for example be Prejudice or aggressive the theory is that qar is either partially or completely trained on synthetic data which could be why it's a breakthrough in any case I'm sure more information about this is probably going to come out and I just want to make a note that this really is just a lot of speculation over here even if these components are true there's probably other piece of the puzzle that we don't know about most advanced AI systems use a combination of different techniques so I'm sure there's other bits and pieces that form this algorithm called qar whether qar is real not real impactful not impactful the one thing that is undeniable is the speed of AI development and how much effect it's already having in many different Industries right now that's why I want to show you some examples of super fast progress in different Industries and keep in mind that the faster AI develops the faster these Innovations will be happen happening and all this is within the past few weeks to few months first off material sign Google deep mind's new AI tool help crate more than 700 new materials newly discovered material can be used to make better solar cells batteries computer chips and more and these are not just hypothetical materials Google AI and robots are joining forces to build these new materials world of healthcare AI scours documentation for cancer studies and AI helps diagnose and manage kidney diseases there's predictions that Healthcare is one of the industries that is going to be most disrupted going into 2024 in the world of customer service investments in AR are accelerating after seeing what recent AI events are capable of Future Ready support leaders have already started thinking about how to integrate AI power technology into their Tech stock with 69% planning to invest more in AI in the next year customer service predicted to be one of the most impacted Industries which is why I really appreciate zenes who is a leader in this field for opening up conversations on how AI is being integrated into a customer services send is also the sponsor of the video they asked me to spread the word about an economist impact Roundtable discussion that they sponsored discussing the impact of AI on customer services such as operations for efficiencies but also the risk Round Table features top leaders in the customer experience and customer service field Adrian starts off by explaining the way that zendesk is thinking about AI um which is to enhance customer experiences rather than automation were replacing expert I found really interesting about how it's impacting customer service professionals and how is shifting their work a subject that I know a lot of people are interested in so what we see is that a lot of the rote repetitive work that an agent has to do will be taken over uh by AI by automation um and operations will become a lot more efficient because you'll be able to for example detect the intent of a customer inquiry and then route to the right agent with the right skills so the consensus of the panelists is that they do agree that agents would remain the point of escalation for the ne next decade at least that will be the fuel that um gen will provide it will supercharge many of these roles but um you know the human touch will be needed for a long time as Adrian explains humans ultimately like to talk to humans and they intend to always keep humans in the loop they were all really excited about this technology but they were also very open and expr saying that we needed to exercise caution because even though the technology is good at making predictions it can also be very confidently wrong it lacks human reasoning and is prone to errors which has severe Regulatory and legal implications everybody agreed that generative AI proved an opportunity that can greatly improve customer services but it needs to be thoughtfully implemented I think that this is a thought that should be echoed across all different Industries and all different companies you can watch The Economist impact webinar that Zenda sponsored or check out the events summary both are linked Below in descriptions all right so before I end this video I want to tie everything together that we discussed today I think that every single person should be keeping an eye on AI development especially in relation to the field that they're currently in we already saw how much AI is already changing the world around us and how fast Innovations are happening if there's one thing I want you to take away from this video is realizing that AI is here to stay and it's very important to start embracing it because the changes exponential and it's only going to get faster and faster from here so it's much better to be in the loop taking advantage of the opportunities that AI presents and being the one shaping how AI is going to be influencing the things that you care about if you're interested in learning more about AI um or maybe even getting your hands dirty and integrating AI into different products I do have a weekly completely free lunch and learn Series where we talk about Ai and also do workshops and tutorials now is definitely a time to start learning about these things there are a lot of opportunities in the AI space especially if you know how to code where you're interested in learning how to code all right thank you guys all so much for watching and I will see you guys in next video or live stream