ChatGPT Study Mode - Explained By A Learning Expert

ChatBT recently released study mode. It's OpenAI's version of trying to make ChhatBT even better for studying and learning. But how good is it really? I've been a learning coach for over 13 years, and in this video, I'm going to do an in-depth review on ChatBT's study mode. I'll go over the good parts, the current issues with it, and some tips to make the most out of it. It's definitely a powerful tool, but there are some nuances with how you need to use it. Uh, and in my testing, I found that if you don't use it in the right way, it was actually slower than like just literally reading a book the oldfashioned way. But if you're not familiar with study mode, what is it? A lot of people are already using chatbt for learning. But with any new tool, there is a right and a wrong way to use it. And I've made previous video about how if you use chatbt in the wrong way, it's actually going to be more harmful than beneficial for your learning. And it just creates this illusion of learning where you feel like you're understanding everything but you're not actually gaining knowledge that sticks with you. And one of the big reasons for that was because it's so easy to extract information from chat GBT that it's easy to become passive in our learning and passive learning is ineffective. So OpenAI clearly was also worried about the same thing because study mode is basically their attempt to address this. In their release, they say that it offers stepbystep guidance instead of quick answers. It's really easy to activate. You just go onto the website, you click this button that says study, and it will just put it into study mode automatically. And now it's like having a personal tutor right next to you. And OpenAI say that they have worked with learning science researchers to get chatbt to interact with you in a way that actually supports real learning is what they say. The idea is that it's meant to be more interactive, more engaging. It challenges you and facilitates your actual thought process rather than just spoon feeding you information which is the big problem. And this is something that I'm really excited about because if they can get this right that is a huge step in equity of education. A lot of you probably don't know this but uh you know I've been a learning coach for over 13 years. More than half of that time at the very beginning was in the nonprofit space. So I was uh going into schools from lower socioeconomic positions and I was teaching them uh these academic skills. One of the major issues I saw was that a lot of these students are driven and motivated but they just do not have access to top academic resources. Like they can't get a private tutor. A lot of their teachers are turning over very quickly. One of the schools I was partnered with in a single year half of their entire teaching staff turned over. like they left for a different school cuz they're just so underfunded and underresourced. And so at the time my solution was to just go into these schools and tutor these kids for free. But obviously that's not sustainable. And so if ChatgBT is able to be like an unlimited almost free tutor available 24/7 for these students, that is a big deal. So to see how good study mode really is and how reliable it is for you guys to use, uh I ran a series of tests on it for around four or 5 hours of studying. These are the tests that I ran. First, I studied a topic that I have some background knowledge on, but I'm certainly not an expert. I'm trying to simulate like a first year university undergrad kind of knowledge. And so, the topic that I picked for this was LLMs and transformer architecture. And I tried studying as two different types of learners. The first type of learner was someone who isn't very initiated with learning science or learning methods uh and doesn't have a lot of metacognition, which means that they're not really aware of their own thought processes. So, I'm asking fairly basic questions. If I'm feeling confused, I'm not using the strategies that I know to work through that confusion. I'm trying to work through it in a way that I know that most students uh would typically work through this, being really led by the teacher, in this case, study mode. If I got stuck or confused, I would just ask questions that I felt naturally a student might ask about the topic. In the second type of learner, I fleck open to a much higher order of thinking. I was more metacognitive. And this would reflect a learner who is much more experienced with their learning, maybe has been trained on learning methods. I would evaluate and reflect why I'm getting confused and why I'm getting stuck. And I'd ask very targeted questions to see if it's able to lead me in the right direction. Typically, when you use chatpt without study mode, the value of the interaction is really dependent on the way you interact with it. So someone who is in that second type of learner who is doing that higher order thinking and asking really targeted reflective questions, they're able to get to the answer uh that's meaningful for them much more quickly. So they can make it click very rapidly. Whereas the first type of learner that's a little bit more passive uh and doesn't really know how to work through their own confusion, they're going to struggle. I then made a new chat with a blank history and then repeated those tests for medicine. This is to see if study mode interacts differently for different types of subjects. Medicine is a subject that I have background knowledge on. I have a medical degree and I was a medical doctor. Uh but also I used to teach those subjects extensively like hundreds maybe even thousands of times. So I'm

very intimately familiar with the types of questions that students tend to ask and where they get stuck on certain concepts. So I'm able to simulate a beginner learner very accurately. And then once again I made a new chat and then I tried it on learning science specifically with self-regulated higher order learning which is my area of expertise. This was to see if study mode is useful even if you are an expert at that subject. This is the type of information that there isn't really a wealth of training data widely available on the internet. Uh you'd really be getting most of your information from up-to-date recent research papers. In this final test, I assume that if you're already an expert, you're already thinking at that higher order. So I didn't simulate a lower order, more passive learner. So that's how I tested it. And now on to my findings. Let's start with what it did really well. First of all, the accuracy. I don't have enough expertise around AI and LLM and transformer architecture to know if there was some subtle hallucination there. But at least for medicine and learning science, there was no hallucination or the information was very accurate across the entire study session. So, especially if you are studying a curriculum or like an established field where you know that there's lots of information available on this online, um I would feel that the information provided is pretty accurate. The second part is that it does seem to be a lot more interactive. Normally, when you're using chatbt, you're asking something and it will just give you the answer. Like, it's just responding to you. In study mode, it does a better job at asking questions, having follow-up questions, kind of guiding you, and walking you through that process. The answers that it gives is definitely more sequential. It's designed in a way for you to logically be able to follow along. It's also pretty good at testing you. So with normal chatbt, you have to do a little bit of prompt engineering to give it some guidelines on how to test you and what types of questions to ask and how to sort of figure out the right level. In study mode, you can basically just say, I want you to test me on what we're learning. And it will do a really good job at generating some meaningful, relevant, targeted test questions. So straight off the bat, just with those improvements, it does seem to be a direct upgrade from normal chatbt for studying and learning. So if you're using chatbt for learning, just flick it on to study mode. it just seems to be better. And another benefit which I think a lot of people will be able to resonate with is that you can feel free to be dumb when you're learning through study mode. You don't have to worry about someone judging you for asking a stupid question. You can be as dumb as you want with chatbt and it's not judging you as far as I'm aware. And it may seem like a simple thing, but having that psychological safety to just ask and explore your own thoughts and your confusions very freely without fear of judgment is an important thing for learning. So overall, I give study mode generally a thumbs up, but there are some caveats and I think these caveats are crucial if you want to make the most out of this tool. So let's talk about the issues because there are some issues. First one I'll get out of the way uh is one that OpenAI themselves have admitted as a current limitation, which is that it's not very good at teaching for your exact level. It's generally just not very good at figuring out why you might be confused with something. If you're teaching someone face tof face, a good tutor or a good teacher is able to explain something and then get immediate dynamic feedback. As you explain something, you can figure out that this person has not understood this particular part of the concept. And so instead of just finishing off your monologue of explanation, you would stop, pause, pivot, check the understanding. Study mode is not going to do that. It's going to give you a pretty decentsized explanation. And if you're confused about it, it's up to you to be very clear and explicit about which part of that is confusing for you. For a beginner learner, that's incredibly challenging. When you're a beginner learning something, you don't know what the right knowledge is meant to look like. And so, it's hard to see if it's right or wrong. All you know is that something isn't clicking and it feels confusing. So this is a problem that open AAI has acknowledged and based on what I understand about the AI technology, I think that this is a problem that they will be able to solve eventually. It's probably just more of a cost and compute issue at this stage. Likewise, the second limitation which I definitely felt myself uh that OpenAI also acknowledges is the fact that it's not really good for uh multimodal types of learning. So, for example, if you are reading a really textheavy, conceptually dense explanation, it's really helpful just to have a diagram or an image or a flowchart or something like that to anchor your thoughts on and it makes it much easier to understand. At the moment, it's pretty much all textbased. And you can get it to generate an image for you to help you understand it, but I found that the images weren't particularly that useful and I found myself reaching for Google images instead because you know that those images are crafted by experts who really know what they're talking about. And so again, based on my understanding of the underlying technology, uh I think this is something that CHWT will be able to solve as well eventually. Again, probably a cost and compute problem. But in the meantime, until they solve that, I would definitely recommend uh having something like a Google images tab uh open next to you so that you can look for images when you feel like you don't

see how something fits together. It's going to be a lot faster than just trying to piece together the words in your head. But now, let's move on to what I think may be the bigger issues. The main problem that I felt was that the interaction of using study mode is very userled. And if you're talking about chipt as a general tool, you know, it's like that's just how it is. That's how tools work. You know, you get more out of a tool when you know how to use it properly. That's why an expert using AI gets better results than an amateur using AI. But this becomes really problematic when we're talking about studying and learning because generally people that are learning something for the first time are not experts in that thing. And so one of the things that a great teacher is able to do is they're able to teach the student how to think about this subject so they can navigate their own path and when it's too confusing show them the path to walk in the first place. Now, the way that chatbt tries to combat that is at the end of each response, it gives you like a suggestion. Like it'll explain something and it'll say, "Hey, do you want me to break that down in this way for you instead to make it easier to understand? " Which is a great attempt, but in my experience, I actually found that it didn't really help very much because it didn't target the reason I was actually confused. And this is actually about a very core principle of learning which is that learning is a messy chaotic uh very omnidirectional process and understanding a single concept correctly. There's actually hundreds maybe thousands of tiny subconcepts that you have to understand. So, if you're talking about learning an entire topic, it's very hard to read a big explanation and then point specifically to which sub subconcept you didn't quite understand or you may have misinterpreted that is leading you to not understand the big picture. And what that means is that your experience is that you are learning stuff. You are understanding what it's explaining to you, but it still isn't clicking. The big picture of it isn't coming together, and we're remaining sort of confused about it. The natural consequence of not having it integrated into a big picture network is also that our memory of it is going to be very short. That's one of the great illusions of learning is that you feel like you are learning because you understand what you're reading at that point in time thinking that that's going to translate into long-term memory, but that's just not how memory works. And then after you've read the explanation, you feel confused and then it will say, "Hey, uh, do you want me to explain it to you in this way instead or break it down into this kind of table? " And you say, "Yeah, okay. that might help me. So you say, "Yes, do that for me. " And it will generate that explanation. And again, you'll read it and it's much the same. You're still kind of confused. And actually, you can become even more confused because you're being taught more and more information, which means there are more and more possible places where that confusion is arising from. And it gets harder and harder to pinpoint where it's really stemming from. Now, normally if it's a human to human interaction, if someone is not understanding something like three to four times in a row, you wouldn't just keep trying to explain it to them like reframing it each time. You would have more of a conversation to figure out why they are confused and then tailor your explanation to specifically the part that they're misinterpreting. But I noticed wasn't doing that after even five or six iterations of confusion. So to combat this, I tried to get it to test me. I instructed Chatbt to test me on my knowledge to see where my confusion stems from. It generated a pretty good question and I answered it in the way that a genuinely confused student would. Uh, and it was much better at figuring out why I was confused, but it was a little too obedient. If I'm teaching someone and they clearly don't understand an important core concept, it's really important for that person's learning that I explore and probe the way that they have tried to understand this topic. The idea is that most of the benefit you get from learning a topic. And the way that topic becomes easier for you is not in understanding more facts and concepts. It's about getting a way to think about that topic. You have to create a mental model. And it takes a bit of time and effort to create that mental model. A great teacher is able to facilitate that process. They're able to see how they are trying to think about a topic and then compare that with the way that they, the expert, thinks about the topic. And I continually responded in a way that shows I do not have a basic big picture understanding about how to think about this topic. But it took a lot of very active work to get it to the point where it took a step back realize that was the issue and then addressed it explicitly. If this was a human teaching, they would put a mirror up to that student and make them reflect on the way that they are thinking about it and it would be challenging effortful. But after going through that process, things would start clicking a lot more easily. And at this point, I wonder if this is a conflict that OpenAI has because CHBT at the end of the day is a commercial product. And if engaging with Chat GBT makes learning

feel challenging because you're having to do this difficult thinking and it's actually making you think about it in a certain way, then that could actually reduce the amount that people use this as a tool. This is what we see universally across basically all learning which is that effective learning is active involves lots of thinking and is mentally difficult. Like it takes mental effort to do effective learning. You cannot bypass that biologically. It's actually impossible. But the great misinterpretation is that people feel that learning is meant to be easy. But actually when learning becomes easier it's becoming easier because your brain is doing less work which means it's doing less learning. And so I find this really interesting because for chatbt to really crack the code on making learning more effective and efficient for the user, they need to navigate that expectation of the learner themselves understanding that in order to learn effectively, they're going to have to do some of the heavy lifting inside their own brain. This as a self-regulated learning coach, like this is one of the big lessons that I make sure to teach all of my students, and it's a real challenge. So I'm interested to know how OpenAI is going to handle that. to give you a comparison of the difference it makes being a more passive learner versus being a more engaged active learner that's more metacognitive. I spent 30 minutes being confused with a specific concept, letting chat kind of guide me. When I did the same test again with the same concept, uh, and instead of just being more passive, I ignored its recommendations and suggestions, I looked inside my own set of knowledge to understand why I'm being confused and I asked very specific targeted questions and I was able to break through that confusion in 2 minutes. So the first instance 30 minutes of confusion constantly learning and reading and going back and forth with chatbt. The second case 2 minutes basically two rounds of back and forth to clarify it. And FYI I actually feel that it would have been faster for me to learn that concept through like a 15minute YouTube video instead of that 30 minute back and forth interaction with chatbt. So in that case it was actually slower than a more conventional style of learning. On the other hand, uh using a higher order thinking framework, it was much faster. The answers I was getting were very targeted because questions I was asking were exactly the questions that I needed to break through my own knowledge plateaus. And so it's interesting because the type of learner I was when I was interacting with CHBT made the biggest difference than anything. In fact, I would actually say that using ChachiBT as a active, engaged, metacognitive, reflective, higher order learner without study mode is vastly more effective than using it with study mode while being a passive disengaged learner. Being a higher order learner that knows the right question to ask, that knows how to navigate their own confusion led to much faster learning. I mentioned that I spent about four to 5 hours on this. probably 80% of that time was being the passive learner because it was just so slow being stuck in that rut of confusion because I just didn't know what questions to ask that would get me out of that rut and chach wasn't figuring it out for me as well. So with all that being said, here are my tips on getting the most out of study mode. First of all, and probably the most important tip, I would use it for more targeted study. Get to a point in your studying where you have specific questions or specific areas of confusion that you want to work through, then go to study mode. It will do an excellent job at helping you through that, probably faster than a Google search or, you know, like reading through the textbook and figuring it out the oldfashioned way. Second, try to avoid getting sucked into the rabbit hole of suggestions. If you read a response uh and at the end it suggests a way to make it easier for you, before you just say yes and then commit some more time to reading the next response, take a step back, evaluate what you've just read, and if you're still confused, try to understand what specifically about this is making it confusing for you and respond with detail. The specific points of confusion, try to explain and articulate in as much detail as you can. I found that when I did that, it was much better at correcting itself and then creating a much more targeted explanation. And the third tip, which is kind of like the general learning tip for all learning that you're ever going to do for the rest of your life, don't try to make learning easy. Your goal is not In fact, your goal is not like the learning at all. Your goal is just to have knowledge. That's the reason you're learning in the first place. You want to go from not knowing stuff to just knowing stuff. If you could click a button for that to happen, that would be the ideal scenario. But we can't do that. So unfortunately we have to do all of this like studying and learning to make that happen. When you make learning easier for yourself, most of the time you are shortcutting the process that turns external information into internal knowledge. And there are lots of things that you can do externally to make it easier for yourself. But when it comes time to do the hard work of thinking and

pulling it apart and exploring the confusion and making sense of it, these are all skills that you need to have. You cannot expect any more in the external space to make it any easier for you. Like it's already so insanely easy to get information. The bottleneck is it has been for a long time but especially now the bottleneck is whether you can take that information and actually turn that into viable useful connected knowledge. But if you build your higher order learning skills and you know how to think and learn like I teach in all my other videos, then chatd study mode is a great upgrade. And I'd recommend it for learning anything and it's only going to get better over time. And if you give it a go and then you find that you're still really struggling to learn a certain topic, I think that may be a sign to realize that uh the issue is probably not with chatbt. But anyway, it's an exciting space to be in and I'd love to hear what your own thoughts and experience have been with using chatbt study mode. Do you agree or disagree with things that I've said? Let's have a discussion. Leave your comments below. Thanks for watching and I'll see you next time.