Vibe Coding a Robotic Hand to Crawl (Inspire RH56DFQ)

what is going on everybody can you believe it we are back so soon with another video I'm going to be continuing to play with these robot hands as well as um just trying to see what are the limits of doing this sort of uh I hate the term Vibe coding so coding with llms all that but that is the word that people are using thank you to Andre karpathy for uh coining that word um so today we're going to be uh continuing along and I got you know a few comments that I just want to sort of um address cuz I've seen these things before where you know people are saying um you know for example okay well you could have done this you know without a large language model of course and I think I addressed that already uh but it would have taken me much longer and then on that same vein also people are saying well um yeah but you didn't learn anything well that's not why I program so like I don't write code to learn things right I learn things to make things happen that's what I care about doing stuff and to some extent like when the hand was screwing up and I just wanted I did want to understand okay well why did that happen so if it ever happens again I can understand how do we get out of this problem right because that is pertinent to what we're doing and continuing to do things right so in those cases I do like to ask the language model as I did you know why did that happen or what is that or and same thing with like modbuz I don't know what that is what are we doing like what are we talking about here when we're working with this stuff you can ask the things that you want but that's the glory in using language models I feel like is that you don't need to know the stuff that you don't care about like a lot of Robotics like what I'm interested in with robotics is like doing stuff with robotics not addressing specific bites of information and figuring out how to send and receive like it's interesting only such that I could use that information to control a robot like I just want to control the robot right but I also can understand and sympathize that other people might be actually interested in learning how to do those things well you can use large language models for that too right you can do all of that it's just that in the world of programming like if you do not allow yourself to do any abstraction at all like you you're you will never get anything done okay first you're going to like I don't even know how deep do you want to go write everything in assembly now like what are we talking about so I don't like those arguments either but anyways that aside I think one of the things that I that kind of didn't hit me until later on in that video was the fact that the hand and we even asked I did ask the question I think it was on video you know are what were those actions like the point and the pinch and all like those things were those things in the actual uh specification like is that like a pre-made gesture basically and it wasn't and like when I scrolled through at least I did not see it there um you know like if I search uh well point my okay no it does it doesn't even point something but uh you know it doesn't look like it's there so in order for Claude to know how to point it would have to have some sort of mental abstraction and I don't want to get too corre use the wrong word and like piss people off because every everyone's going to be so nitpicky about how I phrase things but in order to point not only does it need to know how to send the command right it also needs to know what a point like looks like and then based on the definition set of uh instructions that it's been given it needs to also be able to take that and morph that into what it knows a point should look like right I find I found that very impressive um and I still even though it got the first initial point wrong but like a point a pinch all that stuff it can do thumbs up like so for example um let me yeah so I did eventually put it up on GitHub hold on let me pull that up for you guys um so I did throw up a version of this on GitHub um so if you had the hands I guess but anyways so I did put it up here and you can run this and so it's the code here is going to be a little different but again this code is 100% written the only thing I manually edited was I added this to the very tippity top and then I think I added this or maybe I just wrote this part and then I added this part because I want I just wanted this for my own notes like I made this repo mostly for me um because that's another thing working with llms you have to do like pretty good version control otherwise it you're just going to lose your work so anyways I put up the working version up here mostly for myself but um apparently six people have started so um cool I guess okay so anyway I did I have this um thing so now what I want to do um I've got the CLI set up so I just I'm going to change the camera now cuz you really don't need to see too much of me

anymore okay and now we have the hand I might have to adjust some of this as time goes on but anyway you get the idea um let's do some cable management for this cable uh okay put it there so for example um the open and close I think is is pretty simple but if I type help here uh for example we can do a pinch I can't even remember this one might not actually be functional yet right it just like missed each other I think the thumb has to like get into position first hold on let's do um uh let's see so the pinch didn't work do we have a thumbs up okay thumbs up thumbs up I like how it like Cascades down that's kind of cool um okay so you get this like thumbs up and again um maybe that one's not too difficult what was the other one do we still yeah we have a point here uh so we'll point right and again I think there you know it's like everybody knows what a point is but if you have to think about like how would you describe that to um you know to your program as you're writing the code you know it's not it's trivial but it's still like you would have to have a mental idea in your head on what a point looks like and then how to translate that to the code so anyways I found that to be interesting so then it's got me wondering is that like a fluke or is there more to that like can we do more gestures based purely on um like its mental model of a gesture that probably exists like a point is probably something somebody programmed on a robot hand before but again I can't find any like I'm sure it exists somewhere but like python code for the Inspire hands for example and again there's lots of little subtle differences in a hand like this like you have the different joints at least in this case you do have the fingers but then you know the way that the thumb rotates and bends you would have to make some changes so again like I there's always like little tiny differences that I think it's like really easy to just kind of like discount um how impressive a thing is but anyways okay let me uh let me just open the hand I think it's a it's always a good like safe starting point to leave the hand open so I'm going to minimize this and I'm going to say uh let's see so make sure you're on the agent let's see agent uh Auto Select is off very good and I'm going to move this over to at least here okay um I'm going to say uh hi there uh and in fact uh we probably want to be inside the package so hi there uh hi there inside this Inspire hand package uh we have an Inspire hand module that we can use to interact with this um and also Pro it could be useful um in this case I guess we it doesn't really matter that it's a right or left hand at least to start um it could matter later down the road potentially but I think I'll let that slide for now uh we use with this Inspire robot hand please uh CD into the um ins insire hand package uh directory and uh read the readme. md file to familiarize yourself with everything then go run the uh clipy file in interactive mode to confirm everything is working so I just want to make sure um that we're feeling good so let me go ahead and run the command here let it read everything potentially index things it should it's pretty simple I really actually like the CLI and the API and stuff that it like built uh for us here uh it it's uh it's like everything I could have ever wanted and that's what I mean is like I wish that when you bought the Inspire hands that they just already had a python API like I did not want to write the python API that's not why I have the hand or why I'm interested in the hand I want to Pro I want to make the hand do stuff right that's my interest so um so again I think uh that's I don't know there's just something really incredible about these large language models that can just do that for you now I just I was really Blown Away by that example um anyways continuing on uh python oh wait hold on let's do cancel here uh when you run with python uh use Python 310 instead please when you run with python uh okay run that command help um we'll just do a point real quick oops not connected to

hand okay so it says is it potentially because I'm actually already connected to the hand uh why would that happen not connected to hand um I wonder if it's because maybe because I was connected on this one um H cancel let's try that again uh let's try the interactive again I was connected with another script I think in the past I've been double connected before but maybe I'm mistaken um uh let's see uh not connected to hand so what would be the uh what is the problem here yeah okay it's trying to run this script which um I don't even think we're using that so I'm actually going to delete that one uh yeah okay uh so then what I want to do is actually we're going to cancel that yeah run with python I really just want to confirm it works oh my gosh um okay I'll try running with that instead okay run that uh Point okay so that's what it was I think you actually can maintain multiple connections at the same time not that it would be a great idea but I'm pretty sure we can um okay so we can do a point all that uh let's go ahead and open and then exit uh okay great so we're connected okay now what I want to do now that we've spent um all that time getting into it and kind of like talking about things uh what I want to do is I want to show um let's try uh like rock paper scissors um what I would like to do now is create some more demo gestures I would like to try to make like rock paper and uh I don't think it cares if this is uh grammatically correct Rock but okay rock paper and scissors um what do you what do you think um uh and then the other thing is I think I can't remember if this opens and closes weird I still kind of needed to that was like another issue that I wanted to address in the I can't remember if it was in the other video or in between videos but uh like you need to get that thumb like out of the way because if this gets stuck trying to open sometimes the finger literally get stuck or it gets like flung open weird and I just I don't really like that like I don't think it's bad for the or anything but like it just kind of like flicks open uh and I think it's like a rubber band or something I'm not really sure like how what is in there that like makes it but like the servos or the motors I don't even know someone correct me below servos or M Motors um probably Motors I don't know anyway uh the only force that can ever be exerted by the motors is D like in to like Bend IN to come out it's actually just like a rubber band there is no like outward force that can be like pressed so it has a hard time breaking through anything when it's like trying to like come out if that makes sense um okay so what demo gestures I would to make rock paper scissors what do you think um for anything that brings the thumb inward please move the thumb in uh last I don't really want to prompted about rock but I just I know rock is going to be like a close and I just want the thumb to get on there uh on top rather than on bottom I think so cuz the thumb does when it rotates out I don't think I think yeah this hinge here I think really both of these like the little I'm not sure on this the tip of the thumb that might actually be a motor that goes either way but like this middle one definitely although it seems to really need to move the whole thumb I don't know it' be interesting to see a break a tear down of uh this hand to be honest um anyways okay uh back to work uh please move the thumb in last and then anytime we open back up again please move the thumb out first okay let's just see if it can do the gestures of rock paper scissors again there is nothing in the code that suggests um first let's understand the hand structure okay multiple fingers BL blah

blah python script with these three gestures following your specific move the thumb last when closing and first when opening okay very good okay RPS gestures rock paper scissors cool we accept we always just bang bang accept um I will say I know you can like Auto accept like you can run it in like YOLO mode and like do auto accepts um I find it uh the behavior to be un unpredictable I suppose because like if you Auto I want to say accept like CH mods or rather let me uh let me just accept this here we'll do that uh can't access no such file okay um first of all I do think I will say I don't like Yello mode or like running immediately because it actually it kind of like disconnects you from what's going on and I definitely I do like a little bit of understanding of like what's happening um oh is it like already ready to okay uh anyway so like imagine you say auto run python 310 and like any python stuff you want to auto run if you auto run but you don't Auto accept edits to python files I don't know if someone's watching from curser I'd like an answer on that like if I didn't accept the edit but then you go ahead and run the script isn't that like confusing to the AI like it doesn't know like it didn't apply the fix that it thinks it applied and then it does the test or whatever so like I don't like that either like there's lots of little examples like that I feel like so anyway uh continuing on um not connected to the hand I feel like it's not using the actual module that's the problem uh yep yeah you have to actually use the module so I'm not really sure why it's trying to skip that honestly both times why did it try to skip that okay so here it is using that and then it's going to import them okay I'm going to accept all accept go ahead and let that run uh I don't really understand like why wouldn't that uh work directly with that module uh we'll have to well I'm going to need to take a pakc into that in a second here uh except uh let's see how did it run it though yeah okay so it is running it um okay so it does do the point okay let's go ahead and open um okay let's run help real quick okay rock I love that it does the like slow gesture thing um let see if open actually yeah it's still a little clippy there um okay so rock makes sense uh paper well really paper should be pretty much exactly what you see here I mean maybe it should curl the thumb up I think paper is just like straight up normal um let's go do a point and then we'll go paper yeah I do I don't know why it adds that cute little wave uh thing but it's cool okay scissors um yeah that'll be interesting let's see if it does scissors that's not bad not as best as it could have been um yeah okay I'll take it it's like a p sign but okay um okay so it got all the I would say all of those gestures are correct again um the fact that it can do that uh I don't know I just think that's pretty interesting U I'm sure somewhere in the data set there's information on all these things but not with this hand not with those exact commands I don't know I I find this to be impressive for some reason um okay now what I'd like to do is um potentially we're going to wind up with an Abomination here but what I want to know is okay so it can do pretty impressive stuff that is likely in the training data but I can think of at least one thing that probably I mean it might be in the training data but I don't think it is um and that is um well you'll see in a moment okay so um actually oh should have opened the hand um will you uh try to use try the module now just CLI and open the hand not interactive just run the open via CLI let's see if it can do that it should be you should be able to

use the CLI module not just in interactive but also just like okay yeah okay good I just want to make sure some of these things work as I'm expecting them to work okay so um oh it can also do like a rock uh okay fine Rock making rock gesture not connected to the hand so did it do the same no it did not we have connection what no I don't want to do that I just don't really feel like doing that so why did the other one work and not that one that's so curious um but I'm actually not curious enough to attempt to resolve that um where am I H the pain okay go ahead and run that um rock yeah open exit Okay the connection appro so why yeah why is that different that's so weird I don't understand why that so that might be a bug um that might be an actual bug but why did the other CLI stuff work so like in this case if I hadn't if I had YOLO or like you know run every time 310 it would have potentially tested this without making that update I think uh someone feel free to correct me if I'm wrong but I'm curious okay so it fixed that um okay go ahead and run the command to open all very good okay this is not even really what I want to do why are you still going trying to run these gestures we already tried the rock uh pay for sure I do like that it moves the thumb first that should be the open new open honestly scissors very good okay um we don't need to okay we're not going to need to run the same thing again um no no stop it uh please open the hand for me real Qui quick let's just get that hand back open and then what I want to do is what I really want to do so open the hand for you right away thank you okay so okay now I want to try something a bit more uh complex um okay how do I want to frame this uh in this case I do think being a it is a right yeah that's a right hand the hand we're working with is a right hand if you can guess where I'm going with this right now congratulations um the hand we're working with is a right hand what I would like to do is create a um sequence or series of movements such that the hand when starting Palm down and open go ahead and open the hand always first before beginning the sequence and open begins to crawl forward does that make sense do you think you could do this one thing to keep in mind as you uh crawl or try is the hand is great at um having bending inward pressure but for moving back out it's only essentially a rubber band of strength it doesn't take much for the finger to not be able to move out so to move Fingers um back out from under the hand for example you would need the hand to rise up a bit by using the other fingers or thumb the thumb I believe the thumb rotates in both directions with enough force to move the hand any other questions or do you think you're ready to make the hand crawl forward okay this is hard mode uh we'll see understand the challenge crawling with a robot hand is

fascinating let me clarify what I understand about the Tas the hand starts Palm down right fingers are extended the goal is to make it crawl forward across the surface the fingers are strong when bending Inward and closing but weaker when opening back up we need to coordinate movements to lift parts of the hand to reposition the fingers similar to how an actual hand might crawl this is definitely doable I believe we can create a sequence that uses the thumb's rotational ability to help shift weight that's probably that would be probably what I would do um I just got lost uh here we are okay uh coordinates finger it's like already produced a solution love it you love to see it coordinates finger closings to pull the hand forward strategic oh my God I can't speak strategically lifts parts of the hand to reposition fingers for the next quote unquote step um okay let me create a new script for this crawling motion I'll just design a sequence that respects the mechanical properties you've described okay um crawl and then crawl CLI okay I don't not really sure I needed a CLI but um we will take a CLI I suppose um except all uh we will go ahead and give it the uh capability to be executed okay for the initial position prepare okay so it should just be open as it is okay of an initial crawl step okay before I put it on I'm trying tode if I want to put it on the surface or like leave it upright and we can determine if we think um I'm going to leave it upright first for testing purposes uh let's go and run the command that's interesting oh that's it so it's going to release those I don't know if that's going to work that will be interesting repositioning for the next step so that would be the next step okay and then it's going to try to release those out but I don't think those will come out personally so it curls all those in it does a little step that releases the weight but because of those it I don't know that might work because that lifts up the hand that gets the f that could be interesting that could be actually pretty interesting okay we got to see it in reality um okay uh we will reset back to open I think I have to see it uh actually happen based on what you observed um no uh let's try running all of those again and I will continue to observe so okay now what I need to do we're going to readjust or adjust this a bit okay let's see where is okay so we're going to let the hand be palmed down um probably about there I don't know here maybe dep it really just depends like I have no idea like what is it going to do initially I has like a have a decent initial first like impression of what that looked like but who really knows um okay let's try running all of those again and I will continue to observe and report back how about that okay so open all very good I'm happy it's open move this keyboard again uh okay prepare we'll run that Palm down I think it's prepared I'm not sure how would it know preparing camper craw what I'm not really sure why that matters or how it could possibly have known that it is Palm down um CLI crawl CLI step single step okay run the command oh my God okay I made some I actually made a decent progress there oh my gosh oh dear oh my God I think I have to like Mark it with tape where it started but uh that's actually a really good motion let's do uh continuation here now it's back flat again oh my God we're running out of cord oh no hold on let me get more cord oh hurry oh my gosh it actually is working oh my God oh my Lord oh man that works really well oh my it's going to press keys oh

my God okay you got to stop pull it back here oh my God that worked really well oh my God I don't know what to say about that um yeah that's insane we just one shot the a hand crawling now I I had to move it cuz we literally ran out of space um give me one second and I'm going to try to make a little more space so I won't have to do that again I also was running out of the chord like the power and all that okay I need to see a longer crawl sequence um okay so okay could we assume could we after preparing do just um do a longer crawl do more um okay I don't know how to phrase this I wanted to know that it worked good so okay that actually works really well could we keep the crawl going for say two or three iterations okay we can modify the number of steps using the steps parameter so how many did it use I don't even know okay so run the command to open very good um now let's run eight steps I might have to actually stop it hopefully that doesn't fall off my desk that would be unfortunate people would be very angry with me okay all right so she's off oh my God I just cannot get over how well balanced that actually is like to act like it actually solved the problem of the fingers getting stuck and stuff and like how like I just oh no oh my gosh we're on four of eight we're going to stop it probably I'm just going to read fix the angle for a sec here oh my gosh oh man this is insane to me I just cannot I just can't believe that works hold on we have to stop this though okay cancel oh my god oh man that worked really good that went like two and a half feet that's insane that is absolutely insane um okay um I okay I I hope that you guys are sufficiently impressed with um cursor Claude even the hand I mean the hand's pretty impressive I mean this was not meant for walking and we made a crawling hand we just zero shot a crawling hand um yeah man the stuff of Nightmares honestly for some people but yeah that was that's freaking awesome that is awesome okay um yeah I'm G to stop it here I hope you guys have enjoyed um if you have any suggestions for uh interesting gestures that we can do with the Inspire hand uh please do let me know um that was shockingly impressive okay for a slightly better view let's run again um let me check let me make sure I'm good on the cord over here okay and send it what have I done

okay oh my God I can't use my hand to get this I uh I hope you guys have enjoyed if you have questions comments concerns whatever uh feel free to leave them below um otherwise yeah I will see you guys in another video I can't think of anything more insane than making this hand crawl honestly so really probably the next video would have to be the humanoid but um you never know my dog says hello bye e