Create dashboard to manage test deployments

just share my screen here okay so can anyone see this slide yeah cool so essen and i teamed up to create poseidon which is one big nan ocean um and so what this is a tool to manage various test instances so you don't have to build locally you can just define a branch a few other metadata and basically push the instance or update that instance so kind of similar to what we have on cloud but the ability to have custom branches and that kind of thing for testing we have something similar like this now mutassum created this awesome basically github actions in heroku workflow it's triggered by a slash command um with that i think talking with mutastium you know that was still on his sort of personal hiroku account and so what we wanted to do is kind of take inspiration from his thing and set something up that can be a simple mvp that we can then build upon so we basically have our own internal tooling for doing this um so i envision us using this for user testing feature testing basically anyone that wants to spin up an instance could act as a time machine you want to go back a few versions you want to try some different branch you don't have to mess about with the terminal to build this and hopefully with the great stuff that the advanced team was working on we'll get it even faster um so

so um i'll go into like stack and sort of how we built in a sec but i'll first start off with a demo um so we used retool for the front end basically retool consumes n and a bunch of stuff under the hood um this is the dashboard page so here we have a list of the instances that are available and i think in future we would want the ability to be able to add more instances right now sort of fixed so there's a finite number of slots and you can choose which one you're assigned to um so from here you can go ahead and open that instance this one has basic auth on it um or you can go open that instance so if i look here there's a second instance here i can view slash modifier and this opens the individual view we see everything's loading and then in the workflow just served all this info here so we have all sort of the different information for the instance the username the password the url the branch that's assigned to it as well as the ram and cpu right now these aren't changeable in future we'd like to make them changeable and probably have some presets for things like an in-cloud start and in a cloud medium so you can actually spin up an instance and see what it's going to be like for a user on a certain one or in future to try and represent things like an ec2 t3 micro instance or something like that so we can start emulating what something might um uh how an nnn might work on a sort of a real instance type out in the cloud um in any event you can open the instance um in this case i already opened this one so the auth is already set but i could copy the username copy the password if i needed to um and then if we want to modify the instance and click this which opens up this model here we could give it another name which this is just so for people to understand what it's for maybe this one's for user testing maybe john stamos wants to use it and then the branch that we'd like to use here perhaps we've got one here we want to switch to this one we want to test that in future of course we could have maybe a little drop down pick up to explore the available ones and perhaps we don't change the username and password right now then we will modify this again we're using nan to handle all the btl for this and in a second here we should get a response that it worked the view behind updates and right now similar to the customs original we basically open up a github actions view which is part of this back end here which shows that this is building in future we'd like to basically have some asynchronicity and bring like a status into this dashboard so you never have to go into github um and then so from here when it's updated you can open that instance right now it's updating um and you can use it um and that's sort of in a p in a nutshell the front end experiences these two views the way we built it because we're basically consuming ended in workflows it's going to be very easy to add like a matamore slash command or something to do some of these steps again an mvp of that could just open this view another one could replace this view because uh basically what we've built is a headless backend right this is just consuming that if we have something that consumes in the same way so any number of things could consume them um you can also report an issue send an email to me and you can tip us in crypto just kidding um but you can donate dunas vault numbers to max and essen if you like um so the tools that we used to make

this happen we have retool in the front end that's what you saw here today um honestly i haven't used retail as much um or as deeply as this time around for me what was really cool was how easy it was to set up things like this kind of asynchronous waiting for the status of things so if let's say i change this here you can see this loading state and whatnot it's waiting for the internet workflow to return if it returned with an error we would see actually some error validation in this case again it was successful so we see that um then we use nan it's basically our back end so it handles the extraction transforming loading of data between retool base row and our nan cluster um we then use nine cloud actually for the instance cluster so using a staging in uh instance and give you a bit more details on what we're doing exactly there sort of he handled that part um and then we're using our internal base for instance for storing various data here's an example of what that database is so most of the data you're seeing in retools actually lives here because it's just basically metadata for humans um to have a record because it's not so easy to fetch this data from the instance um so for example using password et cetera obviously it's in plain text right now we figure that's okay because these are test instances so they're not going to be um have very sensitive data on them and i think in terms of improvements right these names they just sort of uh you just type in text for now so in future that could perhaps be typed as well maybe even integrated with notion um so yeah so that's the stack that we used and if it helps this is sort of how

everything's structured so we have user over here using retool which talks with these n workflows which then consumes these other sort of things here um and i can quickly go over sort of one of these flows and perhaps essen can show you some of the other backend moving parts um filter this down um we had broken up into multiple workflows some of these are deprecated now we realized maybe it was a bit overkill um but the first flow that you use is called the fetch all instances this is what's used to populate that dashboard view that you see over here and basically what this does is it just pulls in base row and acts basically as an api wrapper for base row so we could have gone directly to base row what's nice about this is in future if we want to start pulling in other data sources perhaps the status of the instance isn't saved in base row we've got to pull that live or any number of other sources we can do that all here in the intermediary step tweak this set node which basically prepares the response payload to send back to retool and we're good to go um so that's fetch all instances there's also a single fetch instance which just grabs a lot more detail um to populate that individual view when we open it just so it's a bit more efficient and then we do have an update instance and this is a post action and so what this does again fetching some data from base row triggers another workflow which triggers the build here um actually essential you want to take over and explain what's happening here because i know this is uh there's a few things we want to tweak up before this is quote-unquote production ready sure um so yeah so this update workflow i mean so once you have the details from base draw two things need to happen uh one is like triggering the build from github actions and the other is to actually deploy that generated image on one of the cloud instances that we have um so yeah so the trigger build just like uh sends a api request to the github api um to start the image building process and then pushes the image to our docker hub and the rabbitmq node is basically sending a rabbit message to our cloud infrastructure to create an instance with the details that we got from base row so one thing that needs to like a few things that need to be done here before we can actually use this seamlessly in production is to add a weight because the building and like pushing the image takes around 10 to 15 minutes and after that wait we can like trigger the deploy on the cluster or we can if we manage to add a uh hook on github actions that once the build push is completed then we can trigger the deploy as well so two ways to do it uh obviously during our commission we could make changes on master repo so yeah this is pending after um so yeah that's pretty much it we have uh so we have like five instances that are like reserved for these stage deployments and yeah so one two three four five so basically like once you trigger a build it creates an image tagged with the uh instance id and name and id so there's the tags always remain the same but whatever branch is getting selected from the ui it actually creates the image from that branch and tags it with the uh deployment id and then just deploys that image onto the instance we hope it makes sense yeah and so in terms of like roadmap where we see this thing going is exactly kind of what essence described there is fixing it so that um we ideally can get that hook in when the everything's done as soon as we do that it shouldn't be too difficult within it then have basically an async flow that could furnish retool with a status so we can have a little sort of green light let users know that it's sort of in progress or done just like we haven't edited in cloud in my mind as well the cpu and the ram isn't something that can be changed right now by the user it shouldn't be too hard to do and again we could start thinking about having some presets for those things um and right now you can only change a branch we hard coded one or two things but in future this could perhaps be there's a step before this we select if you want a version of an end or a branch and then you could pick the latest version historical version um etc and then from there i really think we would just kind of model the kinds of customizations that we want to do so mutasm has this great little front-end tool i sometimes use for sort of changing i don't know if it's environment variables but various under the hood settings for testing you can imagine there being like a little json payload where you could tweak those things for various testing especially these things that are kind of difficult to simulate like new user or user was added into the power user prompt to ask them for the email these kinds of things that might be difficult to simulate um and the ability to sort of add multiple instances and obviously to make this tip crypto button work for real so that's kind of on the roadmap for this feature when we get some time to do it um i'm hoping because this is something that we kind of would be very useful for me that maybe before the next hackmation we could find a few hours to sort of make these at least mvp tweaks so it's production ready and share this with everyone who wants to be able to spin up instances um so yeah this was poseidon um by myself in essen thanks for doing all of the heavy lifting on the uh on the in from side