How to Analyze Cloud Based Datasets Locally with Python

foreign my name is rileen and I'm so excited to be one of the assembly AI creators helping you count down the New Year in today's video I'm going to be showing you how you can access and analyze large cloud-based data sets locally on your own machine using Python and for today's example we're going to be looking at data from Hacker News and this data set that we're examining is actually hosted on Google Cloud through the public data sets platform now luckily on my own machine I'm going to get started by importing Ibis which is an amazing python library that lets you access data sets wherever they're hosted whether that's remotely or locally through a database like maybe MySQL or postgres let's go ahead and start by importing Ibis to turn on import Ibis and next we are going to go ahead and set ibis. options dot interactive we're going to set this to true and this just allows us to visualize our data as we go along next we're going to go ahead and connect to bigquery and Ibis makes this really simple we're just going to create a variable called H in con and we'll just run ibis. bigquery and we're just going to connect and to connect to our specific project I'm going to call my project ID which is Marlene's uh Dash project and you want to go ahead and call your own one and not mine uh once you've gone ahead and set up your own Google Cloud account I'm also going to say that the data set ID to the bigquery Dash uh public Dash data dot Hacker News awesome and now we're just going to run this to be able to start that connection next we want to access the full table with information from Hacker News so we want to get everything the comments the stories the votes whatever it is so let's create a variable called full and that is just going to call the connect the connection that we made and access the table from that connection like I said we want to get the full table and the database we are going to be using is the same one that we called earlier on so I'm just going to plug that in here and let's go ahead and run that now this just gives us access to that Full Table locally on our own machine and to get a better understanding of what the table looks like let's go ahead and call the head method so we'll just call full. hid in the same way we would do this with pandas it's run that awesome now this gives us some information about all of the columns that are in our table this is a comment here we can see that in the comments is I would rather just have wired earbuds period status again see it contains over 33 million rows and that's a lot of data but it doesn't really feel like we're working with a very large data set it also doesn't really feel like we're working with data that's available on remotely in the cloud somewhere before I go I was also told to give a quick book recommendation and so I'm going to recommend The Alchemist and this is a book that I have actually brought with me right now I'm in Barcelona and um I've brought with it with me to read along the way and it's an allegory about life and how to approach life in a very cool I would say an adventurous way one of the quotes from the book that's right on the back is that to realize one's Destiny is a person's only obligation and I think it's just such a beautiful book an amazing allegory about how to approach life and so would definitely recommend you read it thank you so much for watching my video and happy New Year