# SQL for Data Engineering -  Full Course for Beginners

## Метаданные

- **Канал:** Luke Barousse
- **YouTube:** https://www.youtube.com/watch?v=UjhFbq4uU2Y

## Содержание

### [0:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y) Segment 1 (00:00 - 05:00)

Hey nerds, welcome to this full course tutorial on SQL. This is the video I wish I would have had when I first got into data engineering. To master this tool, we'll break down concepts into short 5 to 10 minute topics. You'll work right alongside me completing real-world exercises to learn not only SQL, but also other in-demand skills like terminal, Git, and GitHub. After each short lesson, you'll have the option to tackle interview-level practice problems that prep you for the job and reinforce your learnings. And by the end of the course, you'll have built two fully customizable portfolio projects that showcase your skills in building and analyzing a production-level data warehouse pipeline. Now, SQL is by far the most popular tool for data engineers. If we look at recent job postings, we can see that it's in two out of every three postings for data engineers. And it only increases in demand for those senior roles, making it even more of a vital skill to learn first. Now, SQL or SQL is the language used to communicate between you and a database. It's by far my most used tool as a data nerd. Starting with my first job working for a global Fortune 500 company, all the way to my most recent job working with Mr. Beast. Yeah, even Jimmy uses it. More recently, I've been using it in my data pipeline for my app, datanerd. tech, but more on that in a bit. So, over the years, I've cataloged everything that's essential in using SQL and bundled it all up into this course. Now, to be clear, I've made this for beginners. There's no coding, terminal, or even engineering experience required. I'll be teaching you everything you need to know step by step. For this, we're going to break up the course into two halves, with the first half focusing on the fundamentals you need to learn to feel confident writing SQL in a real data engineering workflow. We'll start by getting your environment set up and making sure you can run SQL both locally and in the cloud. Then, we'll introduce core query building blocks such as filtering, grouping, aggregation, and sorting data. From there, we'll shift into understanding how data should be structured in warehouses with dimensional modeling. And throughout the sections you'll use the same tools real teams use, a VS Code, the terminal, and Git and GitHub. At the end of this first half you'll build a portfolio ready project where you explore a real data warehouse. In it you'll write SQL scripts to answer stakeholder style questions and your analysis will be published to GitHub. And this will ultimately help you start thinking like the stakeholders you'll be building data pipelines for. Now in the second half of the course we'll shift into writing SQL the way data engineers use it in production. We'll start with how they build and evolve warehouse tables using create, alter, and drop. We'll then cover how pipelines load and maintain data with insert, update, and delete. From there we'll focus on data modeling techniques, especially star schema design of fact and dimension tables, so your warehouse is built for analytics and downstream use. And finally you'll level up your SQL with CTE and subqueries for clean modular scripts, plus window functions for real warehouse analytics and pipeline logic. At the end of this half you'll build a production style end-to-end SQL pipeline, designing the tables with the correct schema, and then loading the raw data into it, and then producing data marts for specialized use cases that can be accessed by a multitude of tools, and prove your job ready data engineering skills. Now I believe in open-sourcing education, so this entire course is completely free. You'll have access here on YouTube to every video lesson and can follow along with exactly what I do. And everything we're using to run SQL is completely free and open source. Oh, and I've even made both final projects publicly available so you can compare your work to it. Now for those that want to support the course, I have some perks that add structure, practice, and certification. First you'll get access to your own course dashboard that tracks your progress. In here it lets you watch each lesson individually in short bite-size segments. Then after each lesson you'll have access to additional SQL interview level practice problems that provide a detailed solution. However, you get stuck, you can leave a comment below and get help from other students. On top of this, I'm giving you all my course notes that walk through the video lessons step-by-step, and you'll get access to my custom cheat sheets breaking down key concepts. When you complete the course, you'll receive a certificate of completion that you can post directly to LinkedIn. So, if you want to support and bring to life more courses like this, use the link in my description to get access to the supporter resources. Now, I have two SQL courses. SQL for data analytics, my first SQL course, it's 4 hours long and ends with an analytics focused project. And then this course, it's 14 hours long and it has a similar analytics project, but then an engineering focused project as well. So, which one should you take? The simple answer is data analyst should take the first course, and data engineer should take this one. Now, neither course is a prerequisite for the other. Both courses teach the same SQL language, but they implement it differently. SQL for data analytics focuses on querying existing data to answer business questions, and then this course starts similarly, but it ultimately focuses on building

### [5:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=300s) Segment 2 (05:00 - 10:00)

production level pipelines, framing this all from an engineering perspective. If you've taken SQL for data analytics and want to take this one, go for it. You'll still benefit. Early sections may feel familiar, so skim ahead if those concepts are already solid. Now, this course was a behemoth to make. It's over 14 hours long. It took me the greater part of last year to make, and it took a team of four to put it all together. So, I'd be remiss if I didn't give a shout-out to these teammates. For this, we're going to do it the office style. Ricky, my content developer, did a lot of work behind the scenes putting together those SQL interview practice problems. Brandon, my video editor, helped condense down about 100 hours of video footage into 14 hours. And finally, Kelly was the producer of the course helping out with both lesson and interview practice problems. So, shout-out to my team for helping me bring this together and make this course possible. All right, one last thing before we jump into the course. If you get stuck along the way, I highly recommend you use any popular chatbot as they're more than proficient in SQL and a heck of a lot faster than waiting on me to reply. Anyway, enough of me yapping. Let's get into this course diving into our first concept to understand what SQL actually is. All right, in this lesson we're going to get you up to speed in understanding everything you need to know behind this powerful tool for data engineers. To understand this, we need to grasp two major concepts. First, where do you write and execute your SQL queries? And secondly, your database, where your data's actually stored. So, let's dive into that first. A database is a collection of data and these things were made to hold a massive amount of data. An Excel file spreadsheet can only hold around a million rows, but a database can easily hold way more than that, 9 quintillion rows. Now, about those databases, there are two main types of relational and non-relational. We'll explore each type more in a bit, but first let's go back to where we're actually running SQL. When you write this code to request data from a database, that's called a query and SQL can do a whole heck of a lot. A common acronym associated with this is CRUD and it stands for create, read, update, and delete. With SQL keywords like create and insert into, you can add new records. Then with keywords like select, you can retrieve specific data you want. Next with keywords like update, you can modify existing records. And then finally, the scariest of all, if you need to remove records, you can delete it. Now, these commands fall under three types of SQL commands. DQL or data query language is used for the reading aspect of CRUD and is what we'll focus on the first portion of the course. Next is DML or data manipulation and this is used for creating, updating, and deleting new data. Specifically in tables that already exist. Then we have DDL, which is data definition language. And this deals with creating or deleting the schemas and tables we need from scratch. So understanding this, we can now understand how SQL actually works. From your computer, you write a SQL query and send it to a database. The database then performs your CRUD operation and then provides you a SQL output depending on what you did to your computer. But this now begs the question, where are databases even stored? Well, there are two main options. You can either run it locally or in the cloud. Local means the database runs directly on your computer. This is ideal for data engineers to iterate quickly without affecting production databases. However, others can't access it. Now, production databases are usually stored in the cloud. This enables your entire organization to connect to the data source you built. You probably heard of popular cloud providers like Google, Amazon, or Azure. Anyway, we're going to be using a combo of both local and cloud for this course. And in the final project, you'll be building out a data warehouse and data mart locally and then push it to the cloud to make it publicly available. So, now that we understand that databases can be stored on your computer or in the cloud, what are the different databases we can store in these locations? Well, two main types of relational and non-relational. Relational databases store data in rows and columns. Very much popular in analytical applications. Say I have a database of job postings that companies go to daily to update on what jobs are available. You could have one table to keep track of all the different jobs. And then you have an associated table to keep track of all the different companies. There would then be things like a company ID within the two in order to relate the different tables. Hence, relational databases. Now, this course will primarily focus on using SQL to interact with relational databases. But it's important to know the other type of database of non-relational databases, commonly known as NoSQL. But, NoSQL doesn't stand for not SQL, it stands for not only SQL. These support storing your data in a variety of different forms, including key-value paired, column-based, graph-based, and even document-based. All right, so we understand the types of databases now. Because of that, we can get into

### [10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=600s) Segment 3 (10:00 - 15:00)

narrowing down what database we're actually going to use for this course. Well, I needed it to meet a few criteria. First of all, it needed to be a relational database that was also highly popular, so it has a community for support. From there, I started to narrow it down to a database focused on analytics to make it support building data warehouses. After this, I narrowed it down to free and open-source options so that anybody taking this course can use the database. And finally, with that, I needed this database also to run locally first, along with the cloud. And so, this gave us DuckDB. DuckDB is a blazingly fast database that satisfies all of our needs. So, we're going to be using it for the entirety of this course. We're not only going to get you set up with running it locally on your own computer, but we'll also run DuckDB in the cloud by using MotherDuck. This is a managed cloud platform that will be completely free to use as well. Now, this brings us back to the last concept we need to cover of where we're going to be writing and running these queries. Well, that's commonly done in an editor, whether that's database provided, cloud provided, or just a stand-alone code editor. Now, database-provided ones provide you with an application to use on your computer to access their local database or cloud-based one. This is a popular option from Postgres called pgAdmin, and inside of it, it even allows you to write SQL queries. For data engineering work, I'm not a fan of these. Next up are cloud-provided editors. Typically, you access them in a web browser, and they allow you to interact with your database. We're going to be using MotherDuck's cloud-provided editor in order to connect to our needed databases for the course. We'll actually do this for the first half because it's super easy to get up and running for you. Last up are code editors and this is what we'll focus on for the majority of the course. We're going to be using a popular option of VS Code along with the terminal, which allows you to build all your SQL files locally, but then interact with databases that are either local or in the cloud. You'll be running SQL queries like a pro data engineer after this. All right, so now we understand what SQL is and what tools we're using for the course, but it now begs the question, what data are we going to be using and what pipeline building for this course? Well, covering that next. Welcome to this lesson. We're going to be diving into understanding what data and also what pipeline we're going to be building for this course. But, before we talk about any of that, we need to understand what problem we're trying to solve. And it's a real problem that I had. This all started a few years ago when my subscribers were asking, what are the top skills they should be learning in data? And the best way to answer this analytical question is to look at where these skills are demanded in job postings. So, I built out a data pipeline that solves this. It pulls online job postings from a source and then using SQL cleans up and transforms this data. Finally, it then loads it into a data warehouse. Now, I'll go into this data source and how I built the pipeline in a bit. But first, what's a data warehouse? Well, in order to qualify, it needs to be subject oriented, non-volatile, integrated, and time variant. If it doesn't have these four things, it's just a database and not a data warehouse. This is all from the father of the data warehouse, Bill Inmon, and we'll be going over this definition in much more detail as we build out our projects. Anyway, for our final project, we're going to be replicating what I did by taking raw job postings and building out a data warehouse and also data marts all with SQL. So, what does this data look like inside of our data warehouse? Well, here is an ERD or entity relationship diagram and it shows the different tables in here. We have four major tables. One fact table containing all our different job postings, then a dimension table around the different companies that are posting jobs, and finally dimension tables that contain key information about the skills required for these jobs. Now, if you haven't heard about fact tables, they contain the core data for our analysis. They measure and record actual events, so in our case, the different job postings. The dimension tables describe attributes or dimensions of the data. In our case, skills and company information, and this is really important in supporting filtering and grouping different sets of data. Anyway, that's all theoretical. Let's actually dive into my machine and look at the key attributes from the data. So, querying this data warehouse, we can see that we have 10 different jobs available to look at job postings for. Data analysts, data engineers, and data scientists are by far the most popular, but there are other options available. And with that, these postings come from around the world in over 160 different countries. So, the country you live in probably is in this data set. And this now begs the final question of what platforms did we get all these job postings from? Well, a ton of popular options like LinkedIn, Indeed, and even ZipRecruiter. So, you're probably wondering, where the heck did this data come from? Honestly, it wasn't easy. Back in 2022, I wanted real data on what skills actually get you hired in data

### [15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=900s) Segment 4 (15:00 - 20:00)

jobs to provide to my subscribers. Not opinions, not blog posts, but real job postings. So, I did what any data nerd would do. I tried to scrape LinkedIn. So, over the course of a few days, I built a scraper, learned just enough front end to make it work, stayed logged in with my LinkedIn account, and surprisingly, it worked. Well, for about a day. Then the captchas started to hit. Then my account got banned and it became obvious real fast job posting data is extremely valuable and these sites protect it aggressively and this is a very real data engineering problem. Scraping at scale quickly turns into an infrastructure nightmare. So at this point I thought the project was dead but then I happen to stumble across SerpApi and they had exactly what I was looking for. Instead of scraping LinkedIn directly SerpApi gave me access to job postings at scale through search engines. So SerpApi can pull these jobs from Google that ultimately come from different platforms like LinkedIn, Indeed, and ZipRecruiter. So with this I wasn't fighting captures anymore, managing accounts, or building infrastructure. On a whim I reached out to SerpApi, explained my problem, and asked if they could support the project. You've got mail. Surprisingly the CEO himself replied and they agreed to support the project. So I immediately started by jumping into their platform and building out the code in their playground area which lets you test out different queries and see the results. Clicking search I can see the search results that it gets back in this browser view but what's actually more useful are these structured results that I get back. For example with this first one I can see it's a senior data engineer role in Washington D. C. scraped from LinkedIn. It tells me that it is not only full-time role but also in the job description I can see it list a host of different skills. This right here is the jackpot. Oh and some jobs even have more information like this one with salary. Honestly this is way more detailed than I could ever have gotten by scraping on my own. From there I can grab a code template for any popular language or tool such as curl, Python, or Google Sheets and I can run it locally or in the cloud wherever I need it. Now to be clear SerpApi has way more than just job postings. They support over 50 search engines, everything from things like Google Flights to Google Maps to product listings on sites like Amazon. And oh, they even have an ability to look at YouTube. If I wanted to, I could scrape all of my YouTube data and get it back in only a few seconds. Oh, and if you need responses in under a second, they also offer add-ons for that. Now, back to my job data. I've been systematically pulling these job postings on a daily basis, which is how I've collected almost 5 million jobs over the past 2 years using SerpApi. All the data can be accessed via my site, datanerd. tech, and it's solved my problem of figuring out what skills data nerds actually need to get hired. It's how I figured out that SQL was the most popular skill of data engineers. And all of this job posting data that I've collected over the years will be using throughout this course in order to learn SQL and [clears throat] ultimately build a SQL-based data pipeline. Now, to wrap up, I want to give a huge shout-out to SerpApi for sponsoring this video. Use my link in the description and tell them Luke sent you to get 20% off. They even have a free tier with 250 searches per month to try it out. Now, enough of me yapping, let's get into setting up SQL on our computer so we can start playing around with it. All right, see you in the next one. Welcome to this lesson where we're going to be not only going through setting you up with the environment in order to execute SQL, but also get you set up with the actual database that we're going to be using for the entirety of this course. And jumping into my computer to show you this, we're going to set you up in the browser in order to be able to execute SQL queries. Here I have the results right below here. And you're going to have this environment here that we're working on in the first portion of the in order to investigate and learn some different SQL syntaxes we go through this. Now I'm going to get you all set up to be able to execute SQL in your browser and with the data set. Before that, we need to understand some concepts of actually where we are running this SQL query. Well, as you can recall from the last lesson, we're going to be using the free open-source option of DuckDB for this course. And this is a very lightweight database that you can run right inside of your computer. Now, DuckDB is relatively easy to get set up and installed on your computer, but it requires one pretty big caveat. If I go into the installation guidelines, it tells you how to install it and specifically it has you installing it with a terminal. And I'm assuming for a lot of you taking this course, you don't

### [20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=1200s) Segment 5 (20:00 - 25:00)

have much familiarity with running commands in the terminal and there's quite a few to get set up with not only DuckDB but also the data set. So because of that, I made an executive decision when making this course that we're going to start out working in this web browser environment that has DuckDB on the back end and this is called MotherDuck. Now, MotherDuck, which you can visit via this link here, is the commercial SaaS or software as a service platform that supports running DuckDB in the cloud or as if you will in your browser. DuckDB or DuckDB Labs owns a significant equity stake in MotherDuck. So it's very beneficial for both of them to work together and collaborate. Mainly, they keep it separate so DuckDB team can focus on improving the database and keeping this free and open-source and MotherDuck is the commercialized ver- version that actually makes money to keep DuckDB Labs funded. And the probably the most important part that is relevant to you, MotherDuck allows us to host DuckDB databases on the cloud, basically have data warehousing capabilities. And then you can use other popular services in order to connect to these DuckDB databases inside of MotherDuck. For me, coming from a data analysis background, this is pretty imperative because tools like Tableau and Power BI can connect to a single source of truth inside of MotherDuck. And there's a host of other data engineering tools that will eventually get into later on in the course, that can connect to it as well. So, to recap, DuckDB is a free open-source software that you can run either locally or on another server. Whereas MotherDuck is the platform that allows you to host these databases in the cloud, basically host data warehouses. And honestly, I wouldn't have known about either of these service if it wasn't for Medhi. You can check him out at the YouTube channel here. He heard that I was building this course on SQL for data engineering, reached out and connected me not only with the MotherDuck, but also the DuckDB team in order to put this course together and make sure it was successful. So, huge shout-out to him. Now, let's get into setting up your MotherDuck account. And for this, we're going to be on the free tier. We're going to remain on the free tier for the entirety of the course. And the good thing about MotherDuck is there's no credit card required. All you got to do is just sign up. For those complaining that you got to sign up for a service in order to take this course, as a data engineer, you're going to be signing up for a lot of services and a lot of them don't offer free tiers. So, consider yourself lucky with this one and get used to it. For this, you're going to navigate to the URL on screen of lukeb. co/motherduck. Now, the MotherDuck team has built this special login for us and the course takers of this, so that way we can skip the normal marketing emails that you get if you logged in through motherduck. com. Anyway, you'll sign up by either putting your email below, or I recommend using something like GitHub or like I'm going to do Google. From there, go through the authentication process with Google and your associate Google account. From there, insert your name, organization. I just put Luke's organization. For me, I'm located in North America. That's where I want the region to execute my different queries. And then from there, click continue. From there, you may be asked a few questions about your background. Feel free to just fill it out or skip where applicable. And then from there, it has a let's get started. I'm going to go ahead and just click all of these and say that we did them cuz we're going to go through them in this. So now, we're inside of the environment where we're going to be running SQL queries. Take note of this uh URL that where we're going to be at for executing this so you can come back to it easily. Let's get into understand the UI or user interface of this MotherDuck experience. Now, it's broken into three major parts. Over here on the left is what you'll hear me call the left sidebar or the sidebar. In the middle right here is our SQL notebook, and we can actually add different notebooks as well in that sidebar. But that's where we're going to be executing those SQL cells, which we'll get into a little bit. And then, we also have on the right-hand side something called an inspector, which we can go into inspect data sets. I'm going to leave that open for the time being so that we can get into that. I'm going to go back to that MotherDuck notebook right here. Let's dive into that left sidebar a little bit further. They have capabilities to do search. So if you have certain terms, you can search for it. We can also create tables from files. Then in that notebook section, I can either hide or show it. We just created this untitled notebook. I'm actually going to rename this notebook, and we'll just name it after what this section is. Specifically, we'll call this SQL and database setup. And then I'll click rename. Feel free as we go through these different lessons to create different notebooks to keep track of what we're doing. Underneath these notebooks are our attached databases, And right now, we only have two. We have a my database

### [25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=1500s) Segment 6 (25:00 - 30:00)

and nothing inside of it. And then we have some sample data sets. And we'll explore one of these to explore a notebook here in a second, but I want to call out this. Down at the bottom, you may see something this of, "Hey, your trial ends in 21 days. " Don't worry about that. You will stay on that free plan and the course will or this platform will be continued to be free to the entirety of this course. You're not going to use enough resources to need to upgrade. So, don't worry about that. So, I'm just going to minimize this. Before we dive into those databases, it's important that we understand how hierarchy works in a database, specifically for tables and different data we have inside of it. So, for a database, you can have one or multiple different schemas. We're just simplifying it here, but only allowed to have certain tables within that schema. So, schema A in this case could have table one, two, and three, such as users, orders, and products. And then table or of schema B could have something like employees and products in a separate one. You can keep it separate from each other, but in the same database. We'll apply this to our data set that we're going to be using for this course in a bit. But, I wanted to go over that because if we look inside of here, we have something like my DB and it has the schema of main. Similarly, underneath the database of sample data, it has 1 2 3 4 5 six different steam schemas. And then if I go to something like the Stack Overflow survey, it has two different tables inside of that schema. So, what I'm going to do is I'm going to go to that survey results of the Stack Overflow survey. And what I can do is click these three dots right here and it allows me to query in the current notebook. I'm in the Welcome to Motherduck. That's fine. I'm just going to go ahead and do it. And it puts it down here at the bottom. Now, we're going to get into actually understanding the SQL dialect here in the next lesson, but I wanted to show you that you don't even know any sequel in this program and that you can just query it using the actual UI and it puts in all the relative information, all these different column names for this data set. There are a lot of columns in this one. And then down at the bottom it provides a table view of this and I can go through and see all the different values in it and also scroll down because it returned I can I see from up here 100 rows returned and 374 milliseconds. DuckDB is super fast. If I wanted to, I could copy the results to my clipboard or even download the results to some different file formats. Now what's cool about this UI that I really like is that over on the right hand side in this inspector window, I can go through and view all the different columns right here and it provides some EDA or exploratory data analysis for me to actually dive into it. So looking at something like the country column, it tells me that there are 28 distinct values and it shows a bar chart with this how it relates to the other columns and how many distinct values it has and we can scroll through and see any different column we want. This has like I said, this has a lot of columns in it. So this is great for someone that wants to actually look through a lot of different columns. Now besides viewing columns right here on the inspector, you can also view it down below here. If I just click survey results right here, it will appear right underneath it and once again, I can actually in this case we can see that there's a difference here, right? This one because we only queried 100 rows, 100 of all the different rows that they have, it only showed 28 columns. But over here on the left because this one actually queried all 615,000 rows, it has different results and it says we have 264 different countries. So make sure you understand what we're actually querying whenever you're viewing this neither one of the windows. Anyway, you can also go into and dive into seeing a deeper dive on any one of these as well by just double-clicking on it. So, if you notice when we're going through that, none of that jobs data that we're going to be using for the course is located in here. We actually need to attach that database so that way you have access to it. So, what we're going to do is we're going to navigate over to this notebook that we're going to be working in for the remainder of this lesson that's on that 1. 1 SQL database setup that we created. Inside of here, inside the notebook, we're going to add a cell. Now, we're going to run an attach command in order to attach a database. Specifically, we have to specify this long line in there. So, I'm not going to expect you to actually type that in. I'll include it in the description below for you to just copy and paste into here. And this is actually specifying that data jobs database with an unique identifier. From there, we're going to run our first cell by pressing either this button up here, this play icon, or we can press in my case when I'm on a Mac, I'll press command enter. Now, we can see up in the left-hand corner is my data jobs

### [30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=1800s) Segment 7 (30:00 - 35:00)

database. It has only one schema in it of main and then four different tables. Company dim, job postings facts, skills dim, and then skills job dim. So, in our case, this is what the database hierarchy looks like. We have our schema of main and then those four tables underneath that schema. This is all within the data jobs database. Now, going back to this diagram you saw earlier in the course, this is how all the tables are related. We'll have a separate lesson on how to actually read this diagram, but the main thing to understand here is that this job postings fact table is our main fact table and then these other three tables on skills and company provide skill and company information and they are a dimensional table. So they provide extra attributes, but the core of the information is in our fact table. So for the next few lessons, we're going to be spending the majority of time in that job postings fact table. Once we get some basic SQL down, we'll get into how these actually relate, how to actually join them, what the heck this PK and FK actually mean. So let's actually inspect this data. So I'm going into job postings fact, selecting these ellipses here, and then go into query in current notebook. Now some other things to note with this, right? We have up in the right hand corner, it tells us which database we are connected to. That first one up here was connected to my DB, but that's okay. It's an attached database, doesn't really matter which database we're to attach it cuz it wasn't there yet. Anyway, it's really important to understand which one we actually have right here to actually execute the query cuz it may prevent you from executing it. Anyway, from there, it also provides underneath it those 100 different results and the different information. What I want you to do cuz there's no practice problems for this lesson, what I want you to do from here is just spend some time inspecting this table, understanding the data set itself. Also, feel free to use the table explorer in there to actually dive into and see these different columns and explore them further. Specifically, you could do something like dive into what's this job title short column. You could see all the different job titles available and what amount they actually take up for the data set and how many rows. So that wraps up this lesson. You should have signed up for MotherDuck, got into the UI, and attached that data set. In the next lesson, we're going to be diving deeper into understanding the basics of SQL syntax. So you can actually understand these SQL queries that we just ran here. With that, see you in the next one. Welcome to this lesson on basic keywords. By the end of this, you should be able to well, understand the basics, but be able to read basically any SQL statement from here on out because a lot of the keywords that we're going to be covering in this are used time and time again in SQL queries. Real quick plug before we get into it, and that is to my course notes that you get in the supporter resources. If you haven't purchased them already, they actually go through and describe everything that we're going to be doing in an associated lesson like this one, and then break down the lesson step by step with even more detail. Anyway, I digress. Let's get into it. And the first keyword or keywords we're going to be covering are select and from. And this is how you actually read this. Select identifies the columns or data from a database, and then the from statement or free from keyword identifies the tables we are connecting to. Now, the asterisk in this case is a special command that says, "Hey, we want to select all the columns, everything. " So, in the case of our job postings fact table, we could do select star to select all of this from job postings fact. And showing this visually, we can execute this to get the results. Let's actually do it for real. So, inside of MotherDuck, go ahead and create a new notebook. I'll call this 1. 2 basic keywords. I'm just going to add a new cell, and conveniently it automatically selects our last database that we're using of DataJobs. For this, I'm going to type in select and then star job postings fact. From there, I can press play or use these different shortcut commands to run the cell. If you're on a Mac, you can press command enter. On Windows, I think it's control enter. And this outputs all the results down below it. And it even says that it returns the first 50,000 rows in 926 milliseconds. Now, this is not all the rows of data. Thankfully, it caps it off to sort of limit the data. But if I scroll over here on top of job posting facts, I can see that our data set has over 1. 62 million rows. One quick note on the syntax, you notice I use uppercase letters here and lowercase letters here. That's pretty standard.

### [35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=2100s) Segment 8 (35:00 - 40:00)

But if you wanted to for the keywords itself, you could use actually lowercase letters. And in this case, it's going to still return the same result. So, I can change also change from, press command enter, and bam, same results. I can also change the table name itself to be all caps and still going to run. But specifically for table names and stuff, it's very important that you stick with that syntax of the lowercase because many databases require it to be what it is, the same syntax. And for keywords, it's just best practice to use these as all caps. It also makes it a lot easier to read. Now, of our job posting table, let's say I want to select specific columns such as job ID and job location. Well, instead what I can do instead of using select star, I can list out the columns or column names and then still leave that from of what the table is. So, back inside of Mother Duck, I'm going to put in that job ID. I'm going to just run this real quick. You can select one column. If you want to select multiple different columns, you would need to enter in a comma and then from there add in the next one that you want to use. Oh, let's say job location. And you can see we have some hints coming up as we type something. If I know, so I do know I want job location, I'm just going to press tab and it's going to accept that. Anyway, going to go ahead in here and press command enter. And now we're getting that job ID and job location. Now, I like this a little bit more readable, so I'm actually going to enter job ID and job location onto the next line. and then from there I'm going to press tab to indent each one of those over. I can also do that with the from statement here and press tab as well. Now, this does not affect the execution of the query. I can still run this. It provides the same results. I can also do multiple different indentations on different lines just like this. And when I press play, it's still going to run. It doesn't really matter. This is all just for my readability. And also if I make a good query, it's for my co-workers readability. Now, it's very resource intensive to the computer itself if we were to continuously always pull all the columns from a data set. So, in this case, if we wanted to just pull the top three rows, we'd need to use a keyword to limit it to that. Well, conveniently, we can use limit. And limit goes at the very end of your SQL query. And you specify the number of rows that you want from this. In this case, we're going to return those three rows. So, inside of our query, I'm going to go in and put at the very end and put a limit of We'll just do 10. Press command enter and run this and only get 10 rows returned. Anytime you're building queries on large databases, you need to be using this keyword. You can always remove it at the end after you built it and verify that it's working properly. You'll save your boss a lot of money. So, going back to this diagram where we show the database container, its schema, and the tables and the hierarchy for it, we can specify any one of these things assuming they're in the appropriate order. So, right now we have the database and then the table. Remember, we also have something like the schema itself. So, you could specify the schema in here as well. Whenever I run it, runs just the same. Updating the other portions of the query for this, it works just fine. Now, in the more advanced lessons, we'll get into when you need to put this nomenclature inside of here, maybe whenever we're joining them tables. But for now, I just want you to have that in the back of your head. Next up, let's say we have this scenario where we want to look at distinct values or unique values inside of a column. Take for example, our job title short column. In here, we have things like data analyst, data engineer, data scientist. Anyway, we got millions of rows, we want to get the distinct values out of there. Well, we can use the distinct keyword right after select and it specifies, "Hey, we want distinct values from job title short. " So, for this, I'm going to select distinct values of the job title short column. And we're be getting that from that job postings fact table. Running command enter, scrolling on down, we can see we get 10 rows returned. And just so you understand the concept behind this, originally, I'm actually going to remove this distinct keyword right there, and I'm going to throw in the job title column. First, command enter. The job title column is the actual name listed on the job description. And it can get pretty verbose. And so, if I wanted to pull out like all data analysts or all business analysts, it could be a hassle doing this. So, I created this job title short column in order to make it easy for you, for analysis, and building pipelines. Now, with any of these queries, you want to end them, it's best practice to end them with a semicolon at the end. Running command enter. Still going to have the same results. And I can even put into something like a second query underneath this for those distinct job titles. Now, whenever I run this one, it will execute both queries, at least here, but only one of them will

### [40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=2400s) Segment 9 (40:00 - 45:00)

return. And I can see that by saying, "Hey, 10 rows returned, but two statements ran in 3. 5 seconds. " When we get into building out data pipelines towards the end of the course, these semicolons are going to save your butt. Next up, let's say we want to get specifically filter job titles. In our case, we're going to get them where they have the job title short of data engineer. Well, in our case, we can use a where keyword, and we put that directly after from, and in that we're going to specify being via an expression saying, "Hey, we want from the job title short column values equal to data engineer. " And it will provide those filter results. So, inside of our query, we're already selecting the job title short and the job title, and we want to inspect further what are some other job description names for data engineers. I can then press enter. We'll put in that where statement, and we'll say that is for the job title short column with it equal to data engineer. Now, I'm going to go ahead and execute this and press command enter. We're going to get an error here. It says, "Parser error, syntax error at or near engineer. " The problem, as we're getting help from AI inside of Motherduck, is that we don't have these single quotes. You can actually just if we want to accept and run by clicking this or pressing, in my case, command enter, and it automatically updates it for this. And then inspecting it further, we can see all the different job titles for data engineer, and they have a lot of different random ones. With this syntax structure of the where statement here, it doesn't matter. I did those spaces in the in there for readability. This is still going to work like this, but I like to have space in between just make it a little bit easier. Now, the single quotes is not applied to everything. That's just text values. Let's say I'm going to add in the salary year average column and go ahead and run this bad boy. And then for this, I want to filter that salary year average column for, let's say a salary value. We'll say salary year average equal to let's go with something like 100,000. Press command enter. And now scrolling in through this, we're seeing all these different jobs that come in at $100,000. In this case, we did not put any quotes around this because this is a no number numerical data. And if we're actually curious to learn what's text data, what's numerical data, we could actually go into job postings fact by double clicking on it. And over on the side, we can see that the different values of has different things. Specifically for salary year average, we can see that it has a number next to it and also it calls out that it's double. Double is a type of data type that we're going to go into in a future lesson. But something like job title is a text character by this text symbol. We can see that it is the type varchar. And we're going to change things up a little bit not necessarily go over a keyword in this case, but a special marker that you may see in data sets or that you will see in data sets. Actually, I'm going to remove this where salary year average statement from here and then press command enter to run this. And if you remember from what was returned or if you actually do it, you can see that salary year average column, it has null values. And null is a special marker where a field does not have any value in it whatsoever. It's not zero. It's not anything. It's null. So it's very common that I want to uh filter for this. So I'm going to add back in that where statement that we had. And I want to filter for in this case, null values. Now, you would think you would want to do hey, we want to do where salary year average equal to null. But if I go to run this, press command enter, I'm going to get zero rows returned. This didn't work. What you actually have to use to specify that you want this null is this operator of is. Running command enter, now we actually have the result of all the different null values here. But, if you're like me, you don't really care about those null values. You actually want to get values that are not null. So, if you can use this is operator, you can also use this not operator. Running this, we're filtering now for salary average that is not null, and now we have nothing but numerical values in here that we can go through. Now, don't worry if you're getting too freaked out by these operators of the equal sign, is, and not. We've got a whole lesson on it coming up next. Now, another concept to cover that's not a keyword is commenting in code. Let's say I build this spectacular query, and I want to share it with others, but they may not understand what it is, or even how to read it. I can put comments inside of the code, specifically by

### [45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=2700s) Segment 10 (45:00 - 50:00)

putting these double dashes. If you're in the military, double tax. And I can write something like, this is a comment. even see inside of here that it's formatted differently in this italics. But, when I run this, it's still executing. And this doesn't matter if it's up on the top, off to the side, or even down at the bottom right here. No matter what, it's going to work. What I really like about this, too, is anytime I'm troubleshooting code, I'll usually use a commenting to comment out. So, I can comment out like this line right here. Now, when I run this, I only get the job title short and the salary average column. Now, let's say I have a multi-line comment going on here. This case, I have a description of what this query does, of what fields it likes, what it filters, and shows some cleaning results. Anyway, this is really annoying to have to go through and execute these comments every single time inside of here. And so, what I could do instead is use a multi-line comment. It has a forward slash, and then an asterisk. And if you notice by that, everything in this block is now marked as a comment. So, I need to mark when that comment actually ends by using an asterisk and then another forward slash. And that stops it. Now, whenever I run this, command enter, boom, I have it all below and I don't have this all commented off and it makes it a little more readable. The last keyword that we're going to cover in this is order by. And we can, as shown in this example right here, we have a column of salary hour average. We could sort the entire data set or whatever we select by that column using the order by command. In this, we're going to put order by at the end of our query, but before our limit, if we have a limit on there. And we can specify the column that we're going to do. And automatically, it's going to do this in ascending order. So, for this, I've selected the job title short column, job location, job via, and salary year average. And we're going to be going from the job postings fact table. And we want to order by, so I use order and then by, and then enter in that salary year average column. Running this query, we can see that the salaries in this case start at around 15,000 and then work their way up. So, it's in that ascending order. Now, conversely, we can also look at it in descending order. And we do this by at the end of the order by specifying d e s c or descending. So, inside of here, I can just type in d s c e yeah, d e s c. Press command enter and it sorts it into that descending order starting with that first and highest salary of $960,000. Wowzers. Now, just a note, you can also specify a s c for ascending and provides the same results that we saw previously, but ASC is redundant because you should know that it automatically does it automatically, so we don't include it on there. Now, there is a certain order or you'll get errors that you need to put these keywords in that the machine actually expects them to be in. Select and from gets pretty easy and redundant, but when it gets to these other keywords, some of which we haven't even covered yet, such as this group by and having, it takes some trial and error, but over time you do understand what type of order this goes in. It just sort of takes trial and error. Now, the order that you write commands is important. What happens if I wanted to, in this case, take that order by, stick it right here, and then try to run it? I'm going to get this. We get a parser error, syntax error at or near from. And even the AI auto suggestion is not actually fixing for what it needs to actually do in this case. It thinks we need to change the order by to a from. This is why it's important to understand that you need to understand that there is an order to write commands. Specifically, this order right here. Select, then from, and where. Group by, we haven't covered yet or haven't covered by yet haven't yet. And then order by and limit. So, let's say we wanted to get the top 10 highest paying jobs for data engineer roles specifically. Well, the first keyword we know you can use is where. So, we'd specify that job title short equal to data engineer. After the where, we know we can then also specify the order by. And in this case, we want to get that salary year average, and we want it from highest to lowest, so we need to specify descending order. And then finally, we'll put in we want those uh we want the top 10 results, so we'll do limit of 10. And for good print measure, we'll put in a semicolon, press command enter, and we're not getting anything. That's cuz I fat-fingered engineer. I'm not spelling it right. I need to put an N after this. Actually, running this again. Okay, [snorts] now we're getting those top 10 jobs. Now, you routinely switch this up and always will sometimes put the order

### [50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=3000s) Segment 11 (50:00 - 55:00)

by before the where, and you'll go to run this and you won't get this. And it's really important that you understand the ordering because the actual syntax message isn't very helpful at all in helping you fix this. So, I'm going to go ahead and just move that down below. Running this again, command enter. We have the results that we want. Now, throughout this course, I'm going to refer to something called DuckDB friendly syntax. And basically, DuckDB is super great cuz it's not only fast, but they also have a lot of ways to make it a lot easier to write your code. But unfortunately, a lot of the ways that you can write your code in DuckDB, you can't actually write in other databases. So, that's why I call it friendly SQL. One quick example of this is if say I want to investigate that company dim table, and I click the three dots and go query in current notebook. If you dive into this query right here, you see that in fact well, one, they wrote in lowercase, which is crazy. But, two, they put the from statement up at the top. And if you remember from my order to write commands, from is clearly below select. Now, this generalization is applied so that you understand it, is applied to all databases beyond just DuckDB. But, DuckDB has this friendly SQL that allows us to actually do this and put the from before. So, that's why it's a little bit different here. So, from time and time again throughout this course, you'll hear me refer to certain things as, "Hey, this is a friendly SQL syntax that only DuckDB allows. " Because I really want you to come out of this course being able to be database agnostic and be able to work in any single database. All right, for those that supported the course, you now have inside our dashboard, you have some problems to go through and get more familiar with operating all these different basic keywords. The last problem gets pretty spicy, so good luck. With that, I'll see you in the next one where we get into operators. In this lesson, we're going to be getting into operators. using this to perform some exploratory data analysis on our data set. And we have two type of operators we're going to use of comparison and then logical operators in order to uh further deep into it. Let's get into it. So, let's just briefly cover both. And first up is comparison operators. These are the foundation of data filtering in SQL. As we saw in the last lesson, using the equal to operator. Now, we're going to be diving into these in this table with the exception of is null and is not null we covered in the last lesson lesson. And the like operator we're going to cover in the next lesson on wild cards. The other type of operators are logical operators and they allow you to combine multiple conditions to create more sophisticated queries. Instead of having to run multiple queries, you can combine it into one. We're going to be looking at and, or, and not for this one. Now, in this lesson, we're going to be using it specifically in that where clause because we covered that previous example and it's the easiest one frankly to get an understanding of the concepts of these operators. However, I want to be clear in that you can use it and we will use it in the future in other keywords such as having, joins, and cases. So, these operators are not going away. They're going to be the foundation of what we need to use for the remainder of the course. So, for part one of this, we're going to go just cover the easy ones of equal to, not equal to, greater than, less than. In four of these examples, we're going to be filtering in order to meet certain conditions as going through if we were searching through these job postings from the perspective of a data engineer. Inside of MotherDuck, I'm going to go ahead and create a new notebook. We're going to call this one 1. 3 operators. In here, I'm going to start a new cell. with a standard query that we're going to carry out through the rest of this lesson. In this, we're going to select certain columns, specifically that job ID column. Additionally, job title short, job location, job via, which is like the online platform that we're going to be using for this. As you'll see, that lists platforms like LinkedIn or Upwork or anything like that where jobs can be posted to. And then salary year average cuz we care about salary. All of this is going to be coming from that job postings fact table. So, we'll go ahead and run this pressing command enter just to see what we're getting here. And I'm going to close out of this pane right here and also zoom in a little bit so we can see a little bit better. Anyway, what I like here is we have the inspector over on the right-hand side that we can dive into any of the different attributes with. And it looks like yes, the query below here only shows 50,000 results, but in the inspector, we can see that there's a total of 1. 6 million. So, the first thing we want to do, right? We're going to eventually get into data engineer

### [55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=3300s) Segment 12 (55:00 - 60:00)

jobs, but I want to see remote jobs. And so, we could use this job location column and filter for that using a where clause. Specifically, specifying the job location equal to we're going to say anywhere. For this, remember we need to put it as a string, in quotes. And this comes back and it looks like we get about 144,668 rows. Now, I want to get remote jobs and I actually want to call out something real quick with this is why we're actually doing this. Is I if I copy this query and then I create a new cell, paste this in, what we do have is in this data set, we actually have another column called job work from home. And this is a Boolean value column whether it has true or false values. And we want work from home values that are equal to true. I'm going to go ahead and press play. Now this is going ahead and return the results, but I'm understanding right now that maybe you don't understand what this Boolean value column is. I'm going to just go ahead and add it in. So I'm going to do a job work from home, add it in, run this query again. And it now adds in this job work from home. I can actually scroll over and see it. And we see all these different true values because we filter for true. Boolean being true or false. Anyway, this one for filtering for job work from home equal to true, which would be a remote job, we have 1,000 or 144,532. Whereas if we scroll up to this one, this one has almost 100 more. Even though this job location is anywhere, thus a remote job. So we'll be able to actually identify the discrepancies of what jobs are marked as well, anywhere, and also are not necessarily marked for remote work. This is an and condition and we're going to be covering it in our logical operators coming up. Next up, let's look at the not equal to operator. And for this we can use two different types here. You can see here it's either an exclamation point and equal sign or it's really like two Pac-Man eating each other. For those data engineers that were eventually going to filter for also with this, I want to look at let's say just contractor jobs. So going back to that base query that we have already, go ahead and run this. I'm going to add into here the column that we're going to be filtering for and that is the job schedule type. Running command enter so we can actually see this one. — [snorts] — We can see down and over here at the bottom of the column it has different values inside of here and actually can go in and inspect it further selecting job schedule type over on the right side bar and we can see that the full-time has 1. 42 million and contractor has 75,000. So what I can do is I can filter in a where condition on job schedule type equal to contractor. Running command enter we get our results at 75,000 for that's with the all of them being contractor but we want not equal to contractor I think I said equal earlier. Anyway, in this case all we need to do in our query is do a exclamation point equal to for the not equal to and we get about 1. 5 million results. We could also do the two Pac-Man eating each other and run this as well and same amount of rows appears. Now because I think you're getting the gist of this we're going to briefly cover greater than or less than operators. In this one we'll start with that same base query job ID job title short job location job via salary year average and we're going to filter our salary year average column. Let's say we just want jobs greater than 100,000. So we do where once again specify that salary year average and say greater than 100,000. Running this below we scroll over the column it looks like it's about correct but we can actually verify it's completely correct by going over this inspector panel. We can see that they actually start over at 0. 10 million to 0. 12 million which actually looking at the statistics of the bottom we can see that it starts at 101,000 dollars because it's greater than. Whereas if we want to do greater than or equal to running this now command enter inspecting this column we actually get a min value of 100,000. Conversely instead of doing that greater than or equal to 100,000 we could do less and we could see all the results less than that. All right, we're going to take a break from comparison operators real quick and dive into logical operators to actually start combining conditions. We have three ones we're going to cover in this, and, or, and not. The and operator only shows records when both conditions or all conditions are true. In this case, we have terms A and B. And then we have this nice little fancy Venn diagram off to the side. Unless they meet both of those conditions, it's not going to be included in it, so it's quite limiting.

### [1:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=3600s) Segment 13 (60:00 - 65:00)

So, we're going to be looking for data engineers, which would be term A, and then also remote positions, term B. And if they meet both those conditions, we'll get it. So, starting back with that very basic query, we're going to be filtering where we remember we want to do two things. So, first thing is for data engineers using that job title short column. We're going to use the equal operator to set it equal to data engineer. And now we want to include another condition, so I'll do and. And for this, we want the job work from home column, that Boolean value of whether they marked it as remote or not, equal to true. And I'm going to go ahead and run this, press command enter, and we can scroll down and see this or I can actually dive into it off to the side. We can see all the values are data engineer, and for the job location column, all of them are anywhere. One quick note on this Boolean value, you could also do a lowercase true like this, that's going to work just fine. I like to use all caps, it's pretty standard and it makes it stand out. Additionally, with this multiple conditions, I usually indent this down to the next line to make this a little bit more readable and it still works the same. Now, because we understand now the and keyword, we actually can investigate that issue we saw previously, where we had for job location of anywhere, we had this many values, and then for the condition of just job work from home of true, we had about a hundred less values. So, I'm pretty lazy. I'm going to go ahead and copy this job location anywhere up here. And inside of our query, I'm going to replace this data engineer portion that's going on. I'm going to run this cuz now we're going to be looking for job location of anywhere and remote jobs. And so, in May, we have 144,000. Well, where where's the discrepancy at? And that's with the we know that the anywhere column or the anywhere for job location has more values. So, there are some values that are marked remote or that marked anywhere that actually were marked false in the remote value. Running this now, we get the 140 rows. So, now we have a list of the 140 jobs that are maybe marked incorrectly as job work from home because they have the job location listed as anywhere, but yet the job work from home column is false. Next up is the or operator. And this only shows records when any of the conditions or either of the conditions are true. This would be the case if I'm looking for multiple different conditions. Say I'm looking for not only data engineer roles, but also senior data engineer roles. Starting with that base query, and if I go in to investigate the job title short column of this 1. 65 million, we can see we have about 392,000 data engineer positions. And around 91,000 senior data engineers. So, we can use or to get both of those. So, I specify where and then for job title short, we want it equal to data engineer. Then on the next line, we're going to enter that or operator. Specify for the job title short, we want then senior data engineer roles. Go ahead and run this. And inspecting that job title short column, we can see we have both of those options. Last up on logical operators is not operator. This thing can be used all over the place and it's a way of excluding certain values we may not want. Now, we saw this in the last lesson. I didn't specifically call it out as a logical operator, but remember in the last lesson whenever we were filtering for salary average null values. Let me go ahead and run this. We had null values in that salary year average column and then we used is not null in order to actually get all values that are not null. And then from that we figured out that we had values from 15,000 up to 960,000 dollars. Now, not can be used in any of our previous queries that we generated. So, I had scrolled back up to our previous one where we had the job work from home equal to true and if I wanted to search for instead of putting in false in this case for job work from home, I could just specify not. Run command enter and it provides exactly the opposite of this. And in this case we get 1. 4 million non-remote jobs. Now, not could also be used in a more complicated example. Going back up to that previous example that we did where remember we were looking at basically finding those jobs that were marked incorrectly. They had a job location of anywhere, but job work from home of being remote should have been true, but it was actually false and we had 140 rows for that. Well, if we wanted to basically exclude those values cuz we think it's bad data, we could use a not for this, but the key thing is we need to apply it for both of these conditions. What I'll do is I'll wrap them inside of parentheses to put it one joint logical statement. And now when I run this, we have that 1. 6 million rows and it's about a Well, it is 140 rows

### [1:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=3900s) Segment 14 (65:00 - 70:00)

removed from that of those that are potentially bad data. All right, only two more comparison operators to cover of between and in. We'll cover like in the next lesson and we already covered is null and is not null. For between, we're using this to get values within a given range. So, starting that base query that we've been using, let's say we want to get salaries between 100,000 to 200,000. So, we'll go ahead and put a where statement and we'll say we want for the salary your average, we want it between 100,000 and 200,000. Going ahead and running this, pressing command enter. Looks like I mistyped between. It has syntax error at between. We'll go ahead and accept and run this. Then, going over to the inspector, I can see, okay, I have values between 100 and 200,000. Now, this I like to use. I'm going to actually comment this out because previously, not knowing this, you would have probably written the query something like this, where salary your average is greater than 100,000 and less than 200,000. So, I'm going to go ahead and run this, pressing command enter. And I'm realizing now I actually have too many uh zeros in here. I said 1,200,000. Nothing appeared in this query. You can actually, little shortcut uh that uh the syntax or that the engine will pick up on is you can put these dashes in here. So, I'll put those dashes right inside of here. And so, that way you can actually see and count the zeros a little bit easier. I'm going to go ahead and run this. We can see from the results of this, we are still getting values between 100 and 200,000. Basically, the SQL engine's not picking up this tack right here and visually, you can see a lot easier when it's 100,000, 200,000. Anyway, getting back to this point, um this though, as you can see, it's highly it's a lot harder to one to write out and it's a little bit less readable. Additionally, look at this. This current query that we have has 28,631 rows. I'm going to go ahead and comment it out. And then go ahead and put this one back in. This one has more. Has 30,677. Why? That's because this one is inclusive. It includes 100,000 and 200,000. So, this query down here at the bottom is actually written incorrectly to try to match it. It would be greater than equal to 100,000 or less than and equal to 200,000. Running this one now, we have same results, 30,677. Last one up is in and this allows us to specify multiple values within a where clause much more succinctly. Previously, we had a query that we were looking in to find job titles that were either data engineers or senior data engineers. Now, let's say I want to add in something like data analyst. I would have to put in another or statement and put it on in here. Instead, we can use in. So, we're moving this previous portion. I'm going to put in and then we're going to create a list. And a list in SQL is in parentheses and then you put the values separated by a comma. So, we'll include that data analyst, that data engineer, and then also that senior data engineer. Now, go ahead and run this. Where we search that job title short column, we have three values. And this makes it super easy to add or remove values a lot more quickly and a lot more readable because we know, hey, we're specific to this job title short column. Let's wrap this lesson with a final example and I'll be doing final examples at the end of a majority of the lessons to tie all the concepts together and really test your skills. If at this point you're feeling good enough and you have enough understanding, you can feel free to skip ahead. I won't be covering any really major new details for using SQL. Anyway, let's say we have this problem here. For this problem, we're going to be looking at getting jobs for both data engineers and data analysts. I'm going to say I have more advanced skills for a data analyst position than a data engineer. And so, for data engineers, I'm going to say, "Hey, we want a salary between 75 and 100,000. " And then for data analyst, I'm going to say, "I want a higher salary. " So, of 100,000 to 125,000. Now, I also want to filter for location. I'm currently in Bella Vista, Arkansas. So, I want either a job here or a remote position. Or if I have to move, I'm going to move to sunny San Diego, California. For this, we're going to be sticking with that standard query of all those different attributes or all those columns that we want to get from this. So, we'll leave those all the same that we used. I'm going to start first solving this first portion of data engineers of 75,000 to 100,000. So, I'll create that where clause and specify the job title short equal to data engineer along with that salary condition of finding it between 75,000 and 100,000.

### [1:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=4200s) Segment 15 (70:00 - 75:00)

I'm going to just stop there. I like to execute queries as I build them to make sure I'm building them correctly. So, I'll verify that the job title short column is only data engineers and that we're meeting that condition right now for our salaries. Let's move on to now meeting also the data analyst condition of 100,000 to 125,000. So, this will be an and condition. I'm going to wrap all of this for data engineer into a single parenthesis and then write and then underneath it, I'm going to just copy this using command C or control C, paste it in here, and then go change the values to data analyst and then the salary of between 100,000 and 125,000. Let's go ahead and execute this bad boy, make sure it works. And [snorts] we get zero values returned. Oh, because silly me. We want to be this is an or condition, right? We want either data engineer jobs or data analyst jobs. I don't know why I did that. Crazy me. Now, for this, we have even more values. We see that job title short is both data analyst and data engineers. And then I expect the salary between, yep, 75,000 and 125,000. Now, we can move to the third port of we want to meet of job locations in either Bentonville, San Diego, or remote jobs. Remember, we're going to be using for these two, Bentonville, Arkansas and San Diego, California. We're going to be using the job location column. And then for the remote jobs, we learned that job work from home is more accurate. So, we're going to use a different column to filter for that. Anyway, this is going to be an and condition linking, hey, we want either of these roles and they need to be located in any of these conditions. So, let's create an and condition between these two and we're going to use parentheses in order to do this. I'm going to actually wrap this entire whole portion that we did already on filtering for the job title short and the salary information into one giant parenthesis. As you can see, this highlighted on both of these conditions. What I'm also going to do is go ahead and just indent this. I highlighted it all and press tab to move it in. So, this makes it a little bit more readable so you understand, oh, okay, all of this is one giant condition. And now we want to also meet on the condition of the salary. So, I've done an and we're going to put whatever we need inside of this parenthesis. Specifically, we'll just start first with the Bentonville, Arkansas and San Diego, California filtering for that job location. Remember, we could do Bentonville, Arkansas or San Diego, California, but in this case, we're going to use in and make a list of the different locations cuz I could always add in additional locations later with this. So, I'll put in Bentonville, Arkansas and then also San Diego, California. Um we will add in uh the remote work upcoming, but I want to just double-check make sure that this query runs correctly. And it goes all the way down now to based on our locations, it goes down to 16 rows. And it does look like this is correct because I can see actually inspecting the results, we have data engineers in both Bentonville and data analyst uh in San Diego and vice versa along with their expected salary. So, this is looking good. And remember, we want to now get remote jobs as well. So, this is only one statement right here. So, what I'm going to do is I'm going to go ahead and put an or in, and then I'm going to specify that we want to uh um filter the job work from home equal to true. Now, this is an or condition, so I expect when I run this the values are going to go up, and they do. They go up to 429. So, there's about 400 remote jobs. It's probably more likely that I'm going to find a remote job than something in Bentonville or in San Diego, unfortunately. The last thing I'd do to dress this up and prioritize where I'm going to focus my effort is actually sorting this. And I'm going to use the order by to sort our salary year average column. And we want to do it in descending order cuz I want it from highest to lowest. Had a typo. Make sure you type them correct. Thank you, AI. Ooh. And I can start now. It looks like a lot of different Well, obviously data analyst is going to have the highest in our conditions that we have. And I can start with these here for where I'm going to do my job search. All right, it's your turn now. Give it a try for those that purchased the supporter resources. We have quite a bit of practice problems for you now to work through and get familiar with all these different logical and also comparison operators. And then the next lesson we'll be jumping into that last comparison operator of like in order to use wildcards to filter further on basically doing some text analysis to filter out certain keywords. All right, with that, I'll see you in the next one. All right, in this lesson, we're going to be exploring two major concepts. The first is on wild cards, and we're going

### [1:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=4500s) Segment 16 (75:00 - 80:00)

to be applying it inside that logical operator of like. And then, additionally, we're going to be looking at aliases. Aliases are how we rename columns and tables to make it basically easier on ourselves to make queries. At the end, we're going to wrap all together and put both these concepts together in a final example. So, to make sure we're using the right terminology, I think I just used the wrong terminology in the intro. Comparison operators we're going to be looking at, the comparison operator of like. And like is similar to equal to, except we're able to do pattern matching, which we're going to demonstrate. So, inside of Mother Duck, I'm going to create a new notebook and call it wild cards and alias. For the queries in this lesson, we're going to be focusing on these columns of job ID, job title, job title short, job location, and job via. As a quick reminder, that job title column is a column that shows the original job posting title in it. And then I used some machine learning, separate from this, to actually clean up those columns and designate it into 11 different categories. Which, closing out this sidebar right here, and open up the inspector on the left-hand side, we can see these all these different values here. And I said 11 because there was 11 right here, but it's actually 10. This I'm hovering over and it says approximate distinct values, it has this wrong, it's actually 10. Anyway, I digress. Let's get into that like operator. So, let's say we want to filter for job location, and we'll just Columbus, Ohio. Previously, we just type in where, put in that job location, we set it equal to in our case Columbus, Ohio. Going ahead and running this, we get a bunch of results from Columbus, Ohio inspecting job location. It's only Columbus, Ohio. Now, let's institute it with pattern matching using like instead. It's pretty easy to swap out. All we have to do is take out the equal sign and put in like. Run this, command enter. Bam! Now we're doing pattern matching and we have the same amount of results of 1,464. So, let's get our first wild card of the underscore. And since we're using that like comparison operator, we can then put special symbols inside of our value that we're searching for. In this case, inside of data engineer, we want to look for one single character. In that case, it is a space, so we could leave it a space. But, this is just a showcase. It is matching on not only that data engineer, but specifically there's one character in between there. Now, let's apply this to a more realistic example. Let's say I wanted to search for Columbuses throughout the United States in different states. Well, in the United States at least, we do a two state or two-letter abbreviation for a state. So, what I can do is use two underscores here because underscore represents one single character and that character can be anything. Now, whenever I run this, pressing command enter, I can see that I get a host of different values actually popping up the job location right here. I can see I got them from all over the United States, even a few in Minnesota, Kansas, and Mississippi. Now, the second and final wild card cover is the percent sign. And this one, I feel is more flexible and frankly, I use it more. The percent sign represents zero, one, or more characters. So, for this query right here, we're searching for engineer, the word engineer, and it can have zero, one, or as many characters it one before or after it. And so this would be a really great way to filter out different types of engineers from our job title column. So let's try it out on our original query. We're going to go ahead and run this bad boy. And for this we can see that there's a lot of data analyst roles in that job title column. If I wanted to, we can match on that using this operator based on what we know now. So we'll insert that where clause, specify the job title column, use that like comparison operator, and then put inside of here the percent sign, data, and then space, and then analyst, and another percent sign. Go ahead and run command enter. And with this we're getting a bunch of different data analyst roles. And inspecting the job title short column, we actually can see like this one, oh it is marked business analyst, but it is in the title business data analyst. All right, so now let's say we want to either share this table or we're maybe building out a data warehouse and other people are going to access it. And we want to create table names that the final end user is going to actually understand what is going on there. Then they won't send me emails asking what they mean by it. Well, we can use aliases to solve that. Remember we talked about the job title column as that original job title in the job description. We can use the alias of as, that's the keyword there. And then

### [1:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=4800s) Segment 17 (80:00 - 85:00)

from there you can assign it a new name. In our case we could do something like job title original. This is just a name I made up on the spot. I'm going to go ahead and run command enter. And now we can see down here that it changed the name of the column that's appearing. It's alias. I can also go through and update other columns as well such as job ID as ID, job title short as job category, job via as job posting site. Now aliases are not limited to within that select statement. We can also use them within other statements as well, such as this from portion. And I could put in as and it's common for us in data engineering to rename this is a pretty long name to something like JP F, which stands for job postings fact. Running this query still operates exactly the same and as you recall from previously, you could put the table names in front of the actual column itself that you're calling. And going ahead and running this, same results. We're not joining multiple tables right now and so because of that, there's only one table in here. I like to leave this off, keep it as clean as possible and not include that naming of the table as well, not necessary. Now, what can be slightly confusing when reading other people's SQL queries is that sometimes people like to go through and not use an alias in there and they just write the alias right next to the column itself. And as you see whenever I run this, it still works. It reassigns it as necessary. So that as alias is not necessary, but it improves the readability of your query. Let's get into the final example. For this, we want to look for not only non-senior data engineer roles, but also non-senior software engineer roles. This is because a lot of the skills of software engineers and data engineers overlap. So in job search, you should probably look for both to make sure you're maximizing your opportunities in landing a job. Anyway, to do this, we've broken down into three steps specifically. We want to only get job titles that include either data or software, but then also include those with engineer in any part of the title. And we don't want to include anything that has something like senior or maybe SR for senior. That's typically followed by like a period or maybe some other character, so we'll just make sure it's no character. For this, we want to just get the job ID, job title, location, and job platform. So, this job title short right here, we're not going to use this time. I'm going to go ahead and remove that. So, let's forget this first condition of only get job titles that include either data or software. We'll then add a where clause, and inside of job title, we want to match the condition using that like operator for data. And this is an or, right? Cuz we want to also look at software. So, we're going to put in an or and put for this one job title like software. Going to go ahead and run this, and there's not a lot of options, right? Because we didn't provide, you know, we didn't use those wild cards appropriately. Right now, we're only getting 107 values, right? We want to use that percent sign operator to match zero, one, or more characters on both sides of these. So, I added in percent signs for both of these. Let's see what we actually go up to from 107 to 1. 3 million. Inspecting the job titles here, inspecting the values here, looks like we're meeting our conditions, although it's kind of hard to see software. I'm not seeing any really pop up. Now, real quick, you may be tempted to, in this condition, right? We've sort of had to repeat what was going on here. So, you may be tempted to use that in operator specifying data and software using those wild cards. However, when you go to run this, you're going to get zero rows because the in operator is not a pattern matcher like like is. Hmm. Anyway, we can't do that. We're not going to use it. So, let's just move on. I wanted to show that. So, now we're going to be getting into we want to also include those with engineer name part of the title. Remember, data engineers or software engineers. So, we're going to wrap this all cuz we want to meet this condition, and the condition of the engineer. So, we'll add in job title, and we'll say it's like engineer. Running this again, command enter. Now, when we look in this job title column, we have a lot fewer jobs popping up and now we are seeing that software actually pop up inside of here. Now, we do want to move into the last part, right? We want to remove or don't include any job titles with senior or SR followed by any character. As we can see here, we have that SR period and we have senior down here. So, we'll add an and condition. But, for both of these, they want it not to be senior or SR. So, I'll say and not and then I'm going to go ahead and copy this one up here cuz there's a lot of rewriting going on here. Add in job title like senior or job title like SR. Remember, it says

### [1:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=5100s) Segment 18 (85:00 - 90:00)

followed by any character. So, we can do We could do the underscore, but we'd also then want to do a percent sign. So, I'm just going to do only the percent sign just for it to make sense. Now, going ahead and running this, bam. Inspecting the job title column, it looks like, yeah, all of those senior roles are now removed, but we still have our software roles in there and that engineer roles. The last thing we need to do is just rename all the columns appropriately. So, I'll change job ID to just as ID, job location as just location, and job via as platform. Going to go ahead and now running this, we have all these results with ID, job title, location, and platform. If I wanted to, I could go ahead and copy this first 100,000 to my clipboard or download that amount to a CSV or whatever. So, that wraps up wild card and aliases. You now have some practice problems for those that purchased the supporter resources to go through and get more familiar with those wild card and aliases. In the next lesson, we're going to be covering arithmetic operations. So, very much following this theme of operators. All right. With that, I'll see you in the next one. In this lesson, we're going to continue learning more about operators. Previously, we did comparison and logical operators along with wild cards. And in this one, we're going into arithmetic operators. Now, assuming you have a grade school level education, you've probably covered all of these different arithmetic operations, well, except for the last one. We'll be going over plus, minus, multiplication, and also division. And then the final one you probably haven't heard of yet is modulus. So, first, we're going to start off with that addition and subtraction. So, inside of Motherduck, we're going to start a new notebook. And we'll call this arithmetic operators. I'm going to close out of the side panel, add a new cell, and I haven't demonstrated this before, but you can actually do very simple uh very simple SQL with only a select statement. So, I can do a select, and then if I wanted to, I could do something like 11 plus two. Run command and it provides me the results. Similarly, I could do something like 11 minus three, press command enter, and results as well of nine. Now, before we go further with a real-world example, I do want to call out real quick that we're not only limited to use these operators inside of a select clause. You could also use it in other places like where, order by, or group by or having, which we're going to cover eventually. The order by is least likely place we're going to carry it. Um the other ones are more likely where we're going to see it. So, getting to that use case of operators, for this, we're going to go into LinkedIn to look at job postings. Oh, by the way, if you aren't following me on LinkedIn, feel free to follow me. Anyway, if I go into jobs up here in the top left-hand corner, and actually search for some sort of data engineer job, it's going to go through and populate this. But the thing to notice with this is whenever we look at anything that has a salary, they actually list a range. This is a little bit This is I mean, this should be criminal. 99,000 to 232,000 job dollars for this job. Anyway, companies list this range because they could have any variable amount of pay based on your experience. They don't typically list a single value. Well, jumping into our data set, feel free to run this query if you'd like. I've queried the job ID, the job title short column, and then two salary columns, specifically the salary year average and the salary hour average. Just as some background to understand why that name average is on it, that's because typically whenever I scraped this data and collected this data, there was a band and I collected and I made all of these things instead of having a band, made it into a single value so we can compare it down the line. So in this case, for this cloud data and analytics manager, whenever I scraped this, this is going to be put in as the minimum salary of 99 and the max salary of 2,000 232,000. And what's the average of that? Well, running some sequel, adding 232 to 99 divided by two, average is around 165. 5. So with the salary year average for that job would be 165. 5 thousand. Anyway, let's recreate some fictitious bands, if you will, of the min and max salary. For this, we're going to be focusing on that salary hour average column because it's a little bit smaller numbers and it'll just make this a little bit easier for us to read. We're still going to understand the same concepts. So for our starting point for this, we're going to start with the job ID, job title short, and salary hour average column, and we're going to filter out any jobs that have null values in that salary hour

### [1:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=5400s) Segment 19 (90:00 - 95:00)

average column. All right, so this is what we have. Now, we want to create a, if you will, a fictitious band of min and max. And what we're going to say is the min band is $5 under this average band and it's then $5 greater than this for the max band. So for this I'm going to create a new column. We're going to start with that min band first. And we'll say salary hour average and we'll subtract five to get the min. I'm going to add a comma on the end. Press command Okay, automatically whenever I did this operation it put the operation as the column title. I'm going to go ahead and give this an alias of as salary year min. Oops, I mean hour. And now running this the column's going to update of salary hour min. And we can see as it goes through for each of these values it just subtracts five from that associated row and provides that value. Similarly for the max I put in that what I want to add five and I name this salary hour max. Running this I got this error. Parser syntax error at or near plus and I can see based on the AI but you should be able to see this too. There's missing a comma at the end of that salary hour average cuz I added an extra row there. Anyway, going to accept and run it. And double checking these results it looks like the salary hour max does add that $5. Now that plus and minus example isn't really realistic of the plus or minus five. Instead I asked ChatGPT, "Hey, what's the typical percentage band of salaries on job posting sites? " And I asked it to give it a percentage of the average salary. And it states that the minimum range is at about 80 to 85% of the midpoint and the maximum is about 115 to 120% of that midpoint. So let's actually use the multiplication operator in order to carry this out and make a more fine-tuned model of what the min and max salaries likely were. So, for this, I'm going to take that original query that we used already, and for the min, we're just going to multiply this * 80% or 0. 80. Similarly, for the max, we're going to multiply this * 1. 2. Now, running this, pressing command enter, we have some values that are way more closer and way more likely to be what would have been the min and max of an associated salary. Now, I also want to demonstrate division, so I have another scenario for you. And in this one, we previously looked at the salary year average column, and in this, we're going to filter for removing any null values. I'm going to go ahead and run this to show it. Anyway, so we have the different job IDs, job title shorts, and that salary year average column. Well, as you recall, we do have that salary hour average. If in the future we wanted to maybe compare the values from these two, we could make it into a similar value, meaning uh let's say I wanted to get from the salary year average, I wanted to convert it to an hourly salary equivalent. Well, how could we do that? Well, we're going to start a new line here, and I'm going to put in salary year average. And what we want to do is it's yearly, so we want to divide by how many hours are in a year. Well, typically, people work 52 weeks in a year, if you will, and there are 40 hours in a week. Now, that's not really necessarily realistic for most people, but we're just going to go with it, all right? Um so, going ahead and running this, we can see how a lot of these transpose over to a hourly salary with this. Oh, I need to also give this an uh an alias. And I'm going to call this hourly year salary. So, in this case, we now have that hourly year salary. And if I wanted to, I could then also bring in that I'll change the salary year average one to salary hour average. And we'll bring in those that are salary year average is not null and salary hour average is not null. Running this, we're not going to be able to compare across the two. And no, we're not It's only returning zeros cuz I got my logical operators wrong. We need to do an or value. Running this now, now we have a way to look basically an apples-to-apples comparison between the job descriptions with that hourly salary and that with the hourly salary of or yearly salary. In the next lesson, we're going to actually get into being able to perform aggregations across entire columns. So, this will be more useful than trying to go through these 50,000 results and trying to extract any insights out of what I'm trying to do here. But, I think you're getting the gist of what I'm trying to do with this. Last one up is modulus, and this returns

### [1:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=5700s) Segment 20 (95:00 - 100:00)

the remainder of a division operation. And many of you probably haven't seen it before, so let's just play around it using it only in select real quick. So, if I did something like select three modulus, which is the percent sign, one, and then run this. Well, three divided by one is well, three, and there's zero remainder. Now, instead, if I were to divide three by something like three modulus two, you would expect to divide this in there, and then when we run this, have a one as a remainder. This type of operation right here is really common, or it's a really common problem to be done to find out if a number is even or odd, so you don't have to like have some list of even or odd numbers. You can put something in like I go like say 10, 10 to modulus two. If it's even, it's going to return zero. In the case of let's say I change it to 11, 11 modulus two is one. So, if it ever has a remainder greater than zero, it's an odd number. So, here's the scenario I have. I'm pretty picky and as you can see when you go through the salary year average column that I've queried here, these salaries are pretty clean thousands. But, then you have every once in a while you have something like this where it's 120,531. Let's say we wanted to filter out values that were only ended in basically three zeros or we're specifying a thousand. So, something like this one is going to be filtered out along with this one right here of 130,500 because it's not three zeros at the end. Well, we can do this with modulus. We can see what the remainder is. So, let's add in a new column and we'll call this salary year average or we'll bring up salary year average and we'll perform the modulus of this by 1,000 or those three zeros if you will. Let's go ahead and just run it to see what it shows. Okay, in these cases that I wanted the value, the value is zero. And then for those that I want, it's basically greater than zero. So, I can now filter these out and I can do that by adding to our where clause and specifying that we want this condition right here, which I'm just going to go ahead and copy this and paste it here and then go ahead and press command enter. And it's telling me we didn't actually even specify anything right now. We just said, "Hey, and salary year average modulus 1,000. " We have to actually set it equal to something. We want to we want it to be equal to zero. So, I'm going to go ahead and command enter that and it returns. Previously, it was removing the zeros because zeros are equal to false and thus for it removes it. All right, for those that purchased the practice problems, you now have some problems to go through and get more familiar with those arithmetic operations. In the next lesson, we're going to be jumping into aggregation functions and actually putting to use what I was hinting towards in this lesson about comparing that hourly and your yearly column. All right, with that, I'll see you in the next one. In this lesson, we're going to be going over aggregate functions and functions are just something that you're allowed to give into that output some sort of result. And in our case, we're going to be performing aggregation like sum, min, max, or count. Now, in addition to those functions, we're also going to be covering two new keywords and that is group by and also having. We need those to use with our aggregate functions. Now, here is a table of the different aggregate functions that we're going to be going through. We're going to be referencing this multiple times throughout this lesson. But, the main point, as we can see from that first column, is that we need to input into our function some variable. In this case, it's going to be our column name. And that is what the aggregation will be performed on. We're going to start with the easy ones first like count and work our ways up to even harder one that don't even have list on this table. Now, as we'll get to in the lesson, you can do aggregate functions by themselves, but they become even more powerful when you bring combine them with keywords like group by and having. Group by allows you to segment by a certain condition like job title shorten this going condition, and then having allows you to filter. But, I'm getting ahead of myself. Let's actually get into some of these functions. All right, the first one we're going to cover is count. It can be used in two different We can either do count star or count column. Count star counts all rows in a table including those with null values, whereas count when we're just using the column name gives only non-null values in a column. So, let's jump into Mother Duck and actually test this out. We're going to create a new

### [1:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=6000s) Segment 21 (100:00 - 105:00)

notebook and we'll call it aggregate functions. Now, it's pretty common when exploring a dataset to find out how many rows are in it. So, we're just going to do something like select and then call out to do a count star and then from there specify the table name of job postings fact. And so, this returns back the number of rows in our dataset, which we know already, 1. 6 million. It's pretty great cuz we can actually apply a condition to this as well, such as filtering for data engineer jobs only. And in this case, we only have 391,000 jobs for them. Now, I could in this case list a column instead of count star. Such as in this case, it'd be more appropriate to use the job title short column if we were not going to use uh the star. So, running this, this provides exactly the same amount of results. And if we go back, remember that count star counts all rows including those with nulls and count column name counts only non-non null values in a column. So, if you remember, I'm querying here salary or average column. If you remember from that column, there are a lot of null values in here and they do have a uh salary every now and then. Well, this would be a good case to just use count of the column name. Specifically, going back to our original query, if we wanted to find out what data engineer jobs have a yearly salary listed, we can put that in here and we're going to see the jobs go from 391,000 to 10,551. Unfortunately, a lot of job postings don't provide the salary. Next up is Well, we're sticking with count, but we're also adding the word of distinct. And this counts unique non null values. So, if I were to query the job title short column, we can see that a lot of these values are repeating in here. I also showed you how you can actually dive in on the right-hand side via this UI, but you necessarily You may not necessarily have access to this all the time when you're doing SQL queries. That's where distinct comes into play if you want to get these distinct values. So, I'll put in that count function, and then inside the percent of parentheses, I'm going to add in distinct. Then I'm going to close off those parentheses, run command enter, and it would tell me that the distinct values or the amount of distinct values is 10. And just as a refresher from one of our earlier lessons on basic keywords, this is that same distinct keyword that we use in order to filter and actually list all those different job title shorts, which is as simple as removing these parentheses and then this count function right here, and then running it as is to show those values. Next up is sum, and this adds up all the non null values in a numeric column, which we have two numeric columns, right? Salary year average and salary hour average. So, in our case, let's apply it to that salary year average column, where we do have results eventually in here. All we're going to do is wrap salary year average in that sum function right here, and run command enter, we can get the sum of this. And the sum is around 6. 3 trillion dollars. Now, with that, we could I don't really find sum is actually that useful in this case. I could get the average, and based on what we learned already from our arithmetic operations, we could divide by the count of that salary year average column, which as we know counts all those non null values, and then in this case, we get the average salary. But, there's actually a function for that. And that's average. It returns the average of all non-null values. Specifically, once again, this is for a numeric column only. So, underneath this column, I'm going to actually put in that AV G function and insert the appropriate column, run command enter. Hopefully, it's the same results. It is Now, up to this point, we've just done aggregation functions by themselves to refine basically an aggregation of the entire data set. We can actually break this data set down and actually segment our data using a group by clause in order to understand characteristics specific to certain values in a column. Basically, we group rows that have the same values into summary rows and provide a summary for it. So, what do I mean by this? Well, what happens if we wanted to get the average salary broken down by country? I'm going to go ahead and remove this line right here because we already have average on there. And remember, we have in our data set a row on job country. Now, if I were to go ahead and try to run this, pressing command enter, it's going to hey say, "Hey, column job country must appear in the group by clause or must be part of an aggregation function. " And thankfully to this AI here within here, it actually tries to

### [1:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=6300s) Segment 22 (105:00 - 110:00)

add in this group by for this to work properly. I'm going to go ahead and just reject that to show you what I would normally do. So, underneath the from clause, I'm going to insert in our group by. And we need to specify any and all columns that are not aggregated uh to be able to group by. So, entering in job country, I'm going to go ahead and run this, and we have over 161 results. It's a little hard to read, so I'm going to actually order this in descending order. So, I'll put in that order by and specifically I want to order it by this portion right here, this average of the salary average. So, going ahead and run it now. Oh, remember we wanted in descending order. So, now we can see the highest country apparently is Panama at $279,000 for the average salary there. Now, let's say I want to take this a step further and that I want to not only categorize it or break it down by country, but also by that job title short column. I would go ahead and put in here that job title short column up in the select statement. Now, if I tried to run this without modifying the group by, I'm going to get an error saying, "Hey, you need to add it here. " It has some AI that's trying to fix it. I'm not going to let it do that. I'm going to do it myself. And under job country, I'm going to add in that job title short. Now, whenever I go to run this, I have it broken down by country and job title short. And I can see that oh, Belarus for data analysts is at $400,000. Now, one thing to note with this, some SQL dialects allow you to, if I wanted to, I could assign an Well, all SQL dialects allow you to assign an alias for this column of average salary year. So, I could say, "Hey, this is average salary. " But only some will allow you to actually use that alias within the order by. Microsoft SQL Server for some reason doesn't allow this. But anyway, going ahead and running this, it still operates the same and I feel this is a little bit more readable with this. All right, next up are max and min. And they fall find the largest non-null value in a column or the smallest in a column. This is not specific for these two specific to numeric columns because you could have uh, something like dates where you want to get the min or max, or you could have something like text values where you want to get something that's close to the A's for the min or Z's for the max. For this example, we're going to stick to only our numeric column. So, in here I'm going to add in that min, and then we'll do that for that salary year average column, and I'll give it the alias min salary. Similarly, I'll add in that max function for the salary year average and give it the alias of max salary. Let's go ahead and run this. And looks like we have some errors because I forgot Well, it's a syntax error at on your average, and that's because I forgot to put in commas at the end. Let's go ahead and run this again. I also mistyped salary year average in there. I kind of like this AI bot saving my butt here and there. Anyway, now we have for each of these different breakouts for this, also the min and max salary. So, that one that is in Belarus is very highly skeptical because apparently the min and max both 400,000. So, I'm guessing there's only one job posting in this country for data analyst. Now, one note on group by, and that is for DuckDB specifically, so that friendly DuckDB, I like it because you can actually replace You don't have to list these columns. You can just write the keyword of all, and it captures all of these keywords that are in here. When I go ahead and run this, it's going to provide the same results. Once again, this is only DuckDB specific, but it looks like other SQL dialects are also picking this up as well. But, I'm going to go ahead and change this back cuz I really want you practicing with things that are generic to most all SQL dialects. Next up is the median function, and I want to first execute it to show you what it looks like and then actually get into the theory behind what median is for those that may not be as familiar with the statistical function. For this, I'm going to clean this query up a little bit. I'm going to remove that min and max, and we're also going to only group by that job title short column to make this just easier to read. Now, inside of here, I'm going to add in that median function, and that is m d i a n, and we're going to be running this on that salary year average column. And we'll give it the alias of median salary. We're going to go ahead and run this. So, inside of here, we can see that we have the average on the left, median on the right for these different job titles, and that for each of these, the median is appearing, uh, it seems very close to this, but if you notice, the median is actually slightly less than, well, all of these values except for the very

### [1:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=6600s) Segment 23 (110:00 - 115:00)

first one. So, it's almost consistently less than. Let's get into the theory behind how you would calculate median. So, let's say you had a list of values, such as our salary values. What you would do first, if the value number of values is odd, you would put all of those values in order from lowest to highest. Then from there, it's pretty simple in that you just select the middle value, the median value, and that is what it is. So, in this case, it's a 100,000. But, for these five values, you see how the salary goes on. If you actually calculated the average for this, we can see that the average is actually much higher because look at this last value right here. It's at $300,000. So, it drags that average up. If I go inside of here and put in that max value in order to find that alias of max salary, we can see that all of these different job titles have like an orbitally high values that honestly, I would think that's actually unrealistic people to even be achieved. That's why I'm really a big fan of using the median because these high values are really dragging that average up. Now, what happens if the number of values that we put in order is even, such as in this case? Well, similarly, we put it in order, and then from there we'd select the middle two values. With those middle two values, we then do the average of those two values in order to get the median, which in this case would be 95,000. Comparing this to the total average of this all the six values, the average is once again dragged even higher once again because of that outlier all the way in the end at 300,000. So, whether we have an even number or odd number of values, we're basically getting about the same amount or the same value every single time for that median. And I really like median for analyzing that salary because of this issue that I've gone through. Because of that, for the remainder of the course, the focus will be for looking at the median salary over the average because I feel like it provides a much more represent representistic value you should be aiming for. Last keyword to cover that can be used with aggregate functions is the having keyword. And this filters queries based on aggregate functions and groupings. It's used because you can't use an aggregate inside of a where clause. So, applying it in our current query that we've been going through analyzing these different salaries for different job titles short, looking at this, I can see maybe I don't want to target roles that are less than 100,000. Sorry, business analyst and data analyst. Well, what I could do is I want to target roles with a median salary greater than 100,000. So, I would use a having clause. Your having clause is going to be used right in between your group by before or after group by and before your order by. And you're specifying having. In this, we not in not we need to not only specify the function, but the condition that we want to meet, that we want it to be greater than 100,000. And I'm putting that underscore there, like we talked about previously, to make just this easier to read. Going to go ahead and running this, pressing command enter. Boom, I can now see that all those different jobs of less than 100,000 median salary are filtered out. Now, for DuckDB specifically, similar to group by, you can use column alias in the having. But, with unlike group by that it is common in other SQL dialects, using it in the having is not common to others SQL dialects. So, DuckDB specifically, I could take this alias up here, median salary, I just command C or control C'd it, and paste it down here. It's going to work the same in this query. So, just to bring this all together, I made this fancy-dancy table that shows other popular databases, and those that allow are marked yes if they allow to use an alias in things like the where, group by, or having. As you can see, it's pretty consistent for DuckDB to allow this via their friendly SQL syntax. So, if you're going through and learning this, I wouldn't be I would definitely would encourage you just not to use alias in those, so that way if you were to switch to another SQL dialect, you're not going to get confused on why it doesn't work. All right, you now have some practice problems to go through and get more familiar with these aggregate functions. Got a lot of fun examples made for you. In the next lesson, we're going to be switching things up a little bit. We're going to be getting into some other data engineering skills, specifically into setting you up with a terminal and how to use it, and then installing DuckDB locally. And then we're going to be running DuckDB locally for the remainder of the course. I think it's really fun and it's really helpful skills that are going to help you out as a data engineer. With that, I'll see you in the next one.

### [1:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=6900s) Segment 24 (115:00 - 120:00)

Welcome to this lesson on an intro to a terminal. And we're going to be taking a short break from learning SQL commands in order to get DuckDB set up on your local computer so that you can operate SQL queries just like a data engineer would. Now, I've broken this up into three lessons that we're going to go through in order to get things set up. The first one is this, which is on the terminal, and you need to understand the terminal in order to go into the next lesson on installing DuckDB cuz DuckDB is run from the terminal inside your local computer. And then, next after that is the VS Code or Visual Studio Code intro. This is going to be our code editor. So, we're actually going to be writing SQL queries and executing it in tandem with the terminal. We're going to build out our entire project in that VS Code, but more on that later. Anyway, on this lesson on an intro to the terminal, we're going to be going through I'm going to assume you have no experience with it or that you don't even have it installed or set up. So, we're going to get it all set up and installed for you and then go through some basic commands. But, before we get into all that, we need to understand what the heck is the terminal? So, here we are inside my computer. I have two virtual machines running. On the left-hand side, we're running macOS for those Mac users out there, and on the right-hand side, we're running Windows 11 specifically, and that's for those Windows users. For this lesson primarily, I'm going to be going through and executing in both environments so you can follow along, but in follow-on lessons, I'm not necessarily going to do that because I think you're going to get the gist after that. Anyway, in both of these OSs or operating systems, I have up the terminal app, and this is a text-based interface where you type commands instead of just clicking around in the GUI or graphic user interface. It's how engineers interact directly with their computers and automation tools. In Mac, it's just a plain old terminal app that we use in for this. And in Windows, they use the same name, terminal or Windows terminal. Now, both of these use different shells, and shells is the language behind the scene that's interpreting what you put into it. Let's say I want to list the contents on a Mac, I could just write LS and it would list all the different folders in here. In Windows right now, I'm using the shell of PowerShell, and I would enter something like this of get child item, and it's going to show me the same exact thing. Granted, this is in a little bit different format, but you get the idea. Anyway, they have we're working in these two different environments in two different shells. Now, different operating systems have different shells, but it's different, if you will, default shells, because you could actually put any shell into your operating system that you want. For example, that macOS that I'm working in, it is in using Z shell, and that's for most modern versions. However, previous older versions are using Bash or Bourne Again Shell. Windows, as we demonstrated, is using PowerShell, and then Linux is using either Bash or Z shell. Now, Z shell and Bash are very similar. Basically, anything you know about Bash can be applied to Z shell. So, basically, if you know Bash, you know Z shell. Except Z shell has a few more extra features, which, as that engineer, not going to really affect you. However, PowerShell is completely different. It does have some similarities, but a lot of differences. In our course notes, I put together this fancy dancy table, and it goes through and shows what are the differences between the two. Now, there are in regards to commands, and there are some similar commands used, but eventually it does get complex enough that there's not really enough similarity between the two. So, we need to use a consistent one. So, which one we're going to use? Well, we're going to reference the 2025 Stack Overflow survey where they interviewed over 30,000 different developers on what is their well, language and then more specifically for us, what is their shell of choice. And out of all of these programming languages because shell languages are a programming language or a scripting language, Bash comes in at fifth on this list right after SQL and Python, a core skill of data engineers. PowerShell is on this list, but it's much further down and I found less likely to be used in my data engineering workflows. Now, to bring this all together on why I even covered this, you need to understand if you need to install something special for your computer to continue with this course and to have the necessary tools as data engineer. If you're on a Mac, newer or older, whether it use Z shell or Bash, either of these are going to be fine because like I mentioned before, Z shell

### [2:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=7200s) Segment 25 (120:00 - 125:00)

is almost identical to Bash. Now, if you're on Windows 10 or 11, you're probably using default shell of PowerShell or if you're on your older Windows, you're going to use you're using CMD or command prompt. For either of these cases on Windows, we're going to get you set up with another shell in order to work properly with Bash. And if you're following along on Linux, one, kudos to you, but two, you also have Bash set up in there as you probably know. So, for this, we're going to walk through opening or installing what's needed for the Mac first and then the Windows users after. Mac users, we're going to be using that terminal app using Z shell or Bash. And then for the Windows users users, we're be using Windows Terminal with the default shell switched to Git Bash and we'll have to install Git Bash. So, let's start with those Mac users. So, on a Mac, pretty simple. On your home screen, you're just going to press command spacebar and it allows you to start searching for things. You're going to search for terminal and that's pretty much it for this section on installing or setting up terminal. It's ready to go. Uh if you want to make the screen bigger, you're going to press command plus or command minus to make it smaller. Um so, I'm going to make it a little bit bigger so we can zoom in on it. Now, inside of Windows Terminal, you're going to have this first line up here that's going to tell you your last login. From there, it's going to tell you who's Mac OS that you're on. You'll also see the location that you're located within your file directory. Don't worry, we'll get more on that a bit, which is this little tilde line. And then the percent sign is basically signifying that's the beginning of where you can now enter in any command, something like ls. Now, switching over to those Windows users, you're going to come inside your search bar and you're going to search for terminal and this fancy dancy terminal app should appear. Note, we're not using this one of Windows PowerShell or of command prompt. We're using terminal app and this terminal app, it's pretty dang good. I'm actually a big fan of it. Now, this one, similar to Mac, you can make the words on the screen bigger or smaller by pressing control and then entering either positive or minus. By default, it should have opened up to PowerShell and I like this one also all right because you can do different tabs and have multiple different shell environments opened up. And it has by default not only Windows PowerShell but also command prompt and it looks like Azure Cloud Shell, whatever. Anyway, we both of these we we're not going to be using for the course. Remember, we're going to be using Git Bash. So, we need to get Git Bash installed so that we can use it inside of here. So, So your favorite web browser, navigate over to getscm. com. This is where we're going to be installing Git, which is a version control system and has the bash environment or bash shell needed to actually execute the commands that we want inside the terminal. Also, I know we have some steps to go through this, but you're actually doing some pre-work as Windows users because we will be eventually installing Git in order to manage the version control for a project. But it's a whole other lesson to get into version control, so more on that in a bit. Let's actually just get into installing this. So then navigate down, go to install for Windows, and it should have your appropriate one that you need for your type of Windows appearing up at the top, and we're going to go ahead and click here to start downloading. From there, just going to open the file up. It's asking to allow it to allow this app to make changes to my device. Yes, I do. Now, we're going to have to go through a little bit of setup. Through this, I would leave everything as default and accept these things on where it's setting it up. So these default things, I'm going to leave all the same. Now, I just went through and left all the default settings that I had set up for me. Just accepted it, and it's now installing it. It says that it's complete, and I want to launch Git Bash cuz I want to actually view that. I'm going to close this a view release notes and click finish. If it didn't pop up for you whenever you did this, you can actually just go in and type Git Bash, and you can get that way as well. I'm going to make this slightly a little bit bigger by pressing command plus and making it bigger. Now, for this terminal inside of Git Bash, it has your username along with the computer using, and then also your directory location, which is that tilde. And then next to the dollar sign, you can go ahead and enter any commands next to that and actually run it. Don't worry, we're going to go into commands in a little bit. But this is not the terminal that we want to run Git Bash in. There's nothing wrong with using this terminal. I'm just not as a fan of it as much as something like the Windows Terminal app. And we can get it installed right inside of here. So, let's get it into here. So, I'm going to press down this down arrow and we're going to go into settings. I could also press control comma. Scrolling on down on the left-hand side, I want to go down here. We see that we have the PowerShell, the command prompt, and the Azure Cloud Shell. I want to add a new profile. Specifically, I don't want to duplicate a profile. I want a new empty

### [2:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=7500s) Segment 26 (125:00 - 130:00)

profile. For the name, I'm going to change this from profile 4 to get Bash. For the command line, we're going to be changing this as well. We need to browse where this is located. You should be able to follow along where I'm going where it's located, specifically in the C drive under program files under Git and then bin and then we have this bash. exe file right here. This is what we want to actually open up. Now, I'm going to modify this slightly. I'm going to put double quotes around this file path cuz I'm going to add a command onto this to automatically log in. So, I'm going to do a dash and then log in and then a dash I. This login portion just makes sure that it's basically a fresh new shell that just opened up. And this I portion makes sure that you're able to run it in an interactive mode, basically type into it. Next up is the starting directory. And for this, I'm going to uncheck this use parent process directory and we're going to use this one to basically navigate to our user profile. Percent signs around it is a special Windows way to get to your user profile. Next up is the icon we're going to use for this. Right now, it's this like looking C looking weird looking thing and that's specific [snorts] for command prompt. Instead, I want to go to a file location where this icon's at. Conveniently, it's in that Git location. So, navigating again to our C file, program files, Git, and then this one of clang arm 64. You may have me mingw 64 or something like that. Something with 64 or something else that looks like this. You want to go into that, then under share, then get, and then you have this get for Windows icon right here. And then everything else can be left as is. I'm going to go ahead and click save. Now I can come and click the down arrow and see that get bash appears and it has the icon and I can locate it right in here. But the one thing you'll notice is if I close out of Windows Terminal Yeah, I want to close all the tabs and I try to launch it again, it's going to go into PowerShell. I want it to go into get bash by default. So I'm going to go back into settings and under this first tab with startup as under the default profile, I'm going to change this to get bash. Going to save this as well. And now whenever I go ahead and open this up, clicking the terminal, it automatically opens up the get bash profile. I'd also on Windows and Mac add this to your taskbar. So now let's get into some basic terminal commands now that we have terminal set up appropriately on both Mac and Window machines. For this, we're going to be going over four or basically five different types of terminal commands shown here. These are all used to navigate within your different folders or where you are inside of the terminal. The first one up is PWD or print working directory. In both shells, I'm going to go ahead and enter press enter for that PWD command or print working directory. It's going to tell you where your terminal is located. Right now it's located in that users on the Mac Luke. engineer and then on the Windows one, I'm in the users/LukeBarousse. It's very important that you understand where you are in the terminal because depending on what command you give, you may not have access to files depending on hierarchies in your file system. Speaking of which, lest let's list the contents of our files and we'll do that with the LS command. Running that on Windows and then running that on Mac. Starting with Windows, we can see that we have all these different folders in here. Specifically, I have something like the documents, downloads, favorites, I even have our desktop. Now, in OneNote, right, these are this is a virtual machine. These are actually looking at two separate locations even though this is on the same computer. So, there's going to be differences between the two, but gene- generally they're about the same. Such as on my Mac, I still have access to things like the desktop, downloads, and things like that. So, now with the information that I gave you, you're probably like, "How the heck do I change into a directory? " Well, for this, we're going to be using the CD command. And we're also going to do it on the Windows one, but before we do that, you notice it's at the bottom of the screen, there's a whole lot of clutter. You can actually type something like clear and it will bring you up to the top. You can do the same thing uh on a Mac as well. I'll type clear, it'll bring you up to the top. Anyway, if we want to navigate into specifically like the desktop folder, I would type CD and then desktop. Press enter. Now, I can see on Mac it went from that tilde to the desktop because that's where we're at. We'll do the same over here and we went into the desktop location. Now, say I wanted to now go back or up one level. I would use CD and then {dot} {dot}. {dot} signifies up one

### [2:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=7800s) Segment 27 (130:00 - 135:00)

level and we see we're back at the tilde. Just execute on the Mac, it did the same thing. We're back at the tilde or that home directory. Now, a cool neat little thing, if you don't know how to spell properly all the time, you can do something like this. If I wanted to go back into that desktop, right, I can start typing it. So, d e s and then if I press tab, it will finish it for me and I navigate into it. Same thing on my Windows, start typing in desktop, press tab, it updates it for that. Now, remember I'm still in my desktop. It is possible to go past that home directory. So, if I do that, CD dot dot, I'm up in that the tilde location, which doing PWD, I can see I'm at I could go up even more, and now I'm in the users profile, and I can type LS and see what's available there. Similarly, I can do the same thing on the Windows over here and get back up into there. They have inside of their users folder, it looks like a little bit different. But, I wanted to navigate further up because I wanted to show that you don't have to necessarily navigate or change directory down in only one folder increment. Instead, what I could do is I could say CD, I know I want to go into Luke {dot} engineer, I press tab to complete, and then I know I want to go into the desktop, so I start typing desktop, probably type tab, and now I can get there that quickly. Doing the same thing on this on the Windows machine, typing in Luke Bruce tab, and then I'm going to go into desktop, typing tab, boom, get into it that quick. So, let's crank things up a notch, and we're going to take these commands now to the next level. In order to do that, we need to actually break down the syntax of how a bash command works. So, in this code block, I have a representation of how a typical bash command is formatted. You have the command first, and that is always required. And then you have two other optional parameters after that. You have the first optional one, which is in these brackets, you have a flag or option, and then finally, you have an optional argument of well, an argument. It's a pretty simple format once you understand the basics of it and that the command is the program or tool you want to run, the flag or the option is how you want to modify it, and the argument is the target that the command should act on. So, let's show a few examples what I mean by this to break that concept down. And for the remainder of this, we're going to be just inside of Windows with this terminal because commands going to be all the same throughout this, and I think you got the idea. Anyway, we saw previously of LS, and this LS command list all the contents of the directories. Well, we can add a flag to this. Specifically, there's some files in here that may be hidden. So, here I'm going to bring this up to the top of the screen by pressing clear and type again LS, and then I'm going to give the flag. I'm going to do two dashes and then type in all. And now, looking at this one, there's a heck of a lot of different folders in here and files in here that we didn't see before. Specifically, these dot folders and files that are automatically because they have the dot at the beginning, they're hidden. So, that all flag helps get into it. Now, clearing this, I do want to show you don't necessarily need to always type out a flag completely. Like previously I did that dash all. Instead, what I can do for simplicity is actually dash A, and it still means the same thing, and that is still going to show all of those different files. At the end of this, I'm going to show you how you can find out what different flags are available to commands. So, that was an example of how we can use a command with a flag. Let's go now into a command with an argument. Well, our change directory command, that of CD, is the command itself, and then the argument is where we want to go. So, if we want to go into the desktop, that's what the argument is for this. So, now running this, we're in the desktop. Similarly, if I do the CD and then dot dot, that's going to move us back up into that home directory up one. Also, say for some reason that, you know, you just go crazy and then all of a sudden you're in a place that you don't know how to get back to your home directory. All you need to do is do CD and then give it the argument of tilde and then you're navigated back into that home directory. Now that we understand that command flag argument structure, we're going to go into actually building out how you can work with files and folders, which is really common for data engineers have to do in the terminal and frankly anybody that's interacting with terminal. We're going to go through these different types of commands in order to make different directories, files, edit them, remove them, and be able to view them. Now, for this, I want to show this like as it's happening in the file explorer. So, what we're going

### [2:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=8100s) Segment 28 (135:00 - 140:00)

to do and what I recommend the same to you is I would have one window to the left with your terminal and then one window to the right with the folder location you're navigated into. For me, I went into the C drive under users under Luke Barousse, which I know what that's where we're going, and then we're going to go into the desktop. So, right now I'm navigated in the desktop in my file explorer. I need to do the same thing in here, so I'm going to change my directory into my desktop. And it says inside of here, "Hey, this folder is empty. " Let's just double-check in the terminal. Yep, looks like it's empty. Well, let's see if it's actually empty of all files. So, for that, dot and double dot. As we learned earlier, the double dots mean the parent directory or one folder up. We use things like CD dot to navigate up a directory. Now, the single dot is a new concept, but that just means the current directory. Anyway, they're both just listed here as they represent where you are and what folder is above it and how you can access it via the command line. Okay, it looks like it does have some dot files inside of there, Not a big deal. So, the first thing we're going to do is we're going to make a folder and that's what's called a directory. So, we want to make a directory that's mkdir. That is the command. So, we now need to give it an argument of what we want to call this folder. We're going to name it something real original. We're going to call it folder one. Now, I can see inside of my desktop I have folder one. Now, that's for making folders. What happens if I wanted to make a file? Well, in that case I would use the touch command and I would give it the argument of the file I want to create. So, I want to create something called maybe file one. txt. Go ahead and run this. We now have file one. txt. And really this could be any type of file, so it could be file two. json. And it's going to create a JSON file. Now, let's say I wanted to create a file inside of my folder one. Right now, if I navigate into it, this folder is empty. I could change directory into that folder one and then make that file. Or I'm going to navigate back up and clear out these contents. Or instead, what I'm going to do is I'm going to create the file by specifying I want to create it in folder one and I'll name this file three. txt. We'll go ahead and run this. We can now check inside of Well, we can actually see Windows makes it pretty visually appealing. But you can see there's files in there, that's cool. And I see I have file three. txt. Now, let's say I want to remove files. In that case, I would use the RM command for remove and I can specify, "Hey, I want to move that file two. json file that we created in the desktop. " And it disappears. Now, removing a directory is a little bit more complicated, unfortunately. So, they do You a command of remove directory or rmdir and I can specify hey I want to remove folder one from this directory. Running this says hey it's failed to remove folder one direct the directory is not empty. Now I could go in and remove that file in folder one which I'll do in this case and then from there I'll actually go in and remove directory of folder one but there's going to be cases where you have multiple files in a folder and so it would be a very nuisance to have to try and delete all those files before you delete the folder itself. So I'm recreating that directory for folder one and I'm using touch to make that file three dot text inside of folder one. We'll go ahead and clear this out. So we covered we can't use remove directory cuz there's contents in it. What happens if we try to use just remove folder one? It's going to say hey you cannot remove folder one it's a directory and we know we had remove directory to try to remove it. Anyway, we can instead provide a flag to this of dash R or dash recursive. We're just going to go with dash R for our Mac friends cuz I think only they on Mac the dash R is going to work. And what this is saying is it's going to recursively remove all of the file contents of folder one and also delete the folder and allow us to delete this. So let's go ahead and try. Okay, and it actually deletes it. So even though remove says it can't remove directories cuz it's a directory it can you just got to give it the right flags. So now let's get into editing the contents of a file. And right now file dot one dot txt, remember we just created it, I'm opening it up inside of notepad just to show that hey nothing exist inside of it right now. I'm going to go ahead and close it. Anyway, yes we can go into notepad to add contents to it, but we may need to programmatically

### [2:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=8400s) Segment 29 (140:00 - 145:00)

add contents to a file via the terminal. So, we need to learn how to do that. Well, in order to do this with commands, we need to actually learn a couple of commands. And the first one is echo. Echo is a command that helps you output text. What do I mean by that? So, I'm going to type in echo and then in parentheses type in data nerd, so it knows that's the text I wanted to do. Okay, whenever I press enter on this, it echoes data nerd to the screen. I can't just type in something like parentheses data nerd. It doesn't let you do that. There's no text output that echo allows us to output this text. Now, echo is not limited just text strings. You could also do something like a number, it'll echo it to the screen. Or something that we haven't really covered yet is variables, such as we have a variable for our home path, and you could echo that to the screen. Variables are designated by that number sign at the beginning of it. And by number sign, I mean the currency symbol at the front of the dollar sign. Anyway, why do we learn this all, right? We want to send stuff into file1. text. And so, we can do this by saying, "Hey, want to echo in something like maybe data nerd. " And I'm going to provide it this greater than symbol, which is the redirection operator, and we're going to put this into file1. txt. Now, I'm lazy, right? I don't want to open this up and actually have to look in here to see that there's data nerd inside of there. Instead, how can I easily see what I did? Well, I can use cat to concatenate and display the file contents of file1. txt. And I can see it has data nerd. Now, let's say I wanted to run that again. I'm actually going to press the up arrow, and when I it navigates through the last command. So, you can use the up-arrow to navigate to previous commands and the down arrows to navigate back. Anyway, let's say I wanted to change this and I wanted to insert something else, such as data analyst. I'm going to go ahead and run this. Now, whenever I look into this file of file1. txt, the contents got overwritten with data analyst. So, if I wanted to let's navigate up back into data nerd if I wanted to also append data nerd to this list, I have to put two greater than symbols. This is the append operator. And now when I press up to get out to cat the file, I see I have data analyst and data nerd in that file. Now, real quick just to go back to how bash commands work, we've covered commands and we've covered how to use uh flags with commands. Then we've also covered how to use commands, a flag, and even an argument. But, we haven't covered options yet. Options require on top of the designation of like {dash} color, it requires some sort of value to go with it. Also, I don't know why I'm using my hands so much. So, let's demo this. We can actually use on top of just cat. Cat's more popular, but you can also use something like head to show the beginning of a file. So, I can use head file1. txt. Our file's only two uh lengths long or two rows long, so it can show the top of this. Anyway, let's say I wanted to only show maybe the second line of this. Well, I can specify head, give it the option of n, which show is how many number of lines to show, and I can specify two in this case, and we'll go And I mistype file1. txt. We have to actually add a one in and it displays it all. Now, we could also just show one row in this. I could change it to this to one, run this as well. We get just that analyst. Anyway, I wanted to run this option just to show you options. Now, you may be like, "Luke, that's crazy. Anytime I want to edit a file, you're telling me I have to write this echo what I want to put into it and then append it into the file and then hopefully I get it in there right whenever I cat that. " And I didn't type that command fully. Just a shortcut, if ever your terminal gets locked up, you can use control C and it would automatically exit out of whatever operation it's doing and you'll start fresh. Anyway, if I you don't want to go through and just type something like cat file 1. txt and do that every single time to make sure that you've input it in right. Instead, the terminal has an editor you can use with this. It actually has a couple. One of them is vim and we can go into file 1. txt. Vim though is a little bit harder to use and so we're not going to go into it. If you ever get yourself into vim, you're going to need to press shift and get the colon going and then from there, you can see a colon in the bottom left right here. You would need to say wq to write

### [2:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=8700s) Segment 30 (145:00 - 150:00)

it and quit it. Anyway, vim's really hard as a text editor to use. We're not going to go into it in this course. What I prefer is the nano text editor. We're going to use this on file 1. txt. Whenever I go into this one, it gives me all the options at the bottom. So, if I wanted to add something else like data scientist, I can and then now I want to write it out. So, I'm going to do control O. That's basically saving it. Press enter. We wrote these four lines. Now, I want to exit it. So, I'm going to do control X. And now, when I take a look at the file of file 1. txt, I can see we have all of those things. Nano in general is just a good text editor to know. Everything's at the bottom for cutting, pasting, and moving things around. Really a fan of it, so we're going to be using it predominantly in this course anytime we need to edit stuff. Let me go ahead and exit. So, just as a recap, we covered how to make not only directories, but also files, and then also how to delete associated files and directories. We also got into how to do text editing from the command line and getting into text editors themselves. Now, you may be like, "Lou, how the heck do you know all these different commands and what type of flags and whatnot you can use with it? " Well, my number one source that I'm going to is either something like ChatGPT or Gemini, and it's really quick at actually going through and helping you out in learning how to use the terminal, what commands to not to use, and what not to use. Now, you may get so efficient with running different commands in the terminal and you want to investigate them, but you don't want to hop over to ChatGPT. Well, here's a tip. Unfortunately, it's they're both specific to either whether you're on Mac or on a Windows. On a Mac, you're going to run the man command, which is short for manual. And if I want to investigate something like ls, I provide that as the argument. And then inside of here, I can scroll around to see what the name of this is. ls is list directory contents. It gives a description. And then scrolling down, I can see all the different flags in here. We can even see that {dash} a command where that includes directory entries whose names begin with a dot, like we learned. Along with a whole host of other options you can look at. Now, in order to exit out of here, I can see there's a colon down at the bottom. I would just press q, and it exits out of it. Now, on a Windows machine, if I try to run man ls, that's not going to work. Instead, what I'll run is the command itself of ls, and then provide the flag of help. This provides all the things we have. It's not really formatted that great because of how zoomed in I am for this, but if I zoomed out, you could see it actually does look cleaned up. Just the way I have it on split screen. Anyway, I do want to show I can't run LS {dash} help on a Mac. It's not recognized. It doesn't work. In either case, they provide the same results and you'll be able to go through this and actually investigate further what type of flags and what type of arguments you could provide to a command. All right, you now have some practice problems to go through and get more familiar with operating inside of a terminal and it's really a skill that's going to pay dividends as a data engineer, so I don't recommend skipping on that. In the next lesson, now that we have the basics on terminals, we're going to get into setting up local deep uh DuckDB inside your own computer to run from. All right, with that, I'll see you in the next one. In this lesson, we're going to be going through how to install both for those Windows users and Mac users, how to install DuckDB. And with this, we'll be walking through all the commands with it and by the end of it, we'll be connecting to our attached databases in MotherDuck. But, you need to watch this whole thing because we need to install the correct version of DuckDB in order to connect to MotherDuck. Now, before we jump into the install, we need to understand how DuckDB locally is connecting to MotherDuck and what the heck each of those two is, just in case you forgot. DuckDB is the database engine that's running behind the scene whenever you're doing any SQL queries. So, previously we were running in MotherDuck and that has DuckDB databases in there and it manages it through MotherDuck. And in this lesson, we're going to be actually installing locally on your computer DuckDB. Now, just to show this simply, we've been accessing MotherDuck where it has our different databases on there, specifically our data jobs database. And we've been accessing it in your computer via your browser. We've been just using it via the data jobs connection. Now, with this setup with MotherDuck, this has allowed us and all the other students in the course to

### [2:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=9000s) Segment 31 (150:00 - 155:00)

connect to the same exact database. Pretty dang cool. But now, we're going to be shifting gears and instead of using the browser, we're going to be using the terminal cuz that's very popular with data engineers in order to automate your different scripts. And in order to do this with a terminal, we need to install DuckDB locally on your computer. And this will allow us also to create local databases as necessary that we can then operate on and do whatever we want to and not affect anything in MotherDuck. This is very common for data engineers for you to work on a database locally and then once it's fine-tuned, you could push it up to your data warehouse like MotherDuck. And this is automated not only through the terminal but specifically using different SQL files and something like code editor like VS Code, which we're going to install in the next lesson. And we'll be using VS Code to interact directly with the terminal. And this, as I mentioned, is a real-life production setup because now, you would work like you would as a data engineer with DuckDB locally on your computer connecting also to your data warehouse in MotherDuck. And then, other people on your team, whether data engineers, analysts, or scientists, could connect to those databases in MotherDuck via their preferred tool. So, we're now going to be getting into installing DuckDB. And for this, we're going to be, at least for one of the versions, we're going to be installing it here from this location. So, I put a URL on the screen, that thing's heck along. I would actually instead just Google MotherDuck DuckDB CLI and you should get directed to this page. Now, we need to pay special attention to this note up here. In that it says, "MotherDuck currently supports DuckDB 1. 4. 2 in the US and then 1. 2 to 1. 4. 2 in Europe. " And then it closes by saying, "MotherDuck supports 1. 4. 1 through 1. 4. 2. " You're like, "Yeah, what does that even matter? " Well, if I go over to duckdb. org, they're having or they're the recent version they have is 1. 4. 3, basically one version ahead. And so, this version is not supported by MotherDuck. And what do I mean by that? Well, as I found out the hard way, whenever I installed the wrong version of DuckDB of 1. 4. 3 when I filmed this lesson the first time and now having to re-record it right now, — [snorts] — is that when I went to connect DuckDB locally to MotherDuck, I got this error message and it says, "Hey, your currently installed version of DuckDB 1. 4. 3 is not supported for connecting to MotherDuck. " And unfortunately, just changing the version of DuckDB is not super easy. So, that's why I want to make sure you install it correctly the first time. Now, this type of issue of versions is pretty common in the data engineering world. In that you have another company that builds their service on top of a certain product, such as in this case MotherDuck builds their product on top of DuckDB, and then DuckDB, because it updates, MotherDuck needs to then go through some different verifications and testing to verify that DuckDB actually works with or acceptable and safe enough to work with Mother Duck before implementing that new version. So, a complete standard and something you get used to in the data engineering world. One last thing on this note, you're going to be watching this at a way different time than as I'm filming this in December of 2025. So, these versions may be completely different. Heck, you may even have it to where the version on DuckDB is actually supported by that of Mother Duck. Anyway, because of that, the safest and easiest option is just to always come to this if you want DuckDB in order to install it. Now, for both Windows and Mac users, I'm going to walk you through this install separately. And for Windows, it's as simple as installing a. exe file or binary file and installing it. And then for macOS, it's as simple as running this curl command in your script. Um although there is actually another step you have to do after that to make sure it gets added to your path. We'll get to it when we get to that. Anyway, that will install the correct version. So, if you just want a simple option, you can just stick with that I'm going to give you. Now, I mentioned this all to the Mother Duck team about all these different issues, and they had a pro option that actually I've implemented in managing and installing DuckDB. And so, if you don't want to do or you can watch that first portion that I do on the easier option, this will be the pro option. And with this, this right here is via this GitHub repository. This is DuckMan, and it manages your DuckDB CLI with ease. Scrolling down to the documentation, what it is this is a version manager. So, unlike the easy option where we just install a certain version and then that's it, you can't really update it that easy if you want to change to a

### [2:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=9300s) Segment 32 (155:00 - 160:00)

different version. Well, with this tool, you can. So, we're going to walk through this on both Mac machines and also Windows machines as the pro option. And once it's installed, it makes it super simple to then go through and install DuckDB for a specific version, setting it as your default, and then just running with DuckDB with that version. So, for both Mac users and Windows users, whenever I go through this install process, I recommend you just watch the easy option and then the pro option, and then whatever you feel your comfort level is with, go with that one. So, let's get into installing DuckDB via the Motherduck option here. We're going to go with this is the easy option. It has that we want to download this binary here, and then we're going to extract out this zip once it's installed. Well, I did it pretty quickly cuz it's pretty small. Then from there, I'm going to just double click on DuckDB to launch the application. And what this shows is that this is the DuckDB executable. This actually ran it inside of looks like command prompt. But, this shows that DuckDB is installed, if you will. But, if I were to open up a new tab, such as where Git Bash is, and I tried to use the command of DuckDB, it's going to say, "Hey, that command's not found. " So, yeah, anytime that I'd want to use DuckDB, I'd have to navigate to this file location, and then click on DuckDB in order to use it. Sort of a pain in the butt. So, right now this application or executable's inside of our downloads folder, and inside of this basically zip file for it. Not a great location for it to be cuz sometimes I like to clear out my downloads folder. So, what I'm going to do is this. I opened up another file explorer, we're going to move this. Specifically, I'm going to navigate into my C file or C drive, and then into program files. This is where it would commonly be be, and it would be in a DuckDB folder. So, I'm creating a new folder. And so, it's asking do I want to confirm this? I do cuz I do want to create one here. And we're going to call this DuckDB. Go ahead and press enter to create it and then go into that file and I'm going to drag the duck DB application into here. So, it asks me again if I want to do that. Yeah, Now, I do want to show something that you could technically now run duck DB from the terminal. What I'm going to do is I'm going to go ahead and I'm going to right click it and I'm going to say, "Hey, copy as path. " And then I'm going to go ahead and paste it in. And so, now I'm going to go ahead and press enter. Boom, it can run. So, you could technically, anytime you wanted to run duck DB, you could just enter that in and run it. But, that's a lot of work. We don't want to do that. We want to be able to use the command just duck DB and then it run. So, how do we do this? Well, we need to update our environment variables. Environments are variables inside of our terminal window right here. So, I'm going to go in here and type in VAR for variables and it says, "Hey, I want to edit the system variables environments or system variables? " I do. In this pop-up, I'm going to select environment variables. And then specifically, we want to update our path variable. And this is every time you run a command, it looks through this path. Actually, I'll just go into it. I'm going to click it. It goes through and looks in these locations for different executables that you may need to run. Specifically, like our duck DB one. So, we'll add it by doing new. And we need to get the actual location of this, not necessarily this one, but this file path up here. I'll right click it and I'll say, "Hey, copy address as text. " Click new again and then I'm going to paste that in by pressing control V. And then from there, click okay. Finally, click okay. Close out of this and close out of this. Now, if I try to run duck DB, not going to work just yet. I have to actually restart the terminal altogether. So, I'm going to close out of this. And then I'm going to restart it up. And now, whenever I run DuckDB, it automatically goes to DuckDB. This is supposed to be the easy option. I don't know. — For the pro version option of installing DuckDB and being able to manage the different versions, we're actually not going to install this DuckMan. I tried to install it just now using Windows and ran into a whole bunch of issues. I did get it to work on Mac, so Mac users are still going to use this. Anyway, we're going to use an option for the pro option of winget. What the heck is winget? Well, it's a command-line tool enabling users to discover, install, upgrade, remove, and configure applications on Windows 10, 11, and Windows Server 2025. Basically, it's a command-line interface package management tool. It's short for Windows

### [2:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=9600s) Segment 33 (160:00 - 165:00)

get. Now, you can see if you have it installed, and you should for Windows 10 or 11 by typing in winget {dash} version, and it will list the version. For those that don't have winget installed for an older Windows version not supporting that, feel free to use ChatGPT to figure out how to get it installed. So, now we can use winget to install a specific version of DuckDB. So, I'm going to type in winget install DuckDB. CLI command-line interface, and then we're going to use the option of version. And for this, we need to make sure that we're specifying the version that's supported by you currently right now, and for me, as of the time of filming this, it's 1. 4. 2. And so, I'll just type that in here, 1. 4. 2, run enter. And for the pro option, this is actually a lot easier. It says, "Hey, the command-line alias added DuckDB, running DuckDBD, and then checking the version. " We can see that 1. 4. 2 is installed. So, I don't know. Maybe the pro version is actually easier than the original version. — Now, the only issue with this is or not really even an issue. If you want to install a different version of DuckDB, you need to first go through and uninstall this version by running winget uninstall DuckDB and then the CLI. It will successfully uninstall it. Now running DuckDB, I can see that it's no longer in my path. And then from there, running the install on the newer version. I'm going to keep it 1. 4. 2 for me. We're now going to get into installing DuckDB from MotherDuck for the correct version on a Mac. And this will be doing the easy option. I'll put easy option because I actually I think the pro option is a little bit easier, but it's a little bit more complicated with the technologies we have to use with it. Anyway, let's just start with the easy version first. We're going to be installing it via this URL that's on the screen here. And it may be easier just to input in Google to get to this. Anyway, in macOS, it says to do this, we need to install with bash using this script. So, I'm going to go ahead and copy this. Into my terminal, paste this in and then run this bad boy. It looks like it's going through a setup. It's always important to read documentation. So, it says it successfully installed DuckDB, but we need to append the following path to our shell profile. Specifically this right here. And I'm going to show you what that actually means. Anyway, yeah, we can also see that it installed the correct version, but it could not find a shell RC file to add the MotherDuck token to. Exiting out. Anyway, the key thing here is we need to understand that if I wanted to launch DuckDB now, I'd have to type out this full command right here. So, I'm going to go ahead and copy this command C, paste this in, and press enter. Okay, I'm now in DuckDB. I can run something like select 42 as answer. Bam. But, running dot quit, that's not I don't want to have to pressing the up arrow. type out this entire thing every single time. I need to add this location of DuckDB to my path variable, so that way I can just put into here DuckDB and run it. And it will give me this zsh command or zsh command not found. So, how do we do this? You run this command, but we need to specifically add it to our shell profile. I'm actually going to just show you real quick. We can So, we can actually run this. I'm going to go command C and copy this file. And then paste it down underneath here. All this is going to do is exports I think it's very similar to echo in that it's going to add this to our path, but it only does it for this terminal right here. So, now whenever I run DuckDB, I can get into it. Um and I'm going to do dot quit. Um we'll cover dot commands in a little bit just to get out of this. But, if I were to close and exit this terminal and then try to run DuckDB, it's not going to find the command. I still have to go through that full on command this in order to get into DuckDB. Um because it resets every time. So, we can fix this by adding this to our path or our script that manages or you runs every time we start up a terminal. So, we need to add this addition to our path variable that I just showed to a certain script in order that you whenever you start up your terminal, it automatically gets added to your path and you can just run DuckDB. Anyway, it's going to depend on what shell that you're in your terminal on your Mac. As I discussed in one of the earlier videos, newer Macs are using Z shell and you can see that up at the top of the terminal. If you see bash, then you're using the shell of

### [2:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=9900s) Segment 34 (165:00 - 170:00)

bash. Anyway, for whether you're using Z shell or bash, there's two type of files and you either have the Z profile, a dot Z profile file, it's a hidden one, or a dot Z S H R C. And RC stands for run configuration. Similarly, bash has the same naming convention instead of using Z, it uses bash. But concepts are still the same. Now, the dot Z profile only runs once and that's at login, whereas the dot Z shell RC file runs every new session. Bash profile and bash RC follow the same convention. Anyway, inside of here, I'm going to see if I have those files because they should be located in your home directory. So, I'm going to do LS-all. Now, they're not in there, so I'm going to go ahead and add both of them. They're just going to be blank files, it's okay. I recommend you do this regardless if you're going to do this install method or not. You should have these files. Making sure I'm in the home directory, that tilde directory, I'm going to do touch to create the file and then dot Z profile. And then I'm also going to do touch dot Z shell RC or run configuration. Now, when I run LSA, I can see I have the Z profile and the Z shell RC file. So, now pressing up, we need to add this right here, this previous command that we ran. I'm going to go ahead and just copy it, command C, and then exit out of it. And then from there, we're going to use the nano editor on dot Z S H R C. I'm going to tab over, select it, press enter. And I want to paste it inside of here. Remember that Z shell RC or run configuration is going to run every new terminal session. This is where I want to add it. I to save it, so I'm going to do control O and press enter. And then I want to exit out of it, and so I'm going to press control X. Now, DuckDB is not going to work right now, right? Because we need to reset this terminal session. So now starting a new window, I'm going to run DuckDB and goes into DuckDB just fine. So now getting into the pro version, which I actually feel is going to be a little bit simpler. But with this, we need to go in and it says, "Hey, for this the scrolling down to the actual documentation on it. " It says, "Before proceeding with the installation, it's advisable to uninstall or remove any existing DuckDB CLI. " So if you installed this and you want to do this version, make sure that DuckDB is uninstalled. Specifically, if I do a LS all, I can see that the DuckDB file or the DB file is in there. So I'm going to do a remove and then flag of R and F to make sure that it deletes it recursively. And then the F portion to force it. So recursively and force. And then I'll put in there. duckdb for it to delete. Now whenever I list it all out, DuckDB is not in there. If I try to run DuckDB, it's going to be, well, command not found, but I need to go into that file that we added in previously. So I'm going to nano into Z shell RC file, and then I'm going to just control K this file to cut this out, control O to uh save it, and I want to save it to that name, and then control X to get out of there. I guess I didn't save it last time, so I'll put yes. Go ahead and put enter. All right. So now it's completely removed. We can go through with this install process. For good measure, I'm just going to close out of this terminal and start a new terminal. Okay. In here, it says this is the command that we need to run inside of our terminal. Go go ahead and copy it. And then in that new terminal that I started up, going to paste that right into here. And it says, "Do you want DuckMan set up to auto complete and add the path for you? " Yes, I do. I don't want to go through that again. Now, it's important that you, before you ran this, that you had run that touch. zshrc and also that touch. zprofile. So, just looking at that. zshrc file, we can see that inside of it added in this portion right here to add DuckMan to our path. But, we run DuckMan right now, it's going to say not found because we need to restart our terminal. Remember, it's every single session that it runs that. Now, whenever I run DuckMan version, it tells me what the version is. All right, so now scrolling on down to usage. From here, we need to tell it which version or which yeah, which version of DuckDB to install. So, you need to go into the MotherDuck documentation for this DuckDB CLI install and actually read this note right here on what's applicable to you right now. Cuz as of filming this right now, I need to do 1. 4. 2, maybe different for you. So, I'm going to do DuckMan install 1. 4. 2. And didn't get any issues, okay. Now, if I go to run DuckDB, not going to work. The command's not

### [2:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=10200s) Segment 35 (170:00 - 175:00)

found. And that's because per the documentation, we need to set a version of DuckDB as the default one to use. So, we're going to run this command of DuckMan default and then we want to use 1. 4. 2. Okay, now whenever I run DuckDB, bam, get into a session and it's using 1. 4. 2. All right, looking good. I can exit out of this by doing. quit. All right, so let's just go through a quick simple demo showing you how to get into DuckDB and also run a query and then exit out of it. For all this, I'm going to be staying inside of my Mac machine, but Windows users going to do exactly the same thing. Anyway, we're going to run DuckDB and this opens up one, it shows what version we are and it says, "Hey, we're connected to a transient in-memory database. " Basically, anything we put into this database that we create here, as soon as we close out of the session, it's going to be gone. Anyway, if I want to write a SQL command such as this of select 42 as answer, I can, but notice this, if I press enter, nothing's happening here. It's like, "What's going on here? " I can actually press backspace and navigate through, add in a colon cuz you need a colon at the end of a statement and it will actually run. Now, notice up here it says, "Enter {dot} help for usage hints. " {dot} help is a {dot} command and there's a whole bunch of {dot} commands that you can actually run with this. Specifically, I can run things like {dot} databases to see the name and files of each attached database, {dot} tables to list the name of tables, or even something like {dot} schema to show the create statements matching patterns for how a certain table is created. Anyway, the main one I think you should know about is this one up here of {dot} quit and that exits the program. They also have this one up here of {dot} exit and that exits this program with a return code. Anyway, for our sakes, I mean, we can do something like {dot} tables. Once again, there's nothing in here, so nothing's going to be shown even if I did that. {dot} databases, nothing's going to be shown. So, we're going to just exit out of here by just doing {dot} quit. Now, to be clear, those {dot} commands are specific to the DuckDB CLI, so I can't necessarily inside of here run something like {dot} databases and expect the databases to be listed. It's just not going to work. It's actually recommending what actual command you should run if you want to see that. So, running just DuckDB by itself is something I rarely do when you use a transient in-memory database. I don't really necessarily do that alone. I'm going to quit out of this. Instead, I like to actually use a database or either attach to one or create one locally that I can work with. How the heck do we do that? Well, let's inspect that DuckDB CLI command by running {dash} help. Now, this allows us to see where you should be able to run help with most CLI commands and see what they do. And specifically up at the top it says the usage. It has the command of DuckDB, any options, these are the options down here below and they explain it all. And then it has file name and then finally SQL. Notice like these square brackets right here, that means that that parameter is optional. I'm not sure why the file name doesn't have it also cuz technically a file name is optional. Anyway, with file name it's the name of a DuckDB database. A new DuckDB database is created if the file does not previously exist. All right, well, we'll do that in a second, but I want to show this real quick, these options, to show how you could also run this with DuckDB. Specifically, one of the ones that we're going to be running a lot to eventually is this one right here where we do a {dash} C and then you input a command. And this runs the command and then exits. So, let's go ahead and try this out. So, I can run DuckDB and do {dash} C and then inside of quotes put something like select 42 as answer, put a semicolon at the end, and double quote. Run this and then bam, it runs it and exits, so you don't actually even have to go into DuckDB. Now, let's actually create that database file. Remember, we can list duckdb and then the file name of the database to create it if it doesn't previously exist. For this though, it's going to create it wherever we say to create it. So, right now we're in the home directory, not necessarily the best one. I'm going to navigate into my desktop so we can also see it on our desktop when we create it. And I'm going to call out duckdb and then for the database, we're just going to call it jobs. duckdb cuz we're going to make a fictitious job data set. Okay, go ahead and run this and nothing's appearing on my desktop just yet. It's going to take a little bit of time before it does appear. And

### [2:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=10500s) Segment 36 (175:00 - 180:00)

mainly, we got to put something in it if we want to actually even create the database. Cuz right now if I run that command of. tables, there's no tables in there. Oh, it did look like it did create it whenever I ran that command. Um and now it's right there off to the side. Anyway, I think you're smart enough at this point that we're going to actually use some more advanced SQL that we're covering in an upcoming lesson on DDL and DML in order to create a table and insert data into it. Let's just try this out see if you can keep up. We're going to use the command create table and we're going to give that name of the table jobs. We're then put into parentheses. I'll enter down. We're going to put in the column names for this. You need to We need to create that first. So, we're going to do then ID column and that's going to be of the data type integer and then also a job which is like a text value such as data analyst, data scientist, whatnot. This is going to be of the data type varchar. Don't worry data types, I have a whole lesson on that. So, just press the I believe button right now. And from there, I'm going to close out that parentheses and put a semicolon at the end. So, let's create this table and now when I run. tables, and this shows that we do have a jobs table. Um I can actually go in and query that table. I can just do select star from jobs semicolon and I see that we have an ID and job column and notice it says what the data types are for each of these and there's zero rows of data into it because we haven't inserted into. Now, if you noticed I had this stuck DB file dot WAL which is write ahead log and this is a safety mechanism that DuckDB implements in order to make sure that if you ever write any data, it's getting updated properly in the final database. Anyway, don't delete that file anytime when you're in this session right here. You can if it doesn't delete automatically delete it afterwards, but don't delete it anytime you're actually working in a database. All right, so we've created this table. Now we need to insert into it the for the jobs table, we need to insert values. We're going to insert a few values. First, we're going to do the ID and then since the next value is a varchar or a text value, we need to put it into quotes. Single or double quotes is fine. I'm going to do single quotes and put something like data analyst. Put close parentheses and then a comma and then now we can insert other values as well. In my case, I did two more on data scientist and engineer. And then finally, I just close this all out with a semicolon. Go ahead and run this. And now pressing up, I can scroll through different commands. I'm going to select star from jobs and I can see all those different values we just inserted into the table. Now let's see if these files or these values stay inside this stuck DB file. So I'm going to actually exit out of this by doing dot exit. And if you notice that write ahead log disappeared during that, so it should have for you as well. And now pressing the up arrow to go back into run this command of DuckDB jobs. duckdb. We're inside of here. And then I press up again. I want to navigate to our last command to query that jobs database and our data's is in there. So, it's inside of this file. Pretty cool. All right, I'm going to exit out of this. So, the DuckDB team recently implemented a feature early this year that now the UI that you saw in Motherduck can actually be run locally via DuckDB-UI. Let's try this out. So, I'm going to run DuckDB and then -UI. And what this is going to do is pop open a new browser and running on localhost. So, locally on your computer, this server is now showing to you the DuckDB UI. So, this has the same structure that we previously saw in Motherduck in that we have all of our notebooks located here, and then any attached database. Specifically, we have an in-memory transient database with nothing inside of it. And then they have an option to show me the basics, but I'm just going to go to an empty notebook. Anyway, I do want to show this with that database that we actually just created how you could um in investigate that. So, I'm going to go back to the terminal and type. quit or exit. It's fine, either one. And if I navigate back to my browser, it says, "Hey, connection to DuckDB lost. " And that's fine. I know what that's what I wanted to do. Anyway, I want to actually not only specify that UI flag or UI option, but also that I want it to connect to the So, now running this, it pop opens the UI, and instead of having that in-memory database, it has our database, which has those columns of ID and job. I'll go ahead and start a blank

### [3:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=10800s) Segment 37 (180:00 - 185:00)

notebook, and inside of here, I can query this now because I see I collected two jobs, and I so do select star from jobs. I run this query, and I see all the values below. This query currently is limited in the UI as I can actually investigate any of the different columns like we can in MotherDuck. Anyway, one of the cool things about this UI is I can via that UI sign into MotherDuck and we can get all of our notebooks and attached databases. So I clicked it just now and now I'm going to go through the login process. As a reminder, if you installed a version that's not compatible with version of DuckDB MotherDuck, you're going to get this. That's why we went through that whole rigmarole to make sure that we installed the correct version. Anyway, going through the sign up process, I'm going to use Google. And now I have all of my notebooks brought in here along with slowly my databases are loading. And I now have my data jobs attached database along with those other ones as well. And the local database. So all in one environment. So now we have this somewhat hybrid approach, right? We have our jobs. duckdb that's locally. And so whenever I run this, this is run, well, locally. And then if I wanted to, I could also do something like query my data jobs. And for this, I'm just going to do a simple query of select count star from data jobs. And then running this, pressing command enter. So let me, I don't know, I've apparently been working on this course too long. I should have done job postings fact to the table. I'll accept and run this. Anyway, this is running not locally, but this is running in the cloud. So this is a very much a hybrid approach. And what things about this, I actually updated this query to just query the job title short column just for fun. But in this one, whenever I actually inspect into the columns, I can actually see that same type of format that we'd see up in MotherDuck. All right, so this UI is cool, but we're going to get now back into the terminal because I want to show how you can connect to MotherDuck programmatically using the terminal. So for this, I want to get out of having DuckDB set up locally before we get into next part. So, I'm going to just quit out of this and then I can see inside of my web browser it's closed out. So, how the heck do we connect to our database programmatically? Well, I'm going to recommend this right here for the easiest. And that's you can connect to MotherDuck directly from the DuckDB CLI by running the following command. And that's DuckDB and then MD colon. The parentheses aren't necessary unless they're Well, we can add databases onto it. And if your database has spaces in it, you'd want to have quotes. Anyway, the MD, well, like I said, is MotherDuck. We're going to specify that and the database we want to connect to. So, inside of my terminal, I'm going to type in DuckDB and then MotherDuck and we want to connect to I the data jobs database. Now, I'm going to go ahead and press run. And you may be like, "Well, you're connected. " cuz it doesn't say in transit memory. But, if you try to do query to see what are the tables inside of here, it's going to pop this up to go through and confirm that you've actually are who you say you are and want to actually connect to MotherDuck. It will then go through the authentication progress process. And since I previously entered in my password, I didn't have to do any of that. And so, it says, "Hey, the token successfully retrieved. You can display the token and store it as an environment variable to avoid having to log in again. " So, every time we want to get into this, we're going to have to log in via those uh credentials we just went through, which is a pain, but in the next lesson, we will go through how to set up this environment variable. We're not going to do this method. We're going to do another method. Anyway, we're connected, so I can run something like select distinct job title short, and that's from the job postings fact table. Put a semicolon at the end, run this bad boy, and bam, we get the results back immediately. These 10 rows. And so, this is how these results are going to be output to us. This isn't as friendly as the DuckDB UI. So, if you're ever getting into complex tables and views, you may want to shift from the terminal to the UI. So, now that we're connected to that database, I can run that dot help command again. Just a reminder, we can run things to inspect the tables like we did. And we can also thing run things like dot databases to see what databases we're in. So, running dot databases, we can see that although I did type in MD and then data jobs, we have those other databases as well. You just if you wanted to connect to a table or another database, you'd need to actually specify it by saying there's something like sample data and then looking at something like Hacker News. And we're just doing a select count star from this. Oops, I didn't do a colon. So, add that semicolon at the end. And silly me, it says, "Hey, did you mean this one? " I didn't specify also the schema. I'll press up one, enter in the

### [3:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=11100s) Segment 38 (185:00 - 190:00)

schema of HN. Now with running this, okay, 3. 87 million rows. So, I do have access to this other mother duck as well. I just like Well, I'm going to quit out of this to show you. I just like the ability that whenever I connect to DuckDB, specify this of data jobs cuz now whenever I query that job postings fact table or any table, I don't have to specify the database and the schema every single time. It knows what's the main database I want to point at and actually look at. Oh, looks like I got to once again confirm my authenticity. I'm all set and it provided the results. And that's a great segue into what they're going to be covering in the next lesson of actually setting up your environment variable to avoid having to log in again and again. All right, so you now have some practice problems for those that bought the support resources and to go through and practice getting familiar with working with a local DuckDB database and also trying out a few different dot commands some of which we didn't cover in this video. With that, I'll see you in the next lesson where we're going to be getting into installing our code editor we're going to be using for this which is VS Code and it's going to make it a lot easier than typing SQL directly into the terminal. And we'll also be able to keep all of our different SQL files for building a project. With that, I'll see you in that one. All right, this lesson is going to be all about VS Code or Visual Studio Code, our code editor that we're going to be using for the remainder of the course and to build our project. Now, we're going to be walking through it all from how to install it, how to actually operate in it if you're not familiar with it, and then also setting it up to where you can work with DuckDB and MotherDuck within VS Code. And as I hinted towards in the last lesson, actually setting up that environment variable with your token so we can automatically log in to MotherDuck to connect to our data set. But before we move forward with installing VS Code, you need to understand why the heck we're actually even installing it. Well, according to the 2025 Stack Overflow survey of over 25,000 different developers, Visual Studio Code or VS Code was by far the number one choice with three out of four developers using it. Now, personally, I use this option right here, Cursor, which is like sixth on the list and it's basically is a VS Code, it's a fork of it, and it is slightly different in that it's AI enabled and has an AI coding system. But we're not really going to use AI for when we're learning SQL. I don't really think it helps, I think it hinders you. So, I'm not going to recommend that. Instead, I'm going to recommend in the vanilla option with the Visual Studio Code. Now, one quick note on terminology. Over here, I've been calling it a code editor, and it's under here under dev IDEs or developer IDEs. Well, there's actually three different ways you can run code, and also in our case, run SQL code. That's a code editor, IDE, or a database client. As I mentioned, VS Code is a code editor, and its main job is to, well, edit code or edit text files or SQL files. It's super lightweight. As far as interacting directly with a database, you'd have to do it either via extensions or like we'll do through the terminal. Um debugging's limited, and you can use many languages. For IDEs, a popular choice right now is DataGrip, and I've tried in the past, pretty good, but this is like a full development environment. It's heavy-duty, it has built-in everything. You can navigate inside of a database natively all inside of here. Super strong debugging, but it only supports one language, and that's in this case for DataGrip, SQL. The last option is a database client, and this allows you to also run SQL similar to an IDE, and it's around a medium weight. It has native database browsing built into it, and similar to an IDE, it only normally supports one language, and like a database client, well, it only does support one language, and that's of SQL. So, I do want you to be aware of that there are three different options. As you progress after this course, you may want to investigate these other ones of DataGrip and DBeaver as far as investigating IDEs and database clients. Personally for me, I'm sticking with the code editor, and I'm using Cursor, which is a fork of VS Code. Basically, it is VS Code with an AI agent in it. So, let's get into installing it. If you're on a Windows machine, you're going to just go into the Microsoft Store, and then install Visual Studio Code from there. Once installed, you can go then and go search for it. It should pop up as it's appearing. I'm going to go ahead and pin this to the taskbar, and then open this bad boy up. And we're there. Now, let's install for Mac.

### [3:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=11400s) Segment 39 (190:00 - 195:00)

Navigate over to code. visualstudio. com, and inside of here, we're going to go ahead and download it. We'll need to select that we're on a Mac operating system. It put it inside my downloads folder since it downloaded it, and it's I'm going to drag it over and put into the applications folder. Now, go ahead and click on this bad boy and pop it up. All right, we're going to continue now, but we're just going to stay in this Mac environment because the operation of the app is the same whether you're on a Mac or Windows. First up, you can zoom in or zoom out of this editor by pressing command positive or command negative, control if you're on a Windows. I'm going to zoom in a little bit. Anyway, let's go through a quick tour. Over here on the left-hand side is our activity bar. We can see things like in our explorer whenever we open a folder, we'll see the folder contents here. We can search our files inside of here. We can manage source control, which we're going to be doing in an upcoming lesson when we go over our Git and GitHub. You can do things like debugging, and then finally, you can add special extensions to basically upgrade your code editor be more closer to an IDE, like installing a Python interpreter to actually be able to run Python. Down in here in the bottom is how you manage your accounts, and then you also have a settings option as well. All right, I'm going to close out of this and minimize this. As mentioned, this is the activity bar, and then when it pops open, that's the sidebar. And then inside of here is where we're going to edit it, and over on the right-hand side is our chat/AI. I'm going to go ahead and just minimize that for the time being. So right now, we have nothing inside of our VS Code. I'm going to show you my current VS Code setup for this project, and walk you through how I can do SQL before we actually do this. So, this portion is just a demo. You're not doing any of this. I'm just showing you what we're going to be able to do by the end of this lesson. All right, so here we are in my VS Code currently. I have a heck of a lot more files. I open this file explorer and inside my folder of SQL. engineeringcourse that I'm building for this course. I built this course and the projects under this one underscore basics folder and I have the lessons in here along with the projects. Now, what's cool about this is like we know we're going to create two projects in here. So, here's our first one doing EDA and the different SQL files with it. And so, the SQL file appears in the text editor off to the right-hand side does all the syntax highlighting and has the SQL query right here. But how the heck do we run this? So, I'm going to open up a terminal by pressing control tilde and then I want to connect to our data jobs database. So, I'm going to type in duckdb and then MD data jobs. We're now connected. And so, what's neat about this, right? I have a key binding shortcut. So, I'm going to highlight the SQL that I want to put into the terminal. And what we're going to set for y'all as well, I'm going to press shift enter and it inserted it in. I'm going to actually make this a little bit bigger so we can see it. So, it inserted in the SQL query and then I have the results below it. So, anytime I want to enter or run any different SQL files, all I got to do is just highlight them, run shift enter, and it runs it in the terminal right below for me to go ahead and inspect it. Pretty cool. So, I'm back in my other Mac environment where I haven't set up any project folder yet or anything like that. Let's walk through setting up the project folder or the folder that we're going to be making the rest of the contents of this course in and then setting up those key bindings so that we can actually execute SQL inside of here in the terminal. So, I'm going to open this file explorer and click open folder. I'm just going to create it here in my documents of a new folder and call it SQL_data_engineering_projects. I want to be very specific of what this is because we're going to put this on GitHub eventually and share it. So, you want recruiters and other people to know what this is all about. So, we'll go ahead and create it. And then — [snorts] — now that the creator folder is created, I'm going to open it. And it asks if I trust myself. Well, most days I do, so I trust this. It has this walk through. I'm going to once again close out of that and minimize this AI window off to the side. And we now have the SQL data engineering projects folder right here. Let's go through and create some different files and folders in it. Specifically, I'm going to create the first one and we'll call this lessons. Right now, we're in 1. 9. So, I'll actually just create a folder for this. I'll call this 1. 9. And then in point 1. 9, I'm going to create a file. So, we'll call it 1. 9_vscode_intro. sql because it's going to be a SQL file. Now, inside this SQL file, you'll notice at the top it has this of generate code and you can use control or command I or select a language start typing to dismiss. So, basically if you start

### [3:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=11700s) Segment 40 (195:00 - 200:00)

typing, it goes away. So, I'm going to just go and put in some random code right now. Select 42 as answer and then put a semicolon in. Now, this file right here off to the right-hand side, I can tell that it's edited and that it needs to be saved because it has this white dot next to it. You can either come up here and click file save or the pro move is just to use something like command S if you're on Mac or control Windows. So now, let's start up a terminal to try to run this code. And for this, you can get the terminal up a couple ways. You can go to view up here and then select the terminal. Or personally, I like the shortcut they have of control and then tilde. Now, for those Windows users, this is with whenever you go to pop open the terminal by pressing control tilde, by default, like Windows terminal, PowerShell is going to open. Now, I can come up here to the down arrow next to this arrow and you can select a different profile. In this case, I selected Git Bash. That's available right here. Go back to PowerShell and then back to Bash. But, we want to stay in Git Bash. So, what we need to do is change this to select our default profile. And this is going to come up the selection of what we want to do. I'm going to just come up here and select Git Bash. And now, I'm going to close out of both of these. And now, when I run control tilde, Git Bash comes up. Notice here also, you have this open chat control I, start typing then to dismiss. You do have some AI features inside the terminal, but once you start typing, it disappears. Anyway, back to Mac machine. We want to get it now, so I'm going to just type in DuckDB. We want to get it to where this code right here, whenever we press shift enter, well, it doesn't delete it. I'm going to command C that. But, when I when you press shift enter on there, it copies this and then pastes it into here. I'm going to just pop up. I'm going to close this. And then, we're able to run it as well. So, that was a sort of cumbersome to control C, control V, and then press enter. We're going to create a key binding for this. So, we're going to go into the command palette. You can either get for this, go to view and then command palette. But, I like the shortcut of command shift P. On Windows, it's going to be control shift P. So, I'm pressing command shift P. And we're going to search for this one here of preferences open keyboard shortcuts. Not the one with JSON. We're not going to mess with the JSON just yet. It's the same thing as this right here. I'm going to close out of this window real quick. It's the same thing as this. It's just in a JSON format. Anyway, we want to type to search in for the key binding that we're going to be editing for this. So, I'm going to go to type in terminal. And I want this one right here of terminal run selected text in active terminal. So, it's this one right here. You would notice right here there's no key binding assigned to it. So, I'm going to just click inside of here. Or I double-clicked inside of there and we want the key binding to be shift enter. That's inserted into there. I'm just going to press enter and it's inserted into here. So, now opening back up my terminal, I have an active connection to DuckDB. If you don't, set one up right now. Now, all I have to do is highlight that test text. Whenever I press shift enter, it sends it into the terminal below. So, now we're almost at where we need to be set up for the remainder of this course. We've got VS Code set up with our terminal to run SQL locally. And now we need to set it up to where it's automatically we can connect to MotherDuck and we don't have to log in with those credentials. As you recall, I'm going to log out of this little session right here. If I go to log into DuckDB, specifically if I try to attach the database of data_jobs, whenever I do this, yeah, it pops up. But whenever I try to query, it makes me go through this authorization. I can confirm it. And then I'm in. And then I gets me actually connected to it. And I'm finally getting the tables back. But this is annoying to every single time you do this. And programmatically, you don't really want to do this. Now, in order to do this and connect securely, we need an access token. And we can go into app. motherduck. com and underneath settings, they have a section here on access tokens. Now, I'm going to go ahead and create this token. And I'm just going to use this default name of Luke token and I'm not going to have it automatically expire and then I have it create the token. Now, this is the value. I'm going to end up deleting it right after this lesson, so please don't try to steal it and use it. It's not going to be It's going to be useless. Anyway, I'm going to go ahead and copy this bad boy. Now, back inside of VS Code, we're going to use the terminal for this. I'm actually going to maximize it and make it a little bit bigger so we can see. And I'm going to

### [3:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=12000s) Segment 41 (200:00 - 205:00)

exit out of here. So, with this token, what I can do is I can save it to my terminal environment. I can do that by, at least in this session, running this command of export and then it has to have a very specific name that MotherDuck knows and that's motherduck_token. And then I must set that equal to and then inside of parentheses, I'm going to paste in that value and then close the parentheses. All right, and then we'll run enter. Okay, and now that variable is saved in the environment. This isn't a long-term solution. This is just me demoing this. So, if we were to run that echo command that we've learned previously and then if you recall for variables, you put a or a dollar sign at the front of it and you put the name of this, so motherduck_token and I run this, it outputs the token right here. Now, whenever I go to log in to DuckDB, specifically into MotherDuck, into the data jobs, it's not going to ask me anything when I run tables. There was no pop-up for me to validate my credentials or anything like that. It just showed me the tables that I've connected to. So, super cool. But, there's one problem with this. I just exported this mother duck token into the current shell session. So, if I were to go ahead and we'll trash this shell session and I'll use control tilde to start another one. If I were to do duck DB and then try to log into mother duck again, whenever I run a command like dot tables, it's going to make me go through the authentication again because that variable is not saved permanently in there. Well, as the Mac users learned about in that previous lesson, there are actually certain login and shell configuration files that whenever you launch a terminal, it loads these files and goes through like a startup script to load whatever in those files into your environment. Mac users are using Z shell and Windows users are obviously using bash and then potentially older Mac users are using bash. Anyway, there's two main files for each of these. One is your login configuration file and this runs only at login. This is such as the dot Z profile or the dot bash profile. And then there's another one of dot Z shell RC or dot bash RC and this is your configuration and this runs every time there's a new terminal session. So, we want to put it into this file, into your appropriate one whether you have Z shell or bash. Now, I'll go ahead and quit out of here and clear this. Now, for the Mac users, we need to get that home directory. I like where we're at right now. So, I'm going to create a new terminal and then in that I'm going to change directory to the home directory. Now, they [snorts] should have LS -a. They should have created a Z profile or a Z shell RC file. And RC stands for run config. Similarly, I want now Windows users to also create your requested or your required file of that bashrc and bash profile. So, I'm going to create a new terminal window inside that second bash. I'm going to come in here. I'm going to change directory to that home directory. I'm going to do an ls -a and I can see that there's no bash profile or bashrc script. So, I'm going to create those by running the touch command and adding bash profile. And then also, I'm going to add that bashrc as well. Now, when I do lsa with a space in it, I can verify that it's inside of there. Now, I want you to create both of these because it's just good practice to have both of them. They're both necessary. You'll eventually need both of them depending on what you're going to be doing. But, we're only going to be adding to the bashrc because it runs every single session. Same with those Mac users that are using the zshrc and that runs every single session every time you start a new terminal. So, we're going to now edit that. bashrc file. Or, if you're on a Mac, you're going to now edit your. zshrc file. And in here, we're going to put that command that we previously had to export the mother duck token. So, I'm going to put export and then mother duck token, the equal sign, and then it's very important you put inside of parentheses. We're going to be putting the rest of this code. Specifically, that access token that we generated. And then I'll paste it inside of here by pressing command V or control V. All right, it's inside [snorts] of here. I'm going to now write it out by doing control O and then also control X'ing out of here. Okay, but we haven't run this profile this script yet. So, if

### [3:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=12300s) Segment 42 (205:00 - 210:00)

I was to write the currency symbol or dollar sign mother duck _token, nothing's going to come out of there. So, I need to either I can restart my terminal or another trick to do is just run this command of source and then the file itself. So, zshrc. Now, it ran that and so now whenever I I'm going to press up, when I run this mother duck token, the mother duck token actually appeared right here. Similarly, for Windows users, just going through that, I'm going to go into the editor for the bashrc, put that export command with a mother duck token, paste in that token by pressing control V, adding that end parentheses, writing this out, exiting out of this, and then running source bashrc I had a problem with that previously. And now, it refreshes that and so now whenever I run mother duck token, boom, it prints out right below it. Anyway, back on the Mac, if you do the same thing on Windows, this should work now. You should type in duckdb and now whenever you go to md data jobs, it should log you in and you should be able to get into it without doing any verification. And load just like this for you. All right, let's move on. I do want to wrap this lesson up talking quickly about GitHub Copilot and their new integration of bringing in AI to it. This may change more and they're rapidly changing it over time. So, I don't want to go too in-depth with a lot of the features cuz you may be watching it may be completely different. I do encourage you to take use of it. No matter what, I do think they're going to make you sign in. Like if I said, explain this file and I clearly have the 1. 9 VS Code intro SQL file attached to this. It automatically did that, by the way. And I say, "Hey, explain this file. " It's going to have you log in. And if you want to, you can now go through and set up a GitHub profile by continuing with GitHub. You'll need to then come down here and create an account. Like I said, I'll walk through creating an account in an upcoming lesson. So, we're not going to go through much of that right now on how to actually create a little account and all that cuz I like I said, I'll cover it later. Anyway, it went through and now executed this cuz I logged in and explained a summary of this SQL file. So, this is pretty cool, especially this chat off to the side. If you're getting stuck along the way, it's a good way to just go to something directly and you don't have to jump into the chat GBT. I'm going to close out of this welcome message. One thing to note though with this, if I toggle go down here to this bottom little icon of it, if I've logged in now, right now it has inline suggestions turned on. So, if I were to come in here and start typing and select, it's going to want to go ahead and auto complete this with some AI generated stuff. While we're learning this, this is going to be very inhibiting and can slow down your progress because you're not relying on your own memory, you're going to start relying on the AI too much. So, I recommend if you turn this on to turn off it all for all files. In this little thing is pretty neat too because you can see how much percentage of used. They have right now I'm using the free plan and it looks like it's pretty generous in the amount of different prompts you can give to it and the amount of suggestion you get with it. All right, so we now have gone through taking a little bit of a break from SQL to be able to learn a lot about terminal, setting up DuckDB locally, and now getting you set up with VS Code. In the next lesson, we're going to be jumping right back into SQL and specifically we're going to be focused on data modeling, some little theory behind SQL. With that, I'll see you in the next one. Oh, by the way, there's no practice problems for those that purchased the supporter resources. It's a little Christmas present for you. All right, with that, see you in the next one. In this lesson, we're going to be more theoretical in that we're going to be diving into our first part on data modeling. We're going to start out first by understanding and breaking down databases and schemas and also tables so you understand how and what they're made up of and what you have access to. Then from there, we're going to be diving further into this diagram on our data warehouse that is our ERD or entity relationship diagram. Specifically, we're going to be diving into understanding what those tables are, what those lines are for relationships, and how they're related with keys. And then finally, we're going to wrap it up by using some SQL in order to investigate these different type of things within a database. All right, with that, let's jump in. First up, just a quick recap on this hierarchy that we've seen before. We have a database and that's our top-level container. Inside of it, you can have one or more schemas and this is a logical grouping inside of a database.

### [3:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=12600s) Segment 43 (210:00 - 215:00)

And then underneath this or underneath each schema are tables and they show the actual data in some sort of a row and column format. Now, this specifically is just showing related to the tables. Databases are a little bit more complicated than this that we're going to actually going to get into. But, this is directly related to how we have our database modeled with our main schema inside that data jobs database and then all those different tables underneath it. So, let's break each one of these concepts down a little bit further so we understand how they're actually made up. First up is a table and we're going to start at the bottom-most layer and then work our way up. Anyway, tables, as you've got by now, are made up of columns and then also rows. Inside of there, an individual unit in the column and row is a value. But, what you may not see on the surface is that a table also can have something called a primary key. And this is a unique identifier number for a column. It shouldn't be repeating, at least in that specific table that it's the primary key for. Now, related to that is a foreign key. A foreign key is a relationship to another table's primary key. So, in our case, that company ID is related to the company ID in the company dim table. And just for fun, let's investigate it real quick. Inside of VS Code, I created a new folder for 1. 10, and I'm going to create a new file. call it 1. 10 data modeling. sql. We want to make it a SQL file. For this, I'm going to query the job ID, job title short, salary column, and that company ID column from job posting facts, and only get 10 rows back. In order to do this, we need to start up a terminal, and then connect to our Mother Duck Data Jobs database. So, from here, all I have to do is select this, and then with our key bindings, enter it right into the terminal. With this, we can see those IDs, seeing that they are unique actual values for each, and then our company ID. Now, there's no repeating values in this one right now, but there are technically repeating values in this table. Anyway, I also want to look at that company dim table, specifically at company ID and name. And so, with that table, we have that company ID, this is the primary key in the company dim table. So, tables are really the building blocks in what we're working with inside of a database. Let's move up now to the next layer of schemas. So, as we've learned already, inside of a schema, we can have things like tables, which are a logical object. Listed underneath it are some examples. Specifically in our case, we have job postings facts, company dim, and skills dim. Now, schemas can actually have other things as well besides tables. That's why I said that hierarchy diagram was sort of a simplified view. One other thing it can have is views, and these are virtual tables. So, I could use a SQL query to find out company info and make it into a view, and others could go to this view and actually, well, view it. And this is made from our tables. Now, the other thing we can have, and I've sort of grouped it all together, is something called a semantic layer. Basically, think of this like logic and also metrics. In here, I can define functions or macros, and if you've worked with macros in Microsoft Excel, it's very similar. I could create some sort of function and then use it within a SQL query. I also could create things like metrics that go through and define how we're going to evaluate things like our salary. And then finally, things like business logic that allows us to have like reusable meanings to create things like flags or buckets. Anyway, both views and semantic layers are built on top of tables. Like we can use them and are built from tables. So, we need the tables as the hardcore undercore of the structure. And the main point I want you to get out of this is that database schemas aren't just related to tables, because you may work in other databases, and there's other things in there that you need to be aware of. All right, moving up last to the highest layer, and that is the database. As we've learned already inside of a database, you could have one or multiple different schemas that has your tables and views and everything else. But, once again, there's a little bit more in there that you need to know about. One area is security, and this defines things like users, roles, and more specifically, the permissions for these different users and roles. These type of things can be managed at a larger company by a database administrator, but if you're a smaller one, you may end up managing it yourself. Next up is the metadata, and the purpose of this is to describe the database, the schemas, and the tables. They describe how to actually go through and build and the structure for all of our records. And at the end of this video, we're going to be going through how to look at things like information schema in order to investigate further

### [3:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=12900s) Segment 44 (215:00 - 220:00)

about our data warehouse. So, if we have the metadata on, you know, how a table is built, well, we have to have the data that goes in that table. And that is in storage. Now, in our case, we would be taking that data would be inside table spaces, but you could also have physical files inside of a database and they'd be stored here. So, now that we have the basics of database schemas and tables, we're going to get into breaking this diagram down. And this is an ERD, or entity relationship diagram. And now, when I'm reading this, this is like repeating diagram twice, like saying ATM machine. Anyway, this is a map of our data warehouse. And there's a lot of key information that you can extract out of it. As we already learned about our job postings fact table, job ID is our primary key, doesn't it by that PK, and company ID is our foreign key, doesn't it by FK. And then we have these lines linking between these other three tables, which has information on things like skills and also on our company, which may get you to thinking, "Look, why the heck do we split up this data across different tables? Why make it complicated and have to use this ERD? " Well, going back into VS code, if you recall, whenever we queried that job postings fact table, for an associated job posting, we just had, well, in this case, a four-digit number describing the company. And that four-digit number doesn't really take up a lot of space in there. Now, if I go ahead and query that company dim table and show all the different columns associated with it. Remember, we have four characters in our original job posting fact table, but now see for each company, we have a lot of different data associated with it. Not only it's name, but also links to the company itself and Google information and whatnot. Anyway, breaking it up into tables helps solve this problem and allows us to basically make a more concise data model. And an added benefit of this is improved data quality. So, say in our case, look at this, I queried a company dim table looking for a company named Facebook and a company named Meta, right it renamed. I now have a separate table that keeps all this company information, so it allows me to keep older companies and prevents me from having to overwrite those with those new companies. So, now we're going to break down three main components of this entity relationship diagram. And the first one is the entities, which are our tables. We're then going to go into keys and then followed by the relationships to understand what these lines actually mean. So, our tables, as we previously discussed, look like this. They're arranged in column and rows, but that's not how they look in the ERD. In there, they look like this where we have our table name up at the top, and then from there underneath it, you have your column names. And then there's a designator off to the right hand side typically that would say something whether it's a primary key, foreign key, or both. So, as we mentioned in job postings fact, our primary key is job ID, and we have one foreign key of company ID. And this is related to the primary key of company dim. Now, job ID can also be used as a foreign key in other tables. In our case, the skills job dim, which is our bridge table that bridges to our skills dim table that has the actual skills in it. So, I queried in VS Code both the skills job dim table and the skills dim table. Now, let's start with that skills dim table first. As a refresher, this has a column of skill ID, the skills, and then what type of skill it is. In the case of Python, it's a programming skill. Anyway, our bridge table that bridge together our job postings fact table and our skills dim table has two different columns in it. And that's skill ID and job ID. Both of these are the foreign keys because they associate to those two opposite tables. So, job ID to job posting facts and skills ID to the skill dim table. But also, when we expect inspect the ERD, we can see that this skills job dim is not only the foreign keys for those associated tables, but also the primary key. Basically, it's a dual primary key. And this is called a composite primary key because the combination of the two columns actually makes a unique value. And remember, a primary key needs to be a unique value for each row. Going back to actually look at that skills job dim, you can see that for each one of these lines, if we were to combine those four digits on one side and the one on the other, those are going to create a unique identifier for that row. So, we understand the entities or tables, and we understand the keys. What the heck now is going along with these lines connecting between the two? If you notice, they have these special symbols

### [3:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=13200s) Segment 45 (220:00 - 225:00)

around where they actually connect to the tables. Well, there's three major types of lines that you can find. There's a little bit more, but we're only going to stick to the main ones that you're going to encounter. You can have a one-to-one, one-to-many, or many-to-many. The diagram line of one-to-one, you can see this in here based on the fact that it has for one, it has just one single line on a side. If we look at the one-to-many, it has the single line on one side, but then on the other side it has a crow's foot. And this shows that it's a many relationship. And then for the many-to-many, you have crow's feet on both sides. Now, what the heck does this actually even mean? Well, we're going to show it visually. For a one-to-one relationship, one record links to exactly one other. So, in our case for 1 2 3 4 5, you would have exactly one match in the other table of 1 2 3 4 5. This one is like probably the second most common one you'll see. And the most common type is one-to-many. And it's where one record links to many others or just one other one as well. But the main thing here is it can link to many different ones. In our case, we do have a direct link between one-to-one, but for two in table one, there's three different values of two in table two. And that's the common relationship or the same type of relationship we have in every single one of our tables in our ERD. Specifically, going to that company table, we have only one unique ID for a company, but then there could be multiple associated job postings in the job posting fact table. So, one company to many job postings. Now, the last one to cover is the least common that you'll ever encounter, and that's many-to-many. You could have many records linked both ways. This is cause is a lot of different confusion, and it just makes querying your database super difficult. We're not dealing with this, and if you see this in the real world, run. For us to duplicate this, we would relating it to our company example that we just did, we'd have to have many duplicated companies, and then connect those to many different job postings. All right, so we went through basically all that theory. How the heck do we actually investigate this? Specifically with like this database here, how do we get in and inspect this metadata? Well, we can use information schema. An information schema is basically a collection of read-only views that tells us information about the metadata inside of a database. We can look at things like tables, columns, views, or table constraints. And you may be like, "What the heck is a view? " Don't worry, we're going to be getting into that and viewing it, but it basically looks like a table. And so, we're going to be covering some different functions from information schema, and then you could check out the docs here for DuckDB on what you could actually investigate with this. But the main purpose of this is this information schema is not only applied to DuckDB. Other databases implement this view or catalog of metadata that you can go in and query. So, what you learn from information schema can be applied to other databases. So, how can we query this? Well, feel free to follow along with me in VS Code. For this, we're going to do I'm going to do something simple right now of just selecting all the columns, and we're going to do this from information schema. And this is a collection of views, so we have to actually use the dot notation to then go in and access different things in it. Specifically, if we want to look at something like the tables, I'm going to type in dot tables, and then add a colon. Remember, in the terminal, we need to connect to our database inside of MotherDuck on what can do we can just connect to it right to the data jobs database. So, this first one we're going to look at the tables. I'm going to run shift enter. Now, this thing is a behemoth because it's including not only our data jobs database, right? It's including other ones as well. We're going to make this simple for all our future queries using this, and that we're going to query or filter for the table catalog of only data jobs. So in here going to add a where statement cuz it works just like SQL and as table catalog. And for that we want to look at data jobs. Now data jobs is a text value so you need to put it in quotes. So going to select this all and then run shift enter. And now we're querying just data jobs. We can see the schema it belongs to, the table name, and then the table type. There's also some other columns that are hidden here. As I can see from down at the bottom it says there's a there's 13 columns but only eight are shown. But those are the main ones I want to show anyway. So let's dive into some other things as well about information schema. We can also look at things like the columns. Selecting this all and running

### [3:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=13500s) Segment 46 (225:00 - 230:00)

it. And then for this one it shows key information about the columns. Now one thing unfortunately, especially in the terminal view, you're limited by how wide your terminal is. Once again, we got truncated to only 40 45 columns but only seven columns are showing. So if I wanted to view pertinent information from there, I'd want to go into the documentation here I can go directly to on information schema, go into columns, and then inspect which ones I wanted from this. And then update my query for what I'm actually looking for. In this case, I could put in hey I want to look for table name, column name, and then also the data type. The other option is use it in a more friendlier program for looking at very wide tables such as my mother ducks UI or duck DB's UI. Now there's only a couple more I want to look at to show you for this and that is regards to hey what if you want to look up things like the keys, where are the primary and foreign keys? Well we could use something like the view on table constraints. Running this bad boy, we can see that we have this one entry here for on the company dim table that the company ID is the primary key. And scrolling on over as well, we can see that hey, the constraint type is primary key. And then conversely related to that, in the job postings fact table, we have company ID and that is a foreign key. And the constraint for this is foreign key. Now, this table constraints also checks if there's not any nulls in certain uh IDs specifically. Make sure there's no null values in a company ID or a job ID or whatnot. So, this is a pretty useful table. But if you only cared about keys, maybe something like key column usage, that would be a better table to look at as it simplifies this a lot more into identifying what's the primary and foreign keys in a very similar type format. Now, another command to inspect metadata that is slightly easier in my opinion is pragma. Now, pragma is the keyword that you'll use and then you could do things like hey, list all tables by saying show tables. Anyway, it's a lot simpler and easier, right? The only thing about this is that pragma is an is a method that was adopted from SQLite. And so, what you learn with this can't necessarily be applied to any other databases except for SQLite of course in this case. So, for this, I'm going to just type in the command of pragma and then we need to know that we can use show tables. Highlighting this all, sending it down there, it will tell us show us the tables that remember we connected into the data. jobs database. So, it's only showing us those associated ones. We can also do something like this of show tables_expanded. And this goes into breaking down not only data. jobs database, but other ones as well and a lot more information about the tables. Now, besides pragma, the other common one I find myself using specific to DuckDB is describe. And this is a great way to just quickly describe tables. So, for this, I'm going to just put in the command of describe and then after it, you can put it in any table. I'll put it in job postings fact. Running this below, we get information about the different columns in that associated table, such as the data type, whether it's null or not, if it's a primary key or not. So, a lot of quick information at a quick glance. And when you're working in DuckDB, it can be a lot quicker than writing out this entire statement trying to go into the information schema to look at the columns. But, I do want to end on this. Remember that information schema is very common across all databases. So, I have some practice problems now for you to go through and get familiar with querying information schema in order to look at different attributes in there. So, now that we have this knowledge understanding how our tables are connected with relationships and how we're using primary keys and foreign keys, in the next lesson, we're going to now be going over joins and how to actually join these tables together and perform analysis with that. All right. With that, I'll see you in the next one. — Now that you've mastered the basics of data modeling and understand why our data is split across multiple tables, we're now going to jump into joins, specifically these four types of joins, and understanding how different joins affect the data returned with it. We'll be going one by one to cover these. So, there are four major types of joins, a left join, right join, inner join, and outer join. These Venn diagrams that are associated with each of these joins are showing you based on a table, assuming we have a table A and table B, what are the records returned from each of these. We'll be going into the specifics of each as we break each one of these joins down. Now, one note is that these are only the four major types of joins.

### [3:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=13800s) Segment 47 (230:00 - 235:00)

There's also one more type of join called a cross join, and this produces a Cartesian product of two tables. And that's fancy talk for that's super complicated and beyond the scope of this for right now, but if you do see it, know it is an advanced concept and it is available. For each of these joins, we're going to be walking through how we can connect the different tables in our data warehouse. And then by the end of it, you'll have a better idea of what join you should actually using to connect this data. First up is left join, and left join and both inner join or two of the most common joins that you're going to be seeing and using. So, you should probably get used to those two. Anyway, left join returns all records from the leftmost table, table A, and then the matching rows from the right table, table B. So, for this example, we're going to be walking through connecting the job postings fact table, which will be our A table, and we're going to be connecting it to the company dim table as our table B. We'll be joining these using that company ID column. The basic syntax of this is we use a select statement to select the appropriate columns from each table. Notice we do have an alias for each table, and that is defined below the from statement in that left join. Specifically, we would start it with left join, specify the table, and then you can give the table an alias, and then we use the keyword on to specify how we want to connect this. In this case, connecting job postings fact ID column with the company dim ID column. Also, as noted in the from statement, we did have an alias as well for this table. All right, here I am in VS Code. Let's go ahead and type this query out. For this, I've created a new SQL file where we actually can go in. Ooh, actually didn't actually create it as a SQL file, so make sure that you're creating a SQL file by ending it in {dot} SQL. Okay, that's working better. For the layout of this, trying something different so we can actually see the output of the queries. I'm putting the terminal to the right-hand side. You can actually change it yourself if you want to by right-clicking terminal, going to panel position, and then changing it to right instead of bottom. As always, we need to connect to the database itself. So, I'll specify duck DB and then connect to our data source of data jobs. First thing I'm going to do is write that select statement, and I'm going to fill in the columns after because what is the aliases I'm going to do? Um I don't know. Just Well, I do know, but uh I want to make a point that sometimes it helps filling in the next portions first with the table alias so that way you know what it is. So, specifically we're coming from from, and that's going to be the job postings fact table. And we're going to alias as JPF, basically the starting letter of each one of these. Pretty common in the data engineering world to do it like this. And then underneath the directly underneath the from statement, that's where you're going to include your join. In this case, it's going to be a left join. And we're left joining that company dim table. And at this point, we're defining that table so we can define it as an alias, and we're going to find the alias of CD. And then I like to put what we're joining on indented on the next line. And specifically, we need to join the company ID from each of those tables. So, JPF {dot} company ID is equal to CD {dot} company ID. All right. And then for this, we're going to just query all the different columns right now. Just And we'll sort them down in a second, but I'm going to indent in and to specify JPF {period} I want to star and CD {dot} star as I want the star from basically all columns from each. Also, there's a lot of different data, so I'm going to initially while we're building this query, I'm just going to limit it to 10. Put a semicolon in there. All right. I select it all and then run shift enter. And this is good now, we can test what is going on with the query if we have any errors. And we did because I can see it says cd. period got confused by this period makes here, and I looked around it and I see, oh shoot, I didn't include a comma at the end of that. Well, let's try again. All right, this time it actually worked. And I want to show more columns, I'm going to slide it over here and try it again. All right. All right, now the columns are actually fitting, so let's now actually fine-tune this down to the columns of interest. Now, I'm going to do something first to just make an example, we're going to correct it after this. I'm going to go through and list the columns without any alias. Specifically, we're going to getting the job ID column, job title short, company name, which is just name, and we'll give it the alias of company name, and then also job location. Okay, when we run this, it should run just fine. Let's go ahead and try. Okay, it does all of these return back. Now, this is very risky, so I don't I wouldn't recommend it. Say in the case of our company ID, right? We have company ID in both of these tables. If I were to put company ID in here and then run this

### [3:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=14100s) Segment 48 (235:00 - 240:00)

query, I'm going to get this error right here. Ambiguous reference to column name company ID. And it will specifically Dr. B D's pretty good, it actually tells you, "Hey, use the alias with that one because it's ambiguous. You don't know which table it's from. " Whereas something like job ID or job title short, it is specific to that table. But as good practice, you should just go through and list it for every single table, even if you know it's not ambiguous. Here I did the company ID from the company dim table, but it didn't matter, I could do that one. And now running this, I went a little too quick, I didn't specify the correct table to take the job location from. Now running this, okay, it works just fine. So, we've now demonstrated how to connect the left table, table A, to the right table, table B, using something like a left join. But, what the heck is actually going on with this left join? So, we've made this fancy dancy diagram for this. And one note, this is fictitious data in here. Specifically, I've made it to where there are, in table A of job postings fact, there are company IDs that don't have an associated company ID in the company dim column. Specifically, here you can see how company ID zero and there's nothing in the company dim column for that. And vice versa, we have this one on three that's going to Experian and we don't have an associated job with this. This is fictitious. This is not what we actually happens in the data set, but I did this to illustrate an example to show what happens when you do a left join with this type of data cuz that's common in the real world. Anyway, let's go through it step-by-step. Remember, remember left join takes all the data from that table A, that first table, and preserves that. And then table B, it only brings over those that are related. So, in our case, table A is preserved and you can see job IDs 1,000 and 1,004 are preserved in there. However, there's no associated company for company zero, so therefore the company name is null because there's no associated one. In situations like this could job ID where the company ID is repeated, it just lists it twice. And then in the case of Experian for company ID three from the company dim table, it is just not brought over with a left join. So, going back to VS Code, I want to actually see how many jobs are returned whenever we do this left join. Now, just to recall, if I counted all of the columns in our job postings fact table, we would see that there's 1. 62 million jobs. And then, whenever we do this one now without our limit statement, should have the same value in it cuz as I said, right? There's no jobs without an associated company ID in it. And actually, that doesn't matter. The more important thing is in table A is preserved. So, in this case, we have 16159303 and the same of 16159 Well, 30. I don't know why I said 03. Um the same amount of values are preserved in each. It shows that the left join is preserving all those values. Next up is a right join, and this returns all records from the right table, table B in this case, and the matching records from the left table, table A. This is so much less common than left join and inner joins, but I want to cover it just so that way you're aware of it. Anything you can do with a right join, you could basically, if you think about it and see how the diagram works, you could just flip the tables and do this with a left join. Anyway, for the syntax of this, it remains exactly the same. The only thing we're going to do is change that left to a right. And then, running this query here in VS Code, we have 1615930, which, scrolling up, is the same exact values from the left join. And this is just because for every company in the company dim table, there's an associated job posting. So, I expect them to be the same. So, let's walk through this right join using some more, well, unclean data, if you will. And remember, for the job posting fact table, we have this company ID is zero with no associated company in it. And along with that, we have the company dim table with this three of Experian that's no associated job with it. So, different from our real data set. Remember with this, in the right join, we're keeping all records or preserving all records from table B and only the associated ones from table A. So, starting with table B in this case, we have Netflix go over. And then, notice this. You would think there's only going to have one entry for Meta, but remember, Meta has two associated job entries in it. So, in this case, it's going to duplicate down and you're going to have meta from that main table duplicated twice. This is called fanning out. Next up from there, we have Google, which has a job posting and it appears. Now, in this case, I mentioned Experian, but also Uber does not have an associated job posting, but that's in table B and we

### [4:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=14400s) Segment 49 (240:00 - 245:00)

want to preserve all records from it. So, it does get brought over, but we have only null values for the job ID and job title short. Once again, just to reiterate, right joins are less common. If you can rewrite with left join, well, you most absolutely should. And if you see this in the wild, run. Next up is inner join and this returns rows when there's a match in both tables being joined. Applying this in our previous query, I'm going to change this right now. All we're going to have to do is change this to inner. And whenever I run this bad boy by selecting all, shift enter, we once again get the same amount of rows of 1615930. Just a reminder, that's because the job postings fact table has associated companies and company has all associating job postings with it. Now, one thing to note from this and what you'll see also is that inner join was the original default join in SQL. So, because that, you may see some people write this out as just join and this is going to do exactly the same thing as an inner join. I'm not a fan of not specifying that this is an inner join, so I would always specify it just for best practice. So, let's show this on this fictitious data set that has some jobs and some companies that are not associated. Remember, inner join is only going to return the rows when there's a match in both tables being joined. On the job postings fact table, we can see that yes, there's no company for zero, but there is one for one, to two, and also four, which are Netflix, Meta, and Google. And then over here, the two remaining, Experian and Uber, they don't have any job postings associated with this. So, from our results table, we can see we just have job postings from Netflix, Meta, and Google because that's what overlaps. The last type of join up is a full join or a full outer join. And this returns all rows from both tables. If keys match, rows are combined. If keys don't have an associated value, then the missing side is null. This is great for checking for data completeness and checking for basically orphan data, data that doesn't have an associated value in another table. Anyway, let's demo it in VS Code. So, I'm going to change that inner that we were previously doing to a full outer join. And then running this bad boy, we get once again, all the same results, similar to all the last tables, cuz the data clean up that I did on building these tables was so good. So, unfortunately, we have to resort to the diagrams to better explain this. Um what I want to go over before getting into the diagrams is this. So, full outer join works, but also a full join only, writing that full join syntax, can also be used. I would continue to write or I would write full outer join, that's pretty common. Using just full join is pretty common, and so I would say it's acceptable uh to write this as is. Now, the reason why that outer is optional is because things that we did previously, like our left join, I'm going to go ahead and run this bad boy just to show that it does work. The left and both right joins are actually, technically, left outer joins and left right uh or right outer joins. Anyway, these produce the same amount of things, but it's very common as well with this not to write the outer in there. So, if you don't want to write the outer, completely understand in the full. So, what's going to happen whenever we combine those two fictitious tables in our results table? Remember, in this one, this full join, it's going to return both values in both tables whether it has an associated record or not. So, in our case over here, we can see that it includes the 1,000 all the way to 1,004 job IDs because it's in the results table. And then additionally, we know that we used one, two, and four in there. So, included along with three and five. So, we have Experian and Uber down here with null values. And then one that I sort of glossed over is that company ID of zero that doesn't have a company dim. That one's going to have a null value in it. So, we just went through the four different types of joins and specifically with that company dim table. Although, that one wasn't really exciting because it returned the same results on each one. Now, we're going to get into looking at going through the most popular joins with bringing in our skills table in order to show associated skills for job postings. Now, to do this, we're going to have to do two joins. We're going to have to join into our that bridge table of skills job dim and then from there connect into our skills dim table. Let's look at these tables, just the top 10 rows of each just to in case you haven't looked at it just yet. In our skills dim table, which is that bridge table, we have associated

### [4:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=14700s) Segment 50 (245:00 - 250:00)

skill ID and an associated job ID. And then our in our final skills dim table, we have for each one of those skill IDs, the skill itself, so something like skill SQL, Python, or R, and then what type it is, programming, analytics, whatever. So, from our job postings fact, we need to connect to the skills dim table and then from into our skills job dim table. Now one note with this one, unlike the company dim table, there are some jobs without an associated skill. There's no skills required for it. Probably a job I want to apply for. Anyway, let's work through figuring out what type of join we're going to be using for this. For this, we're going to start with a blank query. We're going to keep it pretty simple. We want to get the job ID. Additionally, from that job postings fact table, we want to get the job title short. And then for fun, we're going to bring in from the skills dim table, which we're going to give the alias of SD, we're going to bring in that skill ID. And then also from this, we're going to get the skills from it. Now the skill ID doesn't matter. We could have gotten that from the skills job dim or the skills dim table. Doesn't really matter. I think I just said we're going to get this from skills job dim. Doesn't matter. Anyway, for this, we always want to start with our leftmost table. In this case, it's our source of truth. It's going to be our job postings fact table. And we're going to give this the alias of JPF. Now we're just going to start first with a left join just to see how this does. So we're going to do this left join of the skills job dim table aliased as SJD, although we don't even use it in this. And then we need to specify the on and that's the job ID from the job postings fact table and the job ID of skills job dim, which actually we do use the alias. Anyway, I like testing queries out as I build them. So actually I'm going to just go ahead and just comment this out right now and also put a limit on there of 10. Let's go ahead and run this bad boy. And so good that we are testing this out because SD is not defined yet. All right, so we got the J in there. We can now go ahead and run this. Run shift enter. All right, so this is good. It's working. We can see that for something like 4594, this is a it's got three skills associated with this job posting. And then, 4598, which was skipped in between here, this has, well, apparently no associated skills. Right, we're doing a left join. Now, it doesn't mean 4598 isn't included here. It's probably the way the query runs, it does the null values at the bottom, so that's why it's missing, at least in the view for this limit 10. Anyway, let's continue on with getting the other left join. Specifically, we want to get that skills dim table so we can get those skill names. We're going to give this the alias of SD and this one we're doing on the skill ID of the skills job dim table and the skills dim table. Oops, I got a typo right there. Okay, I'm going to also untie uncomment out that skill so we can actually start seeing those skills. All right, let's run this. Okay, good. This query is working. And based on inputting in those different skills, it actually now reorders those job IDs. So now, let's think about this in what case we'd want to use this. We are doing a left join, and left joins are used anytime we want to preserve all of the values from the job postings fact table regardless if they have an associated skill. So, I'm going to go ahead and remove this limit 10. We're going to see how many values we actually get for this. We're going to experience that it's going to fan out because there's a bunch of skills associated for job postings. Now, we're at 7. 48 million rows. And like I mentioned, the null values are all at the bottom. Now, remember I said the two most common types of joins that you're going to be encountering are left joins and then also inner joins. Inner joins are only going to return matching values that exist in both table A and table B. So, in this case, job postings that don't have an associated skill won't appear. So, I'm going to copy this query right here, paste it underneath here, and I'm going to change this left to an inner join in both places. All right, let's see how much this query changes. We should expect the number of job postings to go down. And they do. Went from 7. 48 million to 7. 19 million rows. So, basically, 200,000 jobs don't have an associated skill associated with it. Now, so we're typically always going to want to preserve all jobs regardless if they have a skill or not. Because of that, left is going to be the most common one they use. But just to see what happens if we tried instead of this inner right here, what if we tried full? And remember, that's going to return or preserve all the contents from table A and for table B. In our case, when we

### [4:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=15000s) Segment 51 (250:00 - 255:00)

run it, I would actually expect to get that 7. 48 million job postings. Yep, same one here. So, this yields the same results as our left join. So, that was just for funsies. I would, however, recommend anytime we're wanting to do an analysis and we're wanting to preserve the job postings fact, we're going to be sticking with a left join. Going back up to our company dim table as well, we're going to be maintaining the same practice as well. If we're ever joining with this one in order to preserve all of the job postings in it, we're going to be doing a left join going forward. And this, running this query, would preserve all the records of 1. 62 million jobs. All right, so up to this point, I would say that we've covered all the basic SQL syntax to perform analytical analysis. Because of that, we're going to be jumping into the project after the next lesson on order of execution, understanding how actually the SQL database engine processes queries so you can get some background on what's going on behind the scenes. It's going to be a little bit technical and theoretical, but it'll give you a background and really strengthen your knowledge on understanding order of execution. For those that purchased the course resources, you now have some practice problems to go through and test out when you would use different types of joins in different scenarios depending on what your final need is for your engineering query. All right, with that, I'll see you in the next one. In this lesson, we're going to be diving deeper into understanding the theory behind how your SQL queries are executing. And the main purpose of this is to be able to understand why also you have to write certain queries in a certain order and how it actually gets transferred within the database engine itself. So, first, let's dive deeper into how a query processes cuz this is going to be important for understanding where order of execution actually gets carried out. So, if we recall from this diagram we went out in the very beginning of this course, SQL works as follows. We first go to our computer and we type in our favorite text editor a SQL query. From there, we press enter and it sends it to the database engine, whether that's locally or up in the cloud. Now, the database does some work that we'll go into in a second, but the results are then output to a table, which is then sent to your computer to actually view and save these results. Now, let's build on this slightly more and we're going to be going into the query processing architecture. Inside of a database engine, there's typically three, sometimes more, components, but we're going to focus on the three main components of a parser, an optimizer, and the executor. So, let's break each one of these down. We're going to start first with the parser. And this reads your SQL text and checks to make sure that it's valid. Every time I forgot a comma or didn't name a column correctly, that was done by the parser. Now, it only doesn't only just check for that, it also builds out an internal tree structure called the parse tree. This is a structured representation of what you meant and is then sent into the next thing of the optimizer. This, as the name implies, figures out the most efficient way to run your query. It can reorder your joins or even sort out different strategies for scanning and hashing. This is rewriting your SQL query to make sure it runs as fast as possible. All right, last up is the executor and this actually runs that optimized plan step by step. It reads your data, applies any filters and joins, groups, aggregates, and returns the rows necessary. After it goes through that, you have your final result set. Now, as a quick refresh, there is a certain clause order that you have to implement when writing SQL queries and this is what's checked in that first step of the parsers. Not only checking to make sure that you got those commas and names spelled correctly, but also ensuring that you type them in the correct order. If you write them in the wrong order, the parser is the component that will reject it and throw a syntax error. For example, here in VS Code, I'm connected to our database and if I were to write select star and then put a limit statement above our from, which we know is out of order, if I try to run this, one, it's going to take a second to reply because it has to figure out, oh, there's a mistake here. So, that's one thing that you know that you've probably messed up if it's taking a long time for the query to come back. And then, we finally get this of parser error, syntax error at our new from. And so, it gives you the clue, parser error. That's where it's done, in the parser. So, now we understand that the order the right queries is important because the

### [4:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=15300s) Segment 52 (255:00 - 260:00)

parser is going to check it. But, how the heck do the actual commands or the keywords get executed after this point in things like the optimizer and executor? Well, for this, there's a very similar order that we're going to flow through in showing the step-by-step process for this. First, it looks at any froms and joins to collect all the tables we need. Next, filtering happens with the where expression to get the number of rows down. From there, we group any rows that share any properties. Then, because we did that aggregation with group I, we can then do a having statement to then filter those groups based on those aggregates. We can then select which columns we may need for the final data set that may have been used up above, but now they're not used below. And then, from there, getting to order the results in a certain order, and finally, limiting to the number of results we want. This order of execution is really similar to that SQL clause all order. I don't want you to confuse cuz they are carried out in different parts of the database engine, but they do follow a similar. The only one that is out of order here, if you noticed, is step five of what columns you're selecting. In that, we would name that in the SQL clause order up at the top. But, besides that, how you actually write that clauses in that order is pretty similar to how it gets executed in the query engine. Pretty cool. Well, the order of execution is carried out in technically two areas. Specifically, first, the optimizer implements that order of execution and puts it in the correct order, and then sends that over to the executor that executes that correct order. So, let's go through building a SQL query and showing and understanding how this order of execution actually comes to our advantage and how this all works together. For this, we want to find the top 10 companies for posting jobs, and in order to make this list, they have to have greater than 3,000 jobs. So, the first thing we're going to do is get into selecting the columns that we want. Specifically, I know I want from the company dim table, I want the name, and we're going to name this company name. And then from there, I want the count of job postings. So, I'm going to put inside of here JP F and then period star to count the rows in that column for the count of the job postings. I'm going to give this an alias of posting count. Next up is the from statement. And we want the job postings fact table as our table A, and we're giving the alias of JP F. And we're going to We want to preserve all the job postings, right? So, because of this, we need to do a left join. And we're doing that on the company dim table as CD. And for the ID of this, we want to match on the company ID of each of those tables. Okay, there's enough query. Anytime I get this many lines in, it makes me nervous that there's a mistake. So, I'm going to go ahead and just run this bad boy. And as we can see, we got into our first error, right? Column name must appear in the group by clause or must be part of an aggregation function. And so, let's put that group by in. Specifically, we want to group by that name. Now, we're going to list CD. name and run this now. And now we're getting the counts of those companies. So, the query's working, and it looks like we have around 215,000 unique companies. Something I want to call out with this. Going through that order of execution, we did the from and the joins. That was step one. And then next we did step three of that group by the column. And then underneath that, so step five is that select statement. And the reason why I'm calling this out is because technically, whenever you do this in most SQL dialects, you cannot put the alias, so you could not put the company name inside of here. If I was in Postgres, not going to work, at least as a filming this. Now, however, in DuckDB, they've understood that, "Hey, probably people want to do this, and make it a lot easier on their lives. You can do column aliasly aliases in here. " But, don't get too used to this because other databases don't always support this. So, in this case, right? Well, if I run it, it's still going to provide the same results. So, I'm going to change this back to what I would expect to see across all SQL dialects. Now, notice one thing, right? I said that alias of the column names, but we could use the alias of the join because is specified, or the table alias is specified in step one, and will be available to the group by. All right, next on, let's move on to this list um to get to our final answer. I'm going to add another caveat to this one of our question. We want to look in the US only. So, I'm going to limit this only to US jobs. Now, if you remember from the SQL query order that we need to do this in, we can put the where between the join and the group by. And for this

### [4:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=15600s) Segment 53 (260:00 - 265:00)

we would specify that job country, and in our case, we're going to do it equal to the United States. Okay, first, we're going to just uh go ahead and execute this to make sure this works. Okay, it is working. Now, I want to call out a few things about this. First, we have the table alias here, and we can do that because the where clause is after that from, that step one, where it executes first. Additionally, this where clause, well, right now, we are doing aggregation, but uh let's actually uh we're doing aggregation on all columns, so you may be like, "Oh, well, all columns are there, so that's why we can see job country. " Let's actually change this where job country isn't in the select statement. So, I'll change this to job ID. And now whenever I run this, okay, getting the same results it looks like. Yeah, I'm getting the same results. And the reason why I'm showing this is because if we go back to our order of execution, we haven't got to step five yet where we're actually filtering down to certain columns. So, whenever we do that where clause, it's up above at right after step one where we have all of the different columns available. So, that's why we can list a column in a where clause that isn't necessarily in our select statement down in step five. All right, moving on to the next condition and I think that's well, almost the final one. We need to get the job postings that have greater than 3,000 job postings. So, we're going to do a having clause. We'll do a count. We'll just keep it that job ID listed above. And we want it greater than 3,000. Running this bad boy. All right, we're getting the results here. Now, one thing to note about this, once again, in other SQL dialects, you can't list an alias, in this case posting counts, inside of the having clause. However, call it DuckDB makes it super friendly and you can list alias inside of here. But, if we were to look at that order of execution, that step four for the having clause is above our select clause where we actually define those aliases. So, that's why in most databases you can't use alias in having. All right, moving on. This is just for me. I want to actually order these results. So, I'm going to put an order by. And with this one though, I can put in the alias of posting count, of course, in descending order. And then running shift enter. We have them now in order that we want it. Notice that we go back to the DuckDB with that friendly database, it says, "Hey, we allows column aliases in where, group by, and having. " That's because order by already allowed aliasing. Because if we look at the order of execution, it is step six after step five where we're selecting the columns and assigning aliases. So, regardless of your database, you can put aliases in the order by. All right, and then moving into the last portion, we just want to limit this to 10 results. But, if you notice by this, right now we're only at eight. So, even when I run this to a limit 10, oops, I didn't select the whole query. That's why I'm getting that error there. So, even with selecting it to a limit 10, by the time it got to that limit 10, it was already at only eight values. So, being the last step, it becomes basically obsolete and not really even necessary. So, we just walked through how this query operates step by step using this order of execution. But, we can actually use one or even two keywords of explain and explain analyze to actually see what the steps are. So, diving into the documentation starting with that first keyword of explain, explain inspects query plans. You use the explain keyword right before a query. It can be a searchable query or it can even be DDL and DML used with. Anyway, it breaks it down that the explain statement displays the physical plan, i. e. the query plan that will get executed as an enabled by prepending the query with explain. And so, what happens whenever you execute this with something like explain is it goes through in this block-like diagram and explains what's going on. Anyway, let's demonstrate that. For this, up above it, I'm going to just put in explain. And then from there, we're going to now run this entire query. So, the query ran and the plan is here. Now, important note about reading this. You would think you read this from the top down, but that's not correct. We've been going through with this thing, explaining it from the top down of step one down to step seven. But really, what it actually looks like is this. Step one is at the bottom, and then all the way up to the top is step seven in this case. So, we need to read from bottom to top. So, let's go through this, and we're just going to look at some key terms from this that you should be able to notice. So, I'm at the very bottom, and for this, we have a sequential scan that's looking at the tables. It's looking at the company dim table, and then also the job postings fact table. Now, with this job postings fact table

### [4:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=15900s) Segment 54 (265:00 - 270:00)

remember we did a filter on job country of United States. So, during the scan, it goes ahead and filters it. Underneath here, it has an estimate of how many rows it's doing for this. It's not actually running this query in explain. That's why this is an estimate, and it has these like little curly notation in front of it, the tilde. So, all these rows up here are just estimates. Anyway, going ahead from here from the sequential scan of both these, it then performs a hash join, which we can see that in this case, it switched our left join to a right join. Like I said, you can just switch the side depending on what you're doing to switch from left to right, so not a big deal here. It even tells it what ID it hooks it on. Next from there, we get into the projection, and what it's doing is trimming down the column names that it wants to keep for the rest of the queries. This is like a DuckDB specific thing because it makes it more efficient by removing the columns. Next, it moves into our hash group by, which is effectively our group by, where we're analyzing by company name to get the different counts or the aggregate here that it specifies. Then from there, it moves into our filter of having, where we wanted to get the count or get those counts that were less than or greater than 3,000. From there, it moves into that top end or that limit statement to limit by 10. And then from there, it moves into two other operations that are not really specific to what of execution that are DuckDB specific. Anyway, that is the physical plan. That is explain. Now, with this, I want to be very clear about it. Explain here, the query or it says note that the query is not actually executed. That's why those row counts were estimates. Well, that's where explain analyze comes into play now. As from this, we can see that prepending a query with explain analyze both pretty prints the query plan and also executes it, providing runtime performance numbers for every operator. All right, similar to explain, we just put explain analyze at the front of a query. And scrolling down, we can see that it gives a similar execution plan, but now it includes some additional information, specifically time. All right, so in here, I'm going to update our query to explain analyze. And then from here, selecting it all and running it. Now, the overall plan compared to explain has remained the same of the different steps that it performed. There are a few more up at the top, specifically these extensions here. And these are just specific to, once again, DuckDB doing some stuff behind the scene that's not really important to the our order of execution. Anyway, up at the top, we can actually see now what was the total time to run this query, which was 0. 778 seconds. And in addition to that, scrolling up back to the bottom where it starts, it will actually provide per step what is the amount of time for that step with the exact number of rows for it. Now, one thing to note, cuz you may be confused, especially this one right here, it says, "Hey, it was 3. 35 seconds just for this table scan. " But then up at the top, it said, "Hey, the total time was 0. 778 seconds, less. What's going on? " Well, remember, this is These can work sequentially or it can work across multiple different threads or multiple CPUs. So, this is the total time across all the different CPUs added up. But overall in time that's relative to our time, this total time would only took 0. 77 seconds. And this time down here is just the total labor time across all the different CPUs. Anyway, explain analyze, I'm not going to go through all of this again because it's still it's the same. We can see that the row counts did update and the time did update for each. But, overall, it's now providing us I find this more beneficial, especially for doing some sort of analysis queries. Now, comparing the two, with explain, like we learned, it doesn't execute the query. So, because of this, it's only having estimates about what is the row count, and it doesn't even provide an estimate for the run time in DuckDB. Now, for both explain and explain analyze, they do show the plans. And for explain, they show estimates, and depending on your database, explain analyze will also show estimates. But, in our case, DuckDB it doesn't do that. Anyway, the last key thing that I do want you to get out of this is explain is safe to run on any type of query. If you're running a write query, something that maybe's creating or dropping a table, it won't actually do that. So, it's safe to run it. Whereas, if you use explain analyze with creating or dropping a table, it's probably going to run it. So, use this with caution. But, overall, if it's just this type of query here where we're doing some sort of analysis or aggregation, I find myself gravitating towards explain analyze because it's more accurate, and it tells us what's actually happening through executing it. Now, in our case, this query time was

### [4:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=16200s) Segment 55 (270:00 - 275:00)

what? Less than a second. So, overall, it looks like this query executed just fine. But, if you find yourself in situations where, you know, it's taking minutes or hours to execute a query, you should be using this explain analyze and analyzing to see how you could make your query more efficient. All right, now it's your turn to give it a try. I have a quiz and some practice problems to get more familiar with how the query operates and also what you could actually do in your different queries. Not going to lie, the quiz, I made this one a little bit spicy, but it does test on every single concept that we've covered in this video and also previous videos. With that, I'll see you in the next one and we're going to be diving now into building out our very first data engineering project. All right, I'll see you in the next one. — All right, welcome to our first project. And when we complete this, we'll be halfway through the course. Now, the main purpose of this project is as a data engineer, you need to start thinking of who you're building your products for. Specifically, you're more likely going to be building something like a data warehouse and it's going to be used by data analyst or business analyst. And if you don't understand how data analyst and business analyst are working with the products you build, how the heck can you even build them yourself? So, for this, we're going to go into the project goals and scope, but first, we're going to dive a little bit deeper in understanding on what exactly a data warehouse is. So, for this project, we're going to be continuing to work with that same data warehouse that we've been using in the last few lessons with our job postings fact table and our different dimensional tables that describe company and skills. Now, I've told you time and time again, this is a data warehouse, but what the heck makes this a data warehouse and not just a regular old database. Well, for that, we're going to reference this dude, which is considered the father of data warehousing. His name's Bill Inmon. He wrote the original book that defines this data warehouse in the 1990s and he defines it as follows, that it's a subject-oriented, non-volatile, integrated, time-variant collection of data in support of management decisions. So, that's four key attributes: subject-oriented, non-volatile, integrated, and time-variant. And if you're missing any four of these attributes, you're no longer working with the data warehouse, you're working with just a plain old database. So, let's define each one of these attributes. First is subject-oriented. This means it's organized around key business subjects rather than just an application. So, for us, this is around the area of job postings. In a business setting, you'd see this more around things like customers, products, or even sales. Next is non-volatile. This means the data is stable and doesn't change once it's entered into the system. Data is appended to this, not overwritten. Next is that it's integrated. It combines data from multiple sources into a consistent format. For us, we have taken these job postings from a variety of different platforms such as LinkedIn, Indeed, or Google and putting it all in here. I mean, like making it into a consistent format that we can easily query. And lastly, it's time-variant in that it stores historical data across different time periods for trend analysis. So, you typically wouldn't find different databases for different years. You should be able to go to one location to get all the recent years that are needed to do your analysis. Now, as a data engineer, it's important to understand this definition because there's a lot of different tools out there that are advertised as a data warehouse. But, if you're not implementing this approach of it, you're not actually building a data warehouse, you're building a database. So, for this exploratory data analysis project, we're going to be first extracting or working with the data inside of our data warehouse, which we're going to be using of DuckDB and Motherduck. And so with this, you're going to take the role of a job market analyst looking to analyzing the top paying roles and skills in the data science industry. You're going to be creating SQL queries, specifically three SQL queries, to explore this large data set specific to well, specific to you if you want to. And I'll get into how we can adapt these queries to your specific circumstance, but the end results are going to be very similar in what benefits they actually provide. In that we're going to be uncovering data insights around what are the top demanded skills, highest paying skills, and then my favorite, the most optimal skills. Now for this, if we recall, if we look at the distinct values of the job title short column, you have 10

### [4:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=16500s) Segment 56 (275:00 - 280:00)

different roles you can select from. I'm going to be analyzing it assuming you're going to want to be become a data engineer. That is what the analysis I'm going to perform with, but feel free to adapt it to any of these other ones here. Additionally, what I won't be doing, but you can do and further adapt this, is you could change it to what country you're doing. And there's over 161 different countries you can choose from for it. Now as I mentioned in the intro, EDA is typically a task of something that a data analyst or business analyst is going to perform. But if you think about it, data analysts and business analysts are basically your customers. And so this is going to help you understand what type of questions analysts start to think of when they have to analyze data. And also with that, it helps you understand how data should be structured and also organized, so that way those customers can do more efficient queries. When you start understanding these end goals, it's going to provide you with better understanding to build a better data architecture for your customers. So, navigate over to the URL on the screen here, and this is going to direct you to GitHub on to where my final repo or repository is for this project. Now, we have a whole lesson on understanding Git and GitHub, so you'll be understanding better of how to get things up to there and how to navigate this. But, first here's just a quick little overview of the project and how to read what's going on in here. Up here at the top is the structure of our files. So, for this, we're going to be creating three different SQL files. If I navigate into this first one by clicking on it, I can see that inside of here I have up at the top the question for that first query, the SQL query itself, and then underneath this the results along with some insights that we'll be extracting out and put in here as well. Navigating back to that top folder, it's the same format for the other two SQL squills, and then finally we have what's called a readme. md or markdown file. That's what's underneath here auto appearing. You could also technically just click on it to go into it. Now, this bad boy, we're going to go into how to actually build this in a lesson after our SQL queries, and it's going to go through and explaining what our project is. And this is really good for others or even hiring managers to come in and see your work and see what skills you've demonstrated. For example, with this readme, we're going to give a high-level overview with an executive summary telling them what queries they can check out with navigable links to get to a specific query. We'll go into the problem and context next of what we're trying to solve, what data warehouse we're using for this. We'll explain the tech stack of all the different skills you've built up and utilized during this. And this includes not only SQL, but also your other skills that you're going to learn such as VS Code and Git and GitHub. Then from there we'll get into an analysis overview of what we extracted from a high level from those three queries, and then finally putting words on the paper of what skills you actually used for this. Really helpful for hiring managers to look at this like, "Oh, okay. These are the skills I'm looking for. he has. " Or she. So over the next three videos, we're going to be diving into each of these questions for it. Starting with the next one, investigating what is the most in demand skills. And remember for all of these, I'll be doing it from the perspective of a data engineer that wants to look for remote jobs. You can do the same or modify it as I showed earlier. All right, there's no practice problems for this lesson. I would spend some time going through that repo that I gave you and understanding what generally we're going to be doing with this project. With that, I'll see you in the first query in the next video. All right, let's jump straight into this. I'm starting here in VS Code. I'm doing it in Mac, but Windows is going to be the same regardless. And remember with this we're trying to solve that first query of what is the most in demand skills. We'll get a further problem statement for you, but first let's set up the structure that we need for our project. Right now inside of our SQL Data Engineering project folder, we have a lessons folder. And I want to create another folder as well. So make sure when we go to create a new folder, which we select right here, that it actually inserts it here and not if you have lessons selected and you were to insert a folder, it inserts it underneath there. Totally annoying. Anyway, so with our new folder, we're going to call this, this is our first project, 1_EDA. I like putting a one at the front because it helps organize this. Mainly numbers go first and then followed by letters. All right, so with our 1_EDA or first project for EDA. We're going to stick our first file in here. And I like to do a similar numbering approach for our query, so I'm just going to put 01

### [4:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=16800s) Segment 57 (280:00 - 285:00)

cuz technically we probably maybe could have more than uh 10 and so I want double digits. And for this we want our top demanded skills. And this of course is a SQL file. Okay, this bad boy's open. I'm going to go ahead and close the explorer. So let's get into our problem statement. And for that it's the question of what are the most in-demand skills for data engineers? In this we're going to identify the top 10 in-demand skills for that job role. And I'm going to be focusing on remote job positions. Like I said, you could do something like focus it towards your specific country. And then underneath it I have the importance of that it provides insights into the most valuable skills for data engineers seeking remote work. So anytime I'm working with a problem statement, I like to include it up at the top. I'm going to use a multi-line comment to insert it in. So I'm going to put this in here on what we're trying to focus on and what we should have the outcome to be. So let's start building our query up, but in order to do that we need to connect to our data warehouse in MotherDuck. So I'm running DuckDB command to connect to that data jobs database. So for this we want the top 10 most demanded skills by count. And that's in our the skills name is in our skills dim, but we want a count of our fact table which has the actual postings in it. Now because it's in two separate tables, we're obviously going to have to do a join, but we need to think about what type of join we're going to be doing for this. I like to keep job postings fact as the A table and skills dim as the B table. If we did a left join, yes that's good. We can preserve all the facts from the A table, but remember there's some jobs in the A table that don't have an associate skill, so we also have that in there. And I don't really want to count uh values of no skills required sort of not necessary. Now we could technically do a right join to connect to those job posting fact table, but right join isn't widely used and if you're using a right join, you should probably use a left join, but I have a better solution for this. In that we're going to use an inner join in that from table A on the job postings fact, if there's jobs that don't have an associated skill, we're just going to weed them out. And conversely for table B of skills, if there's skills that don't have an associated job, but spoiler alert, all the skills do have an associated job, they would then be filtered out. So inner join is our best bet in this. So let's start building this out. We're going to first just do a select and then I'm just going to throw a star in here cuz I don't know which columns just yet we want to get in order the name of the aliases of the tables. We're going to be going from our job postings fact table and we're going to be forming an inner join. Specifically, we need to get to do two, right? We have to connect to that bridge table of skills job dim and then to our final table of skills dim. So we'll specify that skills job dim with the alias of SJD and we'll do this on the job postings fact table, which we haven't given an alias for on job ID. And that's to the skills job dim job ID. Now I need to assign the alias for job postings fact up here. Okay, we got one table joined. The next table that we're going to get is our skills dim and we're defining this as just SD and we're doing this on our skills job dim table skill ID with our skills dim skills ID. Looks like I got a typo here. Fix that. All right, I do want to test this query, but I don't want to run this query on a bunch of different rows. So I'm going to go ahead and put a limit 10 on here. This limit 10 also is going to be helping us in understanding the top 10 in demand skills. So we're sort of getting two birds with one stone. So I'm going to go ahead and select this, run shift enter. All and we're getting results back. Not too bad. All right, let's fine-tune this further, right? We want to get the most in-demand skills for data engineers. Let's focus on that first thing first of most in-demand skills. So, we need to in order to get that we need to do an aggregation of the count of jobs based on a skill. So, we'll define in our select statement that we want from the skills dim table we want to get that skills. And then next we want to do an aggregation. Now, for this you could do either the job postings fact table or you could do the skills job dim. We're going to do job postings fact table. And I'm going to just do all columns in that. We're going to give this the alias of demand count. Now, anytime we do some sort of aggregation, right? We need to do a group by. In this case, we're going to be doing the group by on the skills column. Okay, let's go ahead and run this bad boy to see how it's doing. Looking good, although it's not in order, right? I got our skills and our demand count. I would want this in order so I can see those top 10. So, I'm going to put an order by in here, and I want to order it by we can use the alias here cuz we learned in a previous lesson because the alias is now available for

### [4:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=17100s) Segment 58 (285:00 - 290:00)

order bys. And we want this in descending order. Running this, looking good. All right, now this is for all jobs and regardless of where you're working right now, we're seeing Python, SQL, AWS and Azure up at the top. Let's actually get into filtering this, and remember where statements or where clauses are going right underneath your joins. Specifically, we want that job title short equal to data engineer. Along with in my case, I want to look for remote work. So, we're going to do once again the job postings fact table, that job work from home column. Specifically, we want specify that to true. All right, let's go ahead and run this, see what we got. All right, not too bad. SQL and Python are still topping the list along with AWS, but now Azure, Spark, and Airflow are up there all in the top six. Just going back to that problem statement and checking it over to make sure I get all the different attributes. Yep, got the 10 top 10 in-demand skills focused on data engineers and remote job postings. Now, what I'd like to do after verifying that I do have the correct query, I just go ahead and copy this right here and do control C on Windows or command C on web Mac, and then down here underneath it so that way somebody reading this SQL script afterwards, they can see what the results are also. I'm going to just go ahead and paste this in here with command V on Mac or control V on Windows. Now, take some time to look at this and see if any trends that you're noticing with this, especially if you did something, you know, different for me, not data engineer or in a different country. For me, I just went through and did a breakdown of the most demanded skills, specifically the top five, and then did any key takeaways out of this top 10 as a whole list. You don't need to not necessarily be this verbose in how you go through and define it, but this will pay off on what you're seeing here for when we go forward with building out our final read me. Because in that read me, not only are we going over how the project is defined and what we were trying to do with it, but we have a section at the bottom talking about our analysis overview and what we found out overall. All right, so not so bad. That was our first query of three knocked out, and the next one we're going to be getting into analyzing those top paying skills. Take some time now to do look into those skills and what results you got, and start putting together what are some key insights, and you're going to see some trends as we go through all this. All right, with that, I'll see you in the next one. Now that we found out what are the most in-demand skills, we now need to take this a step further and find out what are the highest paying skills for data engineers. Now, to do this, we're going to be focusing on, as we learned about previously whenever aggregation functions, we're going to go ahead and using this using the median function. This is going to be the most accurate representation because we can be skewed by those high salaries. We're going to continue for, me at least, going to be focused on remote positions and of data engineer positions. With this as a bonus, we'll just also include the frequency to not only just see like, okay, if it has a high salary, is there actually a demand with how many counts of a job it has. So, inside VS Code in our 1_EDA folder, I'm going to go ahead and create a new file. Looks like it's created in the wrong place. Okay, I'll start for now. We'll specify the 02 top paying skills and this is a SQL file. I'm going to go ahead and just grab this and then move it into the correct folder. It asks, "Do I want to move it? " Don't show this again. Yeah, move it. Now, in here at the top, to do the same thing and that I want to define in a multi-line comment what we're trying to solve. So, I'll put it in here, "What are the highest paying skills? " along with those three different focus areas along with that why. All right, now, I don't like starting from scratch because this query that we're going to be building is going to be very similar to our last query, which is conveniently right next to us in the terminal. Anyway, I'm going to go ahead and go back into that file, copy it, and then paste it into here. Now, this previous query already aggregates by our skills. And in this case, it does a count, but conveniently, I've written this question in a way that we include the skill frequency to identify both salary and demand. So, we're going to end up actually keeping this. We need to just add in a column to go in and find the median salary. Just as a quick refresher for the rest of this, we're going to be using that same from statement of the job posting facts, doing two inner joins to connect our job postings fact to our skills. We're filtering it currently for data engineers and job for work from home true. We may have to add another one with this and we're grouping by only by skills. And right now we're ordering by demand count. May have to change it for that. And we're limiting it to 10. For this one, I'm going to change it up a little bit. We don't have anything listed. So I'm going to just say let's go with 25. Okay, so let's add in first that median salary calculation. We're going to be using that median function and then J from our job posting facts

### [4:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=17400s) Segment 59 (290:00 - 295:00)

table, we want to use that salary year average column. For this, we're going to specify this as the median salary. Let's get the terminal back up. Let's try to run this bad boy and we get an error parser error. We know what parser is now. It's at or near count and I can see following next to this. Oh, we're missing a comma at the end of this. Run this again. All right, not bad. Noticing with this we do have or the like the skills are organized by domain count so the highest salary isn't at top. But looking at these values, they're looking pretty good. I'm not liking the formatting right for AWS and Scala. So let's actually fix this so they're all one consistent value. Well, anytime you need something like this, highly encourage you to dive into the documentation cuz not only will you find what you want but also something even better. Anyway, we know already that we do have some numeric functions available. We went over operators very early on and they fall under in this of numeric functions. And then scrolling on down, we also have these that allow you to do other type of math operations. Now with this, you could use something like add for adding X and Y values. It's an alias for basically the plus operator. But they have other things like trigonometry functions and then this one here's what we're going to be using of round and it rounds to S decimal places. So with that round we need to put this around in our entire other function of median. So I'll do round and this does take an argument. So I'll do a comma and then I'm going to specify zero for the amount of significant digits for this. And just double check the documentation. Okay, yes, it takes the numeric that's the Y value is what we want to give it a column and then S the integer is the round to S decimal places. Can't And apparently negative values aren't allowed. So that's why we're doing zero. All right, let's go ahead and run this bad boy. And not bad. No, did you notice there is I did round it to zero and it still does have one significant digit after the decimal place and that's because it's of the type double. So it's not an integer but it is a decimal and so it's always going to have this unless we change it to another type of data type which we haven't covered yet but that's coming up in part two of this course. Anyway, just to show that I'm not crazy and that's what I actually mean. If you put in one in here and then run this same query shift enter, you notice now it's not all zeros anymore. We actually have you can see this has three and this has five. I think that's just too many. I don't really care too much about that. So I'm going to change this back to a zero and then run this. All right, so we're almost there but we haven't actually got to kill that first one of what are the highest paying skills for each for data engineers. Everything else though we have hit on for what we need to do. So in order to do this we need to change what we're actually ordering by and that's going to be by median salary. Pulling back up the terminal by pressing control tilde and then running this again pressing shift enter. Bam. This is looking good. It looks like Rust is up there at $210,000. It's been blowing up uh more recently, especially on social media. So, I can definitely appreciate that one. However, as I'm going down the list, right? I get down to like this one right here, and it only has a demand of one. And then down here even further, we have these that are like 05 and 9. So, like not very popular skills at all. I think this list I would be more comfortable removing values, just me looking at this and interpreting it, seeing values that had greater than 100 as a demand. So, we need to do an aggregate filter. In order to do that, we have to do a having clause. And for this, we'll say that the count needs to be greater than 100. Yes, in DuckDB, you can list the alias there, but right we're writing this generic for any database to use to demonstrate your skills. So, we're going to leave it like this. All right, let's go ahead and run this. All right, I'm liking this list. Um it has skills that have a lot more demand to it. So, I think that the salaries mean a lot more with this. Anyway, we can take this a step further if you wanted to. Basically, I'm going to just copy this here, doing uh command C or control C from Windows. I'm going to go something like ChatGPT, and I'm going to say, "Hey, plot the top 10 skills by salary. " And then I'm going to just paste in that table here. Looks a mess in here, but ChatGPT will figure it out. So, this graphed it for us, and we can see that the highest, well, the top 10 at least, are ranging from around 160,000 up to over 200,000. And with this, it

### [4:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=17700s) Segment 60 (295:00 - 300:00)

also gave us some takeaways with this. Specifically, a lot of these skills are niche and also hard to learn. But because of that, there's more money there. But interesting enough, our two most popular skills on that top 25 list of Airflow and Kubernetes, very valuable skills for data engineers to know, have massive demand but don't even crack the top 10. So now back in VS Code, I'm going to go in and put together some insights that I've extracted out of this and I suggest for you to do the same. And not only am I going to put my insights in, but also I'm going to put those values in here as well. So I'll copy this and I'll put this down at the bottom. All right, so overall this is looking good. Got my question up at the top, my query now built, and also that result set and how my analysis goes. I'll go ahead and save this by pressing command S or control S if you're on Windows. All right, only one more query to go and this one's going to get even more complex. With that, I'll see you in the next one. Now that we've identified the most highest paying skills, most highest paying, I don't know if that's right. Anyway, we're now going to need to get into our third uh query, which is going to be what are the most optimal skills for data engineers, balancing both demand and salary. Now previously in our results, we've gotten both the skill with its associated median salary and then also its demand count. In that query, we ordered it by the median salary, but what we really we need to figure out a way to order this that takes into account both the median salary and the demand count. So we're going to develop sort of like a ranking column. Not really sure how we're going to do this just yet, so hopefully we can figure out something as we go along. And for this, we're going to be focusing on remote data engineer positions like mine, like I want to do. Feel free to adapt it to your situation. All right, so inside VS Code, we need to in our 1 _ EDA folder, we need to create a new file. We'll call this 03 optimal skills and make sure that it's a SQL file. Up at the top, I want to carry over this question that we're going to be answering with this. So, I'm putting that question along with those other two attributes that we want to meet with this query. Now, this one is going to be very similar to our last query, right? Because we're using those skills and the median salary. So, I'm just going to go in and select that last query, and we're going to use that as a starting point for this query. So, jumping into this, we need to create a ranking column that combines demand count and median salary to identify the most valuable skills. Now, one disclaimer real quick about how we've been calculating this demand count. Well, let's go ahead and run it in our terminal just to show. And I'm going to run this query. Okay, so we've been doing that demand count, and this has been all of the rows because we're doing star. Now, if you recall, we actually were analyzing the median salary based on a skill, but not all skills necessarily like not every job posting has a median salary with it. And we can show this by I'm going to insert another column right here. We're going to be using the count function from the job postings fact table. We only want to count the salary or average column, which remember a count on that column is going to count all non-null values. And I'll call this as corrected demand count. So, selecting this all and then running it. This came out a hot mess. Let's Okay, not too bad. So, what we can see is, right, that these values go down, and actually, I mean, Rust and Go Lang demonstrate this pretty well, right? We went from 232 to 912, which is a, you know, a pretty significant jump proportionally wise. I mean, it's almost five times bigger. But, when we sort it for those with actually that have a job postings that have a salary value listed, we're going from 23 to 39, not even double what it is. So, I don't whatever multiplier we use for this, I don't want to inflate some of these skills that may just have a bunch of postings, but they're not backing it up with a actual an actual salary. So, what we're going to do is we're going to stick now with this corrected demand count and we're just going to replace that demand count up here with that. And I'm just going to rename this to demand count. And then just double-checking to make sure that this query can work properly. All right, it's still working. And it's showing all of our correct values. All right, so now we need to create a ranking column. And something that we could do is we could create a multiplier column. And what I mean by this multiplier column is

### [5:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=18000s) Segment 61 (300:00 - 305:00)

basically we're going to multiply median salary times the demand count. And hopefully, maybe we'll see how this actually goes. We'll rank it according to that column and we'll see what kind of what we're getting out of this. This is just an experimentation. All right, so going to put a comma, new line, and we're going to take that median value and multiply it times the demand count. And we'll give this the alias of optimal score. All right. And for order by, we'll change this down here. We can list the alias in the order by. So, we'll list optimal score in descending order. Okay, let's just check this out, see what we get. Not guaranteeing it's our solution. All right, cool. What we're getting is well, optimal score over here, it's a exorbitantly big number. And do you notice any trends with this? Well, the key thing I'm noticing is that demand count is overwhelmingly driving this ranking right now. Because if I would even look down this list, it looks like, except for a few that are out of order, that the demand count is really just It's really what we're doing is we're ranking by demand. And that's not really what I want to do. I want to get something that's like a bridge between demand and median and helps clean this up a little bit. One quick note, not related to demand count, we could also list using Duck as you're going to find out. I could also list the aliases in here in the select statement. I'm going to pull up the terminal, running shift enter. It still works the same, but this is only allowed in certain databases. Duck DB, of course, loves being friendly, but like MySQL and SQLite also allows it, but most other ones don't allow it. Because of that, we're not going to use that in here. Get that out of here. All right, so rerunning this query, let's get back to on hand. This demand count. Now, if we actually plot this, it has a really interesting characteristic. So, I had ChatGPT plot this, and if you look at it, we can see that, especially when we start getting up to the higher-up skills, it's not necessarily like a vertical a smooth line going up to SQL Python. Instead, it's almost like these skills go exponentially. So, here's what I did. I had ChatGPT plot those top 25 skills by demand count from lowest to high, and it basically does show this, right? It's a almost an exponential growth with this getting to that Python and SQL. Well, there's a function we could use to help smooth this line out, and that function is natural log. So, on the x-axis at the bottom is basically like our raw count going from zero to a thousand. Now it does have this sharp up swing, but then after we get to around 100 to 200, it then studies off and goes at a more steady value. Natural log is great at compressing large values, so extreme numbers like that of Python and SQL don't dominate. Now it's important to note that this that the scale and it's going to change when we apply this, but the order is not going to. So I think this using natural log is going to be a better option for us. And conveniently, if I go to DuckDB to look at the numeric functions available, we can see we have this one that computes the natural logarithm of X by just using ln. So let's just first convert that demand count into the natural log of it. So I'll define the natural log function and then inside of it put the count of those non-null values of the salary basically job postings with salary listed for the associated skill. And we'll call this ln demand count. Now something's going to happen when we go to run this. Little foreshadowing here, whenever I run this, I'm going to get this error, out of range error, cannot take the logarithm of zero. And the problem is this, I'll actually show it in a query underneath here. I'm just going to run a simple select statement and in it I'm going to do a natural log of zero. Whenever I try to do this, I get this error once again, out of range, cannot take logarithm of zero. Basically there's a zero value within our count at one point for one of these skills and so it's causing it to do that. You have to have values of one or greater whenever you put them into the natural log. Here I'm running it with that and whenever I even use the value of one in the natural log, it gives the value of zero. And after that, it only increases it for there. You're doing it for the natural log of 100, that's 4. 6. All right. Anyway, enough of that history lesson. So, the problem is right now is that some of these counts that we're doing an order by for the

### [5:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=18300s) Segment 62 (305:00 - 310:00)

skills, some of these skills have a count of zero. Well, we need to just remove those skills altogether before we even stick it into the natural log. And so, because that, we know because of order of execution, specifically step two is our where condition to filter, and then three is our group by that performs that aggregation. So, we know if we filter those skills that don't have any associated salary associated with it, we will therefore get it out, and then by the time it gets to step five for that performing the natural log of it in the select statement, that's not going to be an issue anymore. You're probably like, "Luke, you're crazy. I don't know what you're saying. " Well, let's actually demo it, and I think it will make sense. So, inside of here, instead of doing salary or average, for this one, I'm just going to do star, and that's going to be selecting all the columns of the job postings fact. But, I'm going to filter it here for the salary average column of the job postings fact table to remove all null values. Now, whenever I run this, we actually get values back because inside of this natural log, there's no skills with a without a associated salary, so there's no skills with count of less than zero, or no skills with a count of zero. Anyway, I like to maintain the same format throughout. So, anytime we're doing this in the count, I'm going to replace it here, and then also here in our optimal score. So, just moving out of the side so that you can see it a little bit easier. I updated it in three different places. But now, now that we have the natural log demand count, I need to take this portion and we're going to replace this in our optimal score. All right, now I'm going ahead and opening terminal and then shift entering this into here. All the values aren't showing, so apparently I got to move this over more. Running this again. All right, we're getting everything. I do want to do some data clean up before we get into this, actually evaluating what's up at the top. First up is there's just too many decimals and too much going on here uh for this. First of all, that uh natural log demand count, we only really need one decimal place. Also, we no longer need just this demand count anymore. So, I'm going to go ahead and just comment out demand count. If you want to leave demand count in, you can. And then for the natural log demand count, I'm going to do a round function as we've done previously, and I just want to have one decimal place for this one. As always, I like to just make sure that it's operating correctly, so I'm going to shift enter. And I did route. Whoopsie, selecting again. All right, so this is looking better. Now into the optimal score. Right now, these values are Oh my goodness, it's up into the 100,000, almost close to a million. These numbers are really arbitrary, so I'm just going to convert it to a decimal and make it as small as possible. So, this is just me. You could do whatever you want, but I'm going to wrap this entire thing in parentheses. And it was getting up to a million, so I'm going to divide by a million. Remember, we can use underscores between the zeros and ones to help show where the separators are. When it actually goes to evaluate this, query's not going to look at it at all. Now, doing this just to show it. I'm going to shift enter. All right, looking good. They're all basically decimal places. Because of that, I'm going to just do a round on this, and then comma, and then we'll do two decimal places. Now, running this, I did round again. — Try this again. Shift enter. All right, this is looking good, and I can actually bring this over closer to here. So, now, actually getting into evaluating what is that final output, we see that Python and SQL aren't on top the list. Instead, we have Terraform, which is a popular infrastructure as code tool, appearing higher on the list. And so, I mean, look at that salary for that. It's at 184,000 compared to what Python and SQL is, 135,000. So, in my mind, that makes sense of an optimal skill to learn. Other ones that are appearing on this list as well, just looking down this natural log domain count, uh with this 5. 7 is Kafka, which would have been lower down, and also Airflow. I'm realizing now taking off that count of just the regular count is sort of confusing. So, I'm going to bring it back for a second to show what I mean. Anyway, Terraform, right, was only, I mean, two, almost 200 jobs had it compared to almost 1,100 for Python and SQL, and still got up to the top. And then, Airflow was only around 400, and Kafka is around 300. So, pretty, not a bad multiplier that we just came up with

### [5:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=18600s) Segment 63 (310:00 - 315:00)

for an optimal score in order to uncover some skills that we may not have noticed if we hadn't have done this type of manipulation. I'm actually going to leave this for the final results with that domain count in it, cuz I think that'll be good. So, now, as always, I'm going to go ahead and copy this down, and put this into a multi-line comment, and then also take some time to go and actually break down how what this means for these scores, and maybe going by those top 10 skills, that's what I did here, and get some insights from this. We're going to be using this to build our read me out for our final project, for the cover page for this. All right, so we just wrapped up our three main queries for our data project. Our next step is to get into building out that read me. So, we have so that cohesive story to share about this project that we just built. All right, with that, I'll see you in the next one. All right, for this build out of our read me, I'm going to assume you have no prior experience with building this out. So, the first half is going to be with the importance of a read me, and also some basics of building it out. And then from there, in the second half, we'll get into actually building out for our project. So, going into GitHub to just go briefly over what a read me is again. What's cool about it here on GitHub is that if you have a read me, and you navigate into a folder or whatever, the read me will automatically display below here. All right, in our case, a read me is a great way of showing an overview of what our project's about, and what it does, and why the heck it even exists. I made this for the perspective of a hiring manager coming in to maybe verify your skills. And so, that's why I have it up at the top, executive summary for hiring managers, and then have some key areas in there for where they may be looking at. Additionally, I go into our problem, adding context about the data warehouse that we are working with. And then more into what hiring managers care about, the tech stack we're working with, where they can find all our key queries in the repo, and an analysis overview, and then there are skills demonstrated. Read me's are everywhere in the projects. If you remember, at least for those Mac users, we installed Duck Man, which was a Duck DB version manager inside of Mac and also Windows. Anyway, scrolling on down inside of this project, we have the readme at the bottom and it has a brief overview of what's going on here, how to install it, its different usages, and then references. And I have a readme for everything such as my Python for data analytics course. It goes through and explains what this repo is about for the course and how you can actually navigate it. So, readmes are super helpful and it's like a front page of your project or whatever you're working on. Now, let's get into some basics of markdown and what do I mean by that? Well, whenever you're actually building out these files, as you can see in here, I have some different images. I also have things like links. A link me to the queries. Anyway, this is actually an HTML rendering of what is in the markdown file, the. md file. So, if I did a side-by-side, this is what it actually looks like. This is the actual text that I can actually edit inside of here. For example, if I wanted to change the title to just EDA, I could do that and it's going to change in that HTML rendering of what the preview is of the readme whenever we upload it to something like GitHub. So, we have all these special characters and basically it's like a coding markup language that we can use to format this readme. So, let's now go through in our project create this readme and start getting familiar with some different syntax we can use while building out this readme to try some different things in it. All right, so inside [snorts] of our one underscore EDA folder, I'm going to create a file and in this it's going to be a readme in all caps and then a. md. And then MD stands for markdown file. So, you can have markdown files that aren't necessarily readmes. Like I could rename this to something like help. md and that's going to work just fine, too. But, for the purpose of this, we're going to name it read me because also GitHub picks that up. Now, I'm going to close this off to the side and I'm going to open over on the right-hand side, you see this open preview to the side and we're going to try different things. The first syntax is this, using a hashtag you can write a heading. In this case, one hashtag is heading one. Note, you have to have a space in between there it's not going to work. I also have things like heading two and heading three. If I just write in here, this is normal text. However, if I want to bold it, I need to put two asterisks around

### [5:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=18900s) Segment 64 (315:00 - 320:00)

it and then two more to close it off. And it's now bold. Notice it's on the same line and this is because this is a weird little thing with uh markdown that you have to do. At the end of a line, you have two space over two times for it to go down and pop on the next line. So, now it's bold, you can do italics with only one asterisk. Once again, I can't forget the two spaces, so it keeps on doing that period. Why does it keep doing this? Come on, play. Stop. And then also, if we wanted to maybe put in code, we would use a tilde and then write this is code. Now, if we want to include things in bullet points, we could just do a tack and say this is bullet one and then this is bullet two. Or we could do numbering of number one and number two. If I wanted to include a link to something, I would do square brackets for the link text and then in parentheses, I would put what I wanted to go to. Um so, I'd do https and then something like google. com. Now, there's a link going on right there. When I click it, I get directed to Google. It's Christmas time, so nice little logo. Now, also besides links, we can do images as well and that is a similar format as links except we're going to do an exclamation point. In square brackets is the alt text, and then in parentheses is the location of the image. Now, this image can be located online, which let's do that first. Specifically, if we navigate over to our project, I have this up here, this diagram of our what's going on here. This we can actually use. If I right click this image, I can copy image address. And then inside of VS Code, I'm going to go ahead and just paste this with command V, and I can see that it lists the GitHub address that it's located at. And this image now is inside of here. So, you're free to use this image. However, you don't know if I'm going to go crazy and move it this image and replace it, and so you need to be able to I would have this image and just download it instead. So, I'm just going to right click this and go to save image as, and then from there I'm going to navigate into my data engineering project, and I'm actually going to create a new folder inside of here, and I'm going to call this images. So, we'll go ahead and create it. So, images is inside of here, and then we'll just keep this name right now of project one EDA. Save. And now, navigating into here, I open up the explorer, we have this folder now of images of this right here of the actual image. What I'm going to do is I'm going to right click it, and I have two options here. I'm going to copy the path or copy the relative path. Let's go with copy the relative path first. Inside of here, I'm actually going to remove this whole hyperlink thing, and I'm going to write we can paste and link to a file in here. And the relative path is just the images. However, if I go in here and I right click this and select copy path, and say to paste that in instead, that's going to go all the way up into that main directory. That's not necessarily what we want. So, that's why we went with that copy relative path. Anyway, as we can see over to the left-hand side, this isn't working and that's because Sorry, going back into this folder right here, right? This readme is located in the one {underscore} EDA folder. Where right now the images the image is located in the images folder. So, it's not seeing the readme's not seeing this because it's trying to navigate from here into the images. Also, I'm noticing right now that I have the wrong path inside of there. So, I'm going to copy that relative path again and paste it into there. So, it's not locating the right location, right? Cuz it's trying to go from here into images. What it needs to do is needs to go back a directory and then in the images. We can do this similar to going like a CD {dot} {dot}. We're going to do a {dot} and then a slash. Whenever I do that and press the {dot} slash, it goes back up and then VS Code does this cool thing where it then tells you, "Oh, you can actually navigate into these three different folders now that you have access it. " Anyway, we already have the rest of the path and the correct location of the image. I would then also do something like change this alt text to something like project one overview. One last thing that I find myself using from time to time is um code blocks. So, previously we talked about we can do inline code like this, but also I like to have physical or big old code blocks. So, if in the case if I wanted to put in this query into a readme, I could copy it and then inside of our readme, let me close this out of the way. If I were to only do this one dash and then paste it in and then another dash, it makes it look well, okay, this is hideous. Instead, what you do, you do three tildes and then I'm going

### [5:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=19200s) Segment 65 (320:00 - 325:00)

to move this down the next line. And to close this out, I'm going to do three tildes to close this out. Now, right now there's no syntax highlighting on this indicating what the different keywords and everything are. So, I can actually specify what type of code this is right next to these three tildas by saying SQL, and bam, now it colors it. Now, there's a bunch other stuff you can do with markdown as well. So, I highly encourage you to check out this markdown cheat sheet that has a lot of things that we've just covered along with some other features as well. So, you can get some ideas from it. So, let's get into building this read me out. And for any of these different sections, you can feel free if you don't feel it's applicable, you can skip it, and you do whatever you want. I mean, it's your read me. This is what I would recommend, especially for somebody that's a job seeker in data engineering. The one thing we are going to keep on this is this image that we previously did. So, I'm going to delete all the way up to heading one, and then also delete this SQL code underneath here. For heading one, I'm going to give it the title of exploratory data analysis with SQL, and this is a job market analysis. Then we're going to have the image there, but we need to now go and give a brief intro. So, I say this is a SQL project analyzing the data engineering job market using real-world job postings. It demonstrates my ability to write production-quality analytical queries, design efficient queries, and turn business questions into data-driven insights. Now, next let's get into the major sections. We have our executive summary, our problem in context, our tech stack, an analysis overview, and then finally, SQL skills demonstrated. So, now we're going to go through and just build these out. I'm not going to tell you word for word and read every little thing. I really encourage you to go through that and do that. But, these are the main sections I do highly recommend that you do include. So, for this, I go into four main areas. First is the scope, analyzing or specifying that we only did three queries. Next is the data modeling involved and specifically how I had to join tables. Then from there the analytics, what aggregations, filterings, and sort we had to do. And then finally the outcomes, what I'm going to give at the end of this of what we found out from the analysis. Now, I also like including right in this section links directly to the queries if they just want to dive right in. Just make sure that you can write SQL. So, if we wanted to link to something like the top demanded skill, well, we're located at the readme right now. So, we're we just need to select that right there. So, it's pretty simple, right? You put in square brackets the text that you want to see, such as top demanded skills query, and then in parentheses right next to it, I'm going to put the file itself. So, just typing in or you can copy and paste this path or whatever you want to do. Now, it has that link in there and so whenever I go to click on that, it navigates me to the file. Anyway, so I went through and listed these out. I ended up changing to the SQL file itself so they know what they're clicking on and it gives the SQL file along with a quick little snapshot of what we did in that query. All right, next one to focus on is problems in context. In this I specify that, "Hey, job market analyst, the thing that we did the job of, you have to answer questions like most in demand skills, highest paid skills, and best trade-off skills. " And then I wanted to go into demonstrate that we have a familiarity with data warehouses. I wanted to them then provide them with a diagram of the data warehouse. Now, you don't have an image of it, but feel free to steal mine from my GitHub repo. And so just scrolling on down in my repo to that data warehouse image, I'm going to go to save image as and just save it right in that images folder once again. And then to include the image, remember we need to use an exclamation point, square brackets for the alt text, give it the name of data warehouse, and then we need to go to the image. Remember, we're in this readme right here. We need to go up a folder into back in that SQL data analytic SQL data engineering projects folder. So, inside of parentheses I'm going to do a dot and then a slash. Now I have the images folder. I'll use the down arrow to select images and then 1 2 data warehouse PNG. And voila. Underneath this image I then go into a little bit more detail explaining what the fact table is, the dimension tables, and then the bridge tables. I used whenever calling out these tables, I want to make it into a code block so I do put inline or sorry, tilde characters around each one of the tables to make me make it really stand out. All right, next up is the tech stack that we use and we really need to call out all the different skills we've used so far and that we're going to use. So, I call out things like our query engine of DuckDB, the language of SQL, our data model, how we actually had to go into a data

### [5:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=19500s) Segment 66 (325:00 - 330:00)

warehouse, and then even the things that we're using now, right? So, things for development. So, how we're using VS Code for SQL editing, how we use terminal to work with all of this, and then how we're what we're going to do actually in the next lesson on version control using Git and GitHub. All right, next up is the analysis overview. And this is really going to pay off for all that homework you already did looking at all those different queries and what we got from it. With this I first started with the query structure. If they didn't notice it up above in that hiring manager portion, I re-put all the queries into here and link directly to it. And then what we did in each one of those three queries. Anytime you build out any of this stuff, it's good practice to make sure that the link actually works and navigates you to that file. And then underneath this I did key insights and this is really extracting out all the things that we found out previously. Now, this not only includes those insights about how high of a demand SQL and Python are, but also whenever we got into our optimal skills, finding out about Terraform, about how it was a optimal skill. All right, last section on SQL skills demonstrated. This project is primarily focused on SQL, so that's where I wanted to call out specific stuff that we did. I broke it up into two sections. The first one, which is a feel bit more applicable to data engineers, so on query design and optimization. We go over not only complex joins, aggregations, filtering, but also sorting and limiting. And then, after that, we go into data analysis techniques, talking about grouping, control logic, the different functions we've used, calculated metrics, the having clause, and how we handled null values. So, all very important attributes. All right, now just scrolling this over, making sure everything appears like it should. It's looking good. I'm liking this. All right, so if you're liking it so far, make sure that you command S or control S it if you're on Windows to save it. All right, now that we have this readme built, and we have our three SQL queries done, we're pretty much done with this project. We're going to take a light segway from this project and get into learning about Git and GitHub, because we need to get build up this requisite knowledge on how this tool actually works, or tools work, before we even upload our project into there to share this project with the world. All right, with that, I'll see you in the next one. All right, so we have our project all built, and now we need to actually get this project onto GitHub to share it with others. So, this lesson here is going to be going over two important contents concepts of Git and GitHub. Now, Git is the version control system that we're going to be interacting with through the terminal and GitHub is where we're just going to share in our project. The main focus of this lesson will be on Git because there's a lot of different terminal commands that you need to know about in order to manage a project properly within this version control system. By the end of this lesson, we'll have pushed our project up into GitHub and it will be available to share. Now, let's get into this and we're going to be breaking Git and both GitHub down first to start so you have a background on this. To be clear, Git can be installed locally on your computer and also Git is what is behind the scenes working to manage and track all the changes within GitHub. So, Git is a free and open-source distributed version control system designed to handle a lot of your different coding tasks. not only allows you to keep track of snapshots of your project as you build it out, but this also enables you to collaborate with others because of this version controlling. But, how the heck does this actually all work? Well, here I am on my local computer and this is the folder that I'm building out the this engineering course in. Specifically, I have it set up using Git. And you can't see this, right? Cuz all you can see is the course and then these three different files, but there's actually other files. Now, you can list all hidden files on a Mac by doing ls -a, but I'm going to just show you with this shortcut of command shift period. And what this does is that shows the hidden files inside of my folder right now. Specifically, there's a. git folder in there. That. git folder is keeping track of all the different changes of this project within in Now, we're not going to mess with the contents of this. I just want you to understand that this is how it's actually being tracked in here. So, even if you're working offline on a project, it still can be tracked. Now, there are other options for version control systems, but they're nowhere near as popular. The last time that Stack

### [5:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=19800s) Segment 67 (330:00 - 335:00)

Overflow did a survey on version control systems back in 2022, Git won by far with 94% of the votes. It's so popular, in fact, that they don't even do this poll anymore, and 2022 is the last time they actually pulled on version control systems. Now that we understand Git, let's dive into GitHub. GitHub, I like to think of it as like the Facebook for coders. Anyway, it's a hosting platform for Git repositories, where you can then you can share with others, but also work with others. As I mentioned and showed you locally machine of SQL Data Engineering course, I also have it here inside of GitHub. And because it's up here, it then allows me to work with others, specifically like Kelly Adams has helped me out with building this course. She's contributed to it. We can collaborate on it together because it's here in GitHub, and she can access it from where she's at. And like Git, this is completely free. Now, going back to that Stack Overflow survey, but from last year, or actually this year of 2025, for code documentation and collaboration tools, GitHub is by far also number one at 80% of the choices. Interestingly enough, something that we just learned in the last one, Markdown file, is at number four. But, back to GitHub, the closest competitor to GitHub on this list would be something like GitLab. So, now that we understand the basics of Git and GitHub, what are they actually managing? Well, for both, they're managing a repository or a repo. It's like a personal library for your project, where you can keep, manage, and record every change to your documents and files. On GitHub, this is my remote repository. And it's called remote because you can access it remotely like other people like Kelly can access it. And then locally, managed with that dot git file, we have our local repository on my computer. Now, inside your computer, it does get a little bit more complex because you can work inside of here. So, besides a local repository, you have two other areas a working directory and your staging area. The working directory is where you actually edit your code. The staging area is just a temporary area where you prepare your set of changes that you're going to then commit to your local repository. Don't worry too much about those. We're going to be diving into each one of these and you'll get very familiar of what are those different stages of keeping your files. So, now we're going to get into installing Git, but we have to do one prerequisite for those Mac users. Specifically for them, whatever you go to install Mac, the top option that you have to do is for using Homebrew to install Git. And Homebrew is a package manager inside of Mac. Windows users have something similar of Winget. Also, Windows users have already installed Git, which you can verify your terminal by typing in Git and then dash version. We installed Git, remember, because we had to install Git Bash of this terminal to run Bash inside of here. Anyway, the next section is only applicable to Mac users in order to install Homebrew, and then we're going to get into setting up your Git environment. So, for my Mac users, navigate over to this URL for Git-SCM. We're going to go into install for Mac. And in this, we need to install Homebrew if you haven't done that already. Now, you can check if you have Homebrew installed and then you don't need to follow this by typing in brew and dash version. In my case it says command not found brew. So, it's not installed. So, with this I'm going to first go to this link for Homebrew right here. And Homebrew, as I mentioned, is a package manager for Mac OS. Think of this is like an App Store but for developer tools. I've been using it ever since I've been a Mac lover and highly recommend installing this and actually need it to get Git so you don't have a option. Anyway, to install Homebrew, we're going to just copy this command right here. And then instead of our terminal, I'm just going to go ahead and paste it. Anytime you're running any commands, you do need to be aware of what it's actually doing. Specifically, I can see it's using bash to then execute this command of a curl command. And this is used to make a request to a URL. Specifically, this is calling an install. sh script, which is actually conveniently in GitHub. Anyway, I know Homebrew is reputable. I trust this. I'm going to go ahead and run it. If it asks for a password, you just enter in your computer password. It'll ask if you want to confirm by installing these directories. All you're going to do is press enter. Now, Homebrew is done installing. It's important anytime you install any of these softwares to read through here because sometimes there are, well, as this case, next steps. And if we don't do this, right? So, this says run this command in your terminal to add Homebrew to your path. We've talked about path a little bit before, but if I were to just run brew right now, it's still not found. It's not in

### [5:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=20100s) Segment 68 (335:00 - 340:00)

our path. And so, we have to run these commands in order to do that. And as always, you should understand what this is doing. This is adding contents to our dot zprofile file. Now, in a previous lesson, we did create this file. Remember I run ls {dash} a and this has our dot zprofile that I recommended that you install or created that time. If you haven't done that yet, you need to run touch. zprofile in order to get it here in your home directory. Anyway, we do have it, so I'm going to go ahead and copy both of these commands, command C, and then I'm going to paste them right in here, place enter. Hey, and even if I wanted to, I could cat. zprofile to check it out, see what it looks like. And then now, whenever we run brew, it actually has or lists the commands. I was about to do brew version, but it does tell you how you can use brew by just typing in brew. So, Homebrew's installed. Cool story. Let's now install Git, right? And for this, all we have to do is brew install Git. So, we're going to turn them all. I'm going to type in int brew install Git. And bam, it's installed. If you didn't watch the Windows version of this of installing Git, that was a mess. There's so many things and clicker throughs you have to go through. And this for Mac makes it super simple. That's why I love Homebrew. Anyway, let me verify Git is installed by typing Git {dash} version. And it's installed with 2. 39. 5. Now, we're going to be pushing our repo that's tracked via Git up to GitHub. Because of that, we have to put in our credentials. We're going to be using an HTTP protocol, and it says every connection needs a username and a password. So, let's go ahead and set that. This is going to be the same whether you're on a Mac or a Windows. We're going to start with that Git command, and we're going to type in config. And this is going to set our configuration up. And this is like our settings for Git. And then we want to apply this globally. Basically, to us. Now, we're going to set a few different things. The first thing is the user, and then {dot} name, and then fill in your name within double quotes. So, I'll put in Luke Barousse. Now, we need to put in our email address. So, we use Git config, and then specify global. And then for this, we'll be user. email. And then within double quotes, put in your email address. And press enter. Now, one more setting I want to set right now for our globally, and that is what editor or what text editor is going to use by default inside of terminal whenever you're doing any of these terminal commands. So, I'm going to do the core. editor. And for this, I'm going to specify it to be nano. And I'm going to go ahead and run this. If you remember, nano is that text editor that we've been using previously to edit files like our bash profile. By default, it would set up you to use the vim editor, which that's just a lot more trickier. Here, I'm in it right now, and you're like, how the heck do I get out of it? You have to remember to do semicolon, and then wq, and then press enter for you to get out of there. I just like nano, a little bit easier. It tells you what to do at the bottom. Anyway, we can confirm any of these changes or what we've listed so far by having git config, and then list, and then show origin. And we press enter and run this. So, now let's get into working with git using these terminal commands. And so, we've already completed step one of the git install and setup. We're now moving on to the next part, so we're in here where we're going to be creating a local repository. And then, after we set it up locally, we're going to shift to GitHub to create our remote repository. And we're going to then sync in between them using push and pull methods from git. So, for this section, we're going to be focusing on working on git here and using those three different areas of a working directory, staging area, and local repository. So, to get started, we need to run our very first git command. And in our case, this is going to create an empty git repository. You can also use this to reinitialize an existing git repository, but this is more used for maybe a corrupt files or if you need to reset some of the default settings. For this all, I'm going to be doing this from VS Code. You can do it from the terminal if you want, but mainly I want to do it from here because, well, I'm a little lazy in that this takes us directly to or navigates us directly to where our folder is. If I do PWD, I can see that we're in my correct folder for where the project is. Now, with this command, we're going to be creating that hidden folder of Git, so I do want to show right now that there's nothing in here. I'm going to do a list all, and inside of here I see, well, the folders over here to the left-hand side of EDA images lessons. Also, they have a dot and dot in here, and that just directs to dot is the current folder, and dot dot, as we've learned previously, the parent directory. Anyway, there's no dot Git

### [5:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=20400s) Segment 69 (340:00 - 345:00)

file in here. So, whenever I run Git init, cuz we're going to now initialize this Git repository, and I can see on here that this initialized an empty Git repository in my folder. If I do ls-a, now inside of here I actually see this dot Git folder. Okay, I do want to talk about this real quick. This is only applicable if you have greater than Git 3. 0, and this is referring to branches. Now, branching is something that I'm going to cover before the second project. It gets a lot more in-depth, but the core component to understand about branching is that, well, you can create different branches within your Git repository to basically work on maybe things like fixes, releases, or develop a new feature. Basically, they have different names, but the main branch that you work on is this one up here called main, and it's the default branch within a Git repo. Basically, it's the most important branch. It's the branch that everybody works off of whenever you're collaborating together. But, right now in my repo, you may not be getting this message because you may be uh using Git 3. 0 or greater. Right now in my repo, cuz I'm not, it has it as master. So, that main branch is master. Anyway, to avoid confusion for future projects that I may do within Git in here and maintain consistency, I want to switch this domain. So, it gives me a command. These are The both these commands are not commands you need to memorize. Well, the second command you will need to memorize eventually, but the first command you don't need to memorize. I'm going to copy this and I'm going to paste it in here. And what it's going to do, it's going to set the default branch name whenever we create a new and if we ever do get a knit again, it's going to set it as main. So, I type main in and I went ahead and executed this. Now, what I want to do is with our current branch, I want to change the name master to main for our current one because we already did get a knit. So, we can use this command up here to change that name. So, I'm just going to type get branch-m and then change it to main. All right, so a little bit sidetracked, but we just ran get a knit and that was the main command for you to get out of this for creating an empty Git repository. Now that we have this repository initialized, we can now move on to more of the commands that are focus of this lesson. And the main commands are these five, well, technically four here. Get fetch and get merge is thing when combined together as get pull. So, that's why I say only four. Anyway, we're going to be walking through these as we go through this. Now, although we're going to be using Git commands, I do also want to They have this area, this panel off to the side of source control we can select. Right now, it's telling me to download Git for macOS. I've already done that. I need to basically just reload this. And now the source control uh pane has reloaded. Also, I'm not liking this. I want to expand this more and that's making my terminal small. So, I'm going to rearrange the windows real quick by right clicking terminal and changing the panel position from right to bottom. And then I'm going to go ahead and just run clear to get this all cleared out. All right, so pretty cool right now in that this source control over the left-hand side is going to be a little bit of a visual aid for us. Now, I do want to stick to just using get commands, but this definitely can help us visually understand what's going on as we're moving through this. Right now, this is visually showing us that there's changes on all these files. Basically, we just set up get in it, so of course there's change in all these files. And these files correlate to all these files inside of here. Also, VS Code pretty neat, it does this green coloring right now to telling you that there's untracked changes on this file. And what this is saying, these untracked changes, is that these things are inside of our working directory and they have changes associated with them. So, let's use get add and move one of these files into our staging area. So, I'm going to type get add and then I can select any one of these. I'm going to use this readme right here that's inside of our one {underscore} EDA folder and then start typing readme, press tab so that we auto fills it in. All right, going back over here to so we can view what's going on here whenever we do this. I'm going to press enter for this command and it gets added. And notice here it gets added to the stage changes. Now, besides just adding the readme itself, I could also do something like add the entire folder. So, I could do one {underscore} EDA, run this, and now all the contents of that folder are in there. So now, I know that there's other files still in my working directory that have untracked changes in it, but just go with me for the time being. We have a few files, specifically that EDA folder, in the staging area.

### [5:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=20700s) Segment 70 (345:00 - 350:00)

Let's now get it into our local repo by using Git commit. So with this, I'll type in Git commit, type enter, and with this, a file editor or the nano editor comes up for us to put in what is going on here. So up at the top, I'm going to type initial commit. Everything underneath here that has these ampersands in the front of, this is going to be ignored, but it tells you what's going on here. It tells you, "Hey, you added basically all the content from that one _EDA folder. " And there's untracked files still within the images folders and the lessons folders, which we can see here for the changes. We'll get them in a little bit. Now that we've made those changes, we need to write out those changes by pressing control O and enter, and then now we can exit out of this by doing control X. And then I see up here at the top that for our main branch, we had a commit. This is the number of the commit, and it was our initial commit. That was at least the name we gave for it. In it, it says that four files changed with a 291 line insertions, and then it gave us all these different files that we committed to it. Visually, over on the left-hand side, we can also see that we have our first commit. As we add more commits, they're going to stack up over each other and create like a graph as we go. So we just took untracked changes that were in our working directory, used Git add to put it to our staging area, and then used Git commit to get it into our local repo. So right now, only our EDA folder is committed to our local repo. Let's get everything else inside. Now, it would be burdensome to have to go through and add each one of these files line by line. So there's a shortcut instead. What we're going to do is going to do Git add and then period. And this period, as we talked about previously, is or the way representation of the current directory. So everything inside of it is added to it. Then from there I'm going to press enter. Now we can see visually that it updated to be uh stage changes, but say you want to be a hardcore uh data enthusiast and not use this uh graphical user interface. Well, you can also use Git commands to check out what the status is. And it's as easy as Git status. And in this it says, "Hey, we're on branch main and we have changes to be committed. " And it has it in green all the different files that we need to add in here. Or the files that are now being staged to be now committed. Now let's say I went in and say changed the file contents of this file here of 1. 13 project one intro for SQL. I'm going to change this job country to job title short. I can see the file changes visually in VS Code because it has this white dot. I'm going to press command S to save it or control S if I'm on Windows. Notice visually this changes from a green to a yellow. And then also whenever I go into the source control pane here, I can see underneath changes it got added to it. So it it's it is state in the stage changes, but also it's back in our working directory with changes in it. And if I run something like Git status on this, it shows me all those different files that I have once again. And it says, "Hey, changes not staged for commit. " It says we have this new modified one. Now I do want to add that, so I can just do Git add. And then I could specifically specify that folder. And VS Code makes it simple because it allows me to syntax highlight which one I want. You could also do Git add period, but that's no fun. Also not wrong, you can do Git add period. Anyway, we now I can see visually that everything's up to date. And even when I run Git status as well, yes, all the new files are in there. There's no untracked changes. Now that we're in that staging area, we need to get it into our local repo by committing it. So, we're going to do get commit. This time I'm going to give it an M flag. And this M flag allows us to provide a message. So, we basically don't have to go in that nano text editor if we don't want to. We can type the message directly. In this case, I'm going to type this as second commit. Now, normally you would want to provide more robust type of titles with this. And actually, with for good practice, we're going to stick with that. So, I'm going to uh change this to say, "Hey, add in remaining files. " So, that way you understand better what's going on here. All right, I'm going to go ahead and press enter. And making this a little bit bigger, I see my message up at the top of that in remaining files. It says seven files changed, 99 plus insert of the amount of lines that we inserted in this. And then it shows all those different files. Notice inside of here, there are no untracked files inside of here. So, nice clean area. Also, we can see down at the graph, we now went from the initial commit to the add in remaining files. Now, previously we changed this to job title short. And yeah, we saw that the file updated. But what if we want to actually see what actually changed? So, I'm going to

### [5:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=21000s) Segment 71 (350:00 - 355:00)

actually change this back to job country. And I'm going to save it. I can see visually we have an untracked change cuz now everything's white, but this one is this brown color. If I run get status, yes, I can see that we modified this file of 1. 13 for this project intro. Well, what we can do is another command that you should know, and that is get diff, or checking the differences. And with this, it calls out the file specifically that we're talking about, and it compares it to what's in the repo, or what's in our current local repo. Sorry, I had that backward. Compares what's in our local repo to what's in our working directory, which is B. Even shows it here for the A is the negative, B is the positive What got changed in this case? As in A, we removed job title short from what was in the query and we added in job country. So, pretty cool at showing the differences. Now, you can also do this visually by going into the source control pane and then just double-clicking on a specific file and it will show you a little bit more of visually better what has actually changed in the file with the red being the removal and green being the addition. But, nonetheless, you need to know the commands get diff. This is basically pulling get diff to provide this to you. All right, so anytime we do changes to a file, we need to go ahead and add them. So, I'll just do a get add period and then from there we're going to commit this with the message to something like update intro query. All right, so running get status just to do a final check. Yes, everything is cleaned up, nothing to commit, working tree is clean. We are good to go locally. So, now we went through how to get add, get commit. We now need to get into get push, but we can't do get push because we haven't set up a remote repository in GitHub yet. So, you're going to navigate over to github. com and you're going to sign up there. Personally, I prefer to do sign ups with things like Google so that way I can just keep all my credentials in one place, don't have to make a new email and password. Once you're logged in, you're going to navigate over to your profile by clicking this profile icon in the right-hand corner and going to profile. Now, I highly recommend you clean this up cuz this is going to be where we're going to host our final dash or final project on. And so, if you have employers or somebody else coming to check it out, you want to make sure it's presentable. Specifically, I would include a picture, an updated name, a little title underneath here, and then any other of your socials that you use such as LinkedIn. What we'll be able to do by the end of this is that project we're going to be hosting, you're going to be able to, if you want to, pin it on your personal page. So, now that we have our GitHub account and profile, we need to establish that remote repo. Up at the top, I'm going to go to create new, and specifically I'm going to do a new repository. We're going to have a few data engineering projects in here, so I'm going to call this SQL data engineering projects. Now, this is the same as our the name of our folder of SQL data engineering projects, but it doesn't have to be. I like to just keep them aligned, so they're kind of avoiding any confusion. From there, I'm going to add a description. Uh this repo demonstrates my ability to use SQL for building data engineering pipelines. I do want this to be a public repository, so I'm going to keep it public. And then scrolling down, I don't want to create a read me, add a get ignore. We'll handle those two separately, and I don't want to add a license. So, I'm going to go forward with just creating this repository. Now, it directs us to this page. Now that this repository is created remotely, they have this quick setup option for those that are more experienced. We're not going to use that one. They also have this other one of if you're creating a new repository on the command line, basically if you don't have it created already locally, you're going to create it using basically these commands in your terminal. But, the one we fall into is this, or push an existing repository from the command line. So, let's now get into pushing and also pulling from this remote repository. So, we're going to focus first on just using get push to get our local repo up into our remote repository. Unfortunately, we can't just run get push, we need to run some commands to get it set up properly before we do this. Cuz navigating to our project, it doesn't really know, like it doesn't They're not connected right now. They have no idea that they are actually related and they need to be together. So, let's run through these commands that they're telling us to run in order to set this up and then push. So, for this we're going to be using the Git remote command. And this manages commands or it manages connections to remote repositories. And we need to add

### [5:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=21300s) Segment 72 (355:00 - 360:00)

a remote connection. So, we're going to add to this. And we need to give this an alias. Similar to how we alias different columns, we need to also give an alias of what is GitHub. In our case in our case, it's typically given something like origin. And then with this, we need to tell it what is the URL of this GitHub repository. And so, I'm actually going to just go ahead and copy this. I know I could have copied the whole command, but I wanted to walk through that. Then in my terminal, I'm just going to go ahead and paste that in. So, commit Git remote add origin. Go ahead and run this. Now, the next command should look familiar from what we did earlier to go. And that was this Git branch and dash M flag for main. If you remember from previously, we renamed it. It was previously the branch of master. If I run Git branch, I can see that we are on branch main. We already did this. We don't need to run that command. You can rerun the command. It doesn't have any effect or any harm. It will still stay main. The point of this is though that GitHub automatically uses main as the default branch. And so, your local repository and also your remote repository need to be aligned on what that default branch is. So, both of them need to be main. So, either way we're going to have to rename it to main anyway. Glad we did earlier. So, now we've finally done enough to set it all up that we can get it now into Git push. With this, we're going to use a U flag. And this is short for set upstream. Mainly, we need to tell it to go to well, GitHub we're going to be pushing it to which is our origin and we want it to go to that default branch of main. So, that's why we have get push that you origin main. So, go ahead and run this. You'll then need to go through and enter your credentials for GitHub. So, after you've entered in your username and also password, you should then be able to just go through or it should have gone through and pushed us up GitHub. Note, I did have two-factor authentication enabled for this cuz I've had GitHub for a while. If you do, you have to go through the process of generating a token and then inputting in here. Highly recommend you just use something like Perplexity or ChatGPT to uh to go through the steps of that. It's not applicable to most of y'all, so I'm not going to cover it. Anyway, main point is it's been pushed to GitHub and I can go to this link right here and this is our repository on GitHub. If I navigate into our one {underscore} EDA folder, it has our folder with our project and our read me all in here. Not looking bad, pretty good job. So, we just did get push to get our content up to GitHub. Now, let's walk through well, get fetch and get merge which is eventually we're just going to use a get pull, but we're going to walk through each of these individually to get some changes on our remote repo that we can get into our local repo. So, we need to change inside of our remote repository. I've been continually picking on this file in 1. 13, specifically this project one intro sequel file. And so, let's make changes here in a remote repository. I'm going to go up in the top right and go to edit this file and I'm going to once again change job country to job title short. So, I've changed that and now I'm going to go in the right and click commit changes. If It used AI to write a commit message, which is a very good commit message, I'll give by the way. I don't need to add an extended description, and I want to commit directly to the main branch. I don't want to create a new branch for this commit. We'll go over branches more in the second part of this or the second project. Okay, I'm going to go ahead and commit changes. All right, so now project one intro SQL file is updated to job title short. But, on my local repo, I can see that it's still at job country. Now that we have these changes in our remote repository, let's get fetch them to get them in our local repo, and then get merge them to actually start working with them. For this, I'm going to change this to the source control pane to be able to see what's going on the right-hand side or left-hand side visually. All right, so let's go ahead and get run get fetch. And it's important to understand, if you think about it from this, does it know where to fetch from? Well, it does from because when we ran get push earlier, we used this U flag to set the origin main, meaning we set that of GitHub as the origin to pull from that main branch. So, it does know. So, running this All right, so from this message, I could see that it did download a file, and everything seems There's no error message, so everything seems to be updated correctly. Notice here, right, so we still have it still says job country here locally on our file because that's in our working directory. But, in our remote directory, it says job title short. So, what happened is when we get fetched it, it

### [6:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=21600s) Segment 73 (360:00 - 365:00)

grabbed those files and put them into our local repo, but it has not yet updated our working directory. And we can see this visually in the graph on the left-hand side. Origin, once again, is GitHub, and that main, and so we can see that we're actually a commit behind what is happened up in GitHub on a remote repository because we're down here with this main. And that is our local repository's main. This incoming changes is basically remote has commit that we do not have just yet. So, if I wanted to check what these actual differences are, I could do get diff. And then I wanted to look at what I have or where I'm looking at in the files and this is using the nomenclature of head. And that's right here pointing directly to where main is locally for me. And then we're going to use some range syntax and point it to origin's main. So, we're pointing what is the difference between this one to this one. Now, running this oops, I typed organ not origin. So, make sure that you have the to right correct typing on this. Let's try this again. Okay, so with this just scrolling this up a little bit more. And in this it calls out how we change this from job country to job title short. Also, for some reason I guess I didn't catch it. We did some sort of new line addition at the end of the file or remove that new line addition. Not a big deal. Nevertheless, it shows here. It tracks everything. So, now let's get this file updated with those changes that we fetched into our local repo. For this, we're going to do get merge. And we need to specify this branch here of origin/main. Specifically our remote tracking branch that we want to merge with. And as you can see from the file above, it updated to job title short. Now, that was with using get fetch and get merge. But I don't like to use those in practice. Instead, I like to use get pull instead to just get it directly in my working directory. Also, I didn't point this out earlier, but our graph updated to where we now have all four commits and our main, our local repo is aligned. It's head of main is located with the same commit as our origin main, which origin main being our remote repo. Anyway, we're going to change we're going to edit this file again one more time. I'm going to change job title short back to job country. I'm going to commit changes. Add job country as distinct job title query. I want to do that. I want to keep it to the main branch. I'm going to commit those changes. Remember cuz we're going to do a get pull. It has been updated. And now in here, I'm going to go ahead and just run get pull. I don't need to specify because we're like we talked about earlier, I don't need to specify what our origin is because that's been set up already with that U flag when we ran get pull. So, we just need to run get pull. Boom. And it updated to job country and along with our graph here showing that hey, we advanced ahead one more commit. So, if you take anything away from this lesson, it should be those four get commands of get add, get commit, get push, and then get pull. Now, I can understand this can be a tough concept to wrap your mind around. So, we've come up with some practice problems for me to now go through in the supporter resources. In this, you're going to be creating a remote repository from scratch and a local then doing a bunch of push and pulls and different type of commits and actions in order to get more familiar with get commands. In the next lesson, we're going to just finalize our project by getting it making sure it's up to date on GitHub and then also sharing it on LinkedIn. So, with that, I'll see you in the next one. All right, congratulations for completing your first data engineering project. In this video, we're going to be going through and just doing a quick double-check and along with an addition to our project to make sure it's buttoned up and good to go to present on GitHub. And then after that, we'll then transition over to LinkedIn to work through how you can share your projects and also make a post about it. First up, if I navigate to where our GitHub project is right now, inside of EDA, yes, I have my project in here. I would go in and verify that all of the images are uploading properly. That's the biggest mistake I see with other students or that whenever you click a link, it doesn't navigate you to where it needs to. So, make sure all your links are up to date. Anyway, if I go back and navigate into itself, we have a little bit of a snafu in that we don't have on the front page have a read me that is detailing what this repository is about and it's going to capture all of our projects that we have. Now, this is my repo that I'm using that has all of our projects that we're going to build. Anyway, we're going to have

### [6:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=21900s) Segment 74 (365:00 - 370:00)

multiple different projects inside of here. So, what I do is I've built a read me that just showcases that I have these two projects and then from there they can navigate into if they want to learn more about what I did with EDA, they can click that and then they can go see what I did with EDA. Anyway, we need to make that real quick. So, back in VS Code, I'm just going to create a file real quick and we're going to call it readme. md. We'll give it this a heading one and call it SQL data engineering product projects. So, first I put in there this, "The following projects are a collection of SQL projects that I've worked on to practice and reinforce my skills with data engineering tools. " Then I have a call out that says click the project name below to view the tools I used to build these. Now, we're going to be adding a second project, so I'm going to title this the section projects. And then underneath this, I'll title this one EDA, give it I want to make it into a hyperlink. So, I want to go into that one EDA folder, so I do a backslash, go into one EDA and I want it to go right there actually. And then I want to do exploratory data analysis. Then I want the image of the underneath here. So I navigate into our other read me that has this in here. Copy it and then paste underneath. It's not going to work cuz it's trying to navigate up to the parent directory, which there's no parent directory where this read me is. So I'll just take that off and now the image is working. Then I'll put a little description underneath this about what the project's about. Now this should be enough. I am going to make this into this EDA into a heading. So if I put three ampersands to make it a little bit bigger and call out. That's good. I'm going to go ahead and save this. Then from there, you know what we got to do? We got to push this up to GitHub. So I need to add all these files. So we'll do get add. I'll do a get status to see that we've added our read me. I'll do a get commit cuz we want to now quit it commit this to our local repo. And I give it the message of add parent read me. Now we just got to push it. So we'll do get push. Boom. Then inside GitHub, I'm going to verify that's up to date by refreshing. Okay, it's been added. Are all my pictures working? Yes. Are my hyperlinks working? They are. All right, looking good on GitHub. All right, now in LinkedIn we need to do two things. Well, I guess three if you haven't done already, but you should create a profile LinkedIn. LinkedIn's super powerful with being able to connect with employers and network. Even if you're not looking for a job, it's a great way to build a reputation and then have opportunities come to you. Plenty have come to me. That's how I got my Mr. Beast job. No lie. They messaged me on LinkedIn. Anyway, I'm assuming for this you have a profile. We're not going to go through set up a profile, but at minimum have should have your name, about you, an about section, and fill in all your different experience that you have previously. Now on your profile, you have a section down here on projects. And this is where you want to add in this project. Now if that section was not appearing, you need to come up to the top and go to add profile section and then go to add project. So in here we need to put a catchy title for this. I'm going to put this of data engineer skill analysis {dash} exploratory data analysis. And then for the description, I just copied out of the read me what we already had for the description of it. For skills, at a minimum you need to list SQL, but I also recommend including terminal bash and get. Also throw in GitHub, but you could throw in something like VS Code as well. You're only limited to five. Next, I'm going to go into add media. And for this, we're going to be adding a link, specifically a link to our SQL data engineering project. And I'm actually going to go to right directly to this one. I'm going to get the link for this. That way it navigates right to the page that has all the details about this actual project. Going to place the link in right here. And it looks like that's not working right now. However, if I take off this tree main and EDA what it's navigating into what I want it to do, it does look like it's not uh populating below. So, unfortunately, you just have to do it like this. It looks like if you're encountering this error right now. Anyway, in here it already has a title and everything like this. I'm going to leave the description blank cuz I already gave a description for the project, so they should know what it is. Put your start and end date of the project. I've been working on this dang course since June, so I'm going to list this entire 6 months. Add any contributors that helped you with it if you worked with somebody along this. And then from there, click save. Now, after you do this, I also highly recommend that you make a post. I get posts like this all the time where people are tagging me in the projects they complete for my courses. In it, they just do a quick detail about what the project is. They tag my name in it so that way I can see it. And then from there, they also list a link to the project so you can go in and actually view what they actually did. Pretty cool. And to make it a little bit more interactive, they usually include some sort of pictures, something like that. For this project, I highly recommend either including some code snippets in your picture or this image right here of what we actually did. Anyway, as I mentioned, I love seeing these projects. I go to the project, check them out, and I'll comment on it to hopefully improve the

### [6:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=22200s) Segment 75 (370:00 - 375:00)

reach and others can see this. All right, congratulations again on finishing up your first project. We're now going to be getting into the second half of the course, and things are going to be ramping up quite quickly as we're going to get into a lot of heavy engineering concepts around how to use SQL to build out data warehouses. All right, with that, I'll see you in the next one. In this lesson, we're going to be going over data types. And at first, it may seem like this lesson is silly and should be basic understanding that everybody should already know, but it's actually really important that we understand the basics and the reasoning behind data types because we're going to be building out SQL pipelines that populate different tables and views from scratch. And so, we had need to have that understanding of data types so we can build our tables with those data types. Now, I've hinted quite a few times towards how our columns have certain data types set already. If in Motherduck, we've seen this as follows. If you pull the job postings back table that populates at the bottom, and then when I click those ellipses, I can go into show types. And it goes through and shows the different types that we have here of integers, characters, boolean, or timestamp. So, why the heck do these data types or setting a column as a certain data type even matter? Well, I have four main reasons. First is data integrity, and that deals with actually downstream of you. If people are building queries or models off of your tables, if you don't have the right data types or change the data types or aren't consistent with it, you can break their logic. Conversely, it also works with data validation having a set data type in a column. This prevents any bad records from even getting into your table. Now, with that, there's also performance and storage gains. You're going to get to the point where you're working with tables with billions or trillions dollars of data. And at that stage, you need to be able to make sure that you're using the most succinct way to carry that data. Something like a boolean that is either a zero or one or true or false takes up a lot less memory than something like a varchar character that can take up tons of characters. And then finally with this type specific behavior. And that's that we can do certain SQL functions to a column depending on its data type. Which brings us into the more important topic of what the heck are the different data types? Well, I've booked them or put them into five general bins based on what I use on a daily basis. The first four are the most common and that fifth one is just an other or common types of data types that I just sort of lumped all together. They don't necessarily all relate. But the first four are ones that I do use and that's numeric which are used for numbers like integers, decimals, characters used for text or strings, booleans that record whether it's a true or false value, and then date and times which are used for things like timestamps and time based data. Now, for numeric, there's generally about five different ways that you can define what a number is going to be. First one's pretty simple that it's int or integer. It's a whole number. Also, some systems have a small int or big int where small int is whole numbers within a smaller range and big int is in bigger numbers within a bigger range. Right now, we're just going to keep it simple to int. Next are float and double. Both of these are approximate decimal numbers, whereas float is for lower precision. As you can see, there's a little bit more rounding here, whereas double is much longer. The science behind it is that floats using 32-bit to evaluate this and double is using up to or often around 64-bit to evaluate it. Double is commonly used for analytics and statistic calculations. That's why I've ingrained it where our salary year average and salary hour average are both double. Next are decimal and numeric and functionally, depending on the database, they actually can be the same thing. They can just be an alias field. Anyway, in parentheses, you can put things like P {comma} S. And this specifies your total digits and then your digits after the decimal. So, in the case of the 12. 1, the total digits is three and the decimals after the decimal place are one. Next is character, which is used for text or strings. If it's a super short amount of characters, you would use char with a parenthesis of n to specify how many digits. Typically, it's like two or three. Now, the key thing about this is a fixed length. So, whatever number you put at put for n for max characters, that's how many it will be and it will cut it off or it will fill in with blanks if it's too short. Think of like a state or a country code, something like that you know it's going to be set, you'd use that. Now, if the text is going to vary a little bit, you're going to use

### [6:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=22500s) Segment 76 (375:00 - 380:00)

varchar. But, varchar means variable length character. For the majority of our columns in here, that is what I coded them as. Now, with that comes the caveat that that's not necessarily the most efficient way to store data, but there there's no really no other choice for this. And at least I didn't do the last option of text, which is a long unbounded text that you could include. There's no length limit on this. Next up is boolean and that's recording whether a value is true or false. Like we discussed before, these values are behind the scenes held as something as either a zero or a one. In some databases, it may even be written as bool, b o l. We use this in our work from home, no degree mention in our health insurance columns. Next up is date and time, and we have a whole lesson on how to use date functions. We're going to dive deeper into these, but for the time being, I think you can understand that some of these are dates that record a date, time that records a time, and then timestamp, or you could see it in some databases as date time, and this has a date and time object to it. Now, you could also see it with timestamp z, and the z stands for zone, and this has the time zone at the end of it. Right now, this one, the time zone is just UTC or coordinated universal time zone. Basically, what's at the prime meridian. Finally, our last bucket of other types, and this we're not going to get to any of these in this course, but I just want you to be aware that there are other types available, such as arrays, things to handle JSON data, UUIDs, so long identification numbers, or even something like binary. And that list is not an all-encompassing, there's many, many more. Now, how should you be checking what data type a column is? Well, as we learned about before, you can use something like information schema that provides this based on the metadata that's stored inside of a database. So, let's demo it. Inside of VS Code, I'm inside of my project here. First thing we need to do is connect to our database inside of Motherduck of that Jada Jobs database. I'm going to start a new file for working in here for running different commands. Now, for this, I'm going to select all for right now, and this is going to be from information schema, and specifically columns, which information schema is, well, the schema, and then columns is the table, so we're using that dot notation to access it. We'll go ahead and see what we get back with this. Oops, forgot my colon, or semicolon. And, yeah, a lot of stuff in here. Uh so, I need to actually filter down based on the table name. So, I'm going to do a where table name is equal to job postings fact. And then I'm going to clean up the columns too, cuz there's a lot of columns in here. I want that table name, I want the column name, and then also want a column called data type. This time, let's not forget the semicolon. We're going to go ahead and run this bad boy. All right, this is a lot more manageable to read, and we can actually see all the different data types associated with our columns. The same as what we would have gotten in our mother duck view. Now, the other option you can do is using the describe command. And if I run this on job postings fact, so I'm doing this down below, this provides very similar data that we saw previously and a little bit additional. Now, the one problem with this is describe isn't universal to all databases. It is DuckDB's part of their friendly SQL, but some databases don't have this, like SQL Server and Postgres. Now, describe is actually pretty cool because you can use describe on top of another query. Specifically, I have this query right here. I'll just run that query real quick. And this provides back a column on job title short and salary year average. Now, it does say the data types in here, but if I wanted to, I could just run this and it provide a column back with not only column types, but also any other information about it, such as if it's keys or if it allows nulls or not. Now, let's put this knowledge to test of understanding the basics behind data types and actually converting data within queries to a different data type. And we can do this within select clauses by casting, which refers to the operation converting a value in a particular data type to the corresponding value in another data type. We're going to show this first with using the function cast. And inside of here, you have your column or value. In our case, let's say it's the integer of 1 2 3 and we want to convert this to something like a string character. So, I'll just convert this to a varchar. So, let's go ahead and run this bad boy. And we can see we get 1 2 3 and it is of the type varchar. Now, you can't convert everything to just any old data type. Let's say for example this 1 2 3 was a string instead and it also had some digits in it such as like d e f and I

### [6:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=22800s) Segment 77 (380:00 - 385:00)

wanted to convert it now to something like an integer. Well, whenever I try to run this, I'm going to get a conversion error. Could not convert string 1 2 3 d e f to in 32 or an integer because it's got these letters in it. But, if I were to actually remove these letters from here and then run this, it's going to work just fine cuz all those are numbers. So, let's get an example with this. Let's say we have the following columns that we're trying to query, job ID, job work from home, job posted date, and salary year average. Getting this from the job table. I'm only going to limit it to 10 to not overload the database. And we have comments next to each on what we want to do to that particular column. Job ID is by far the most complex, so we're going to start right down here with job work from home. And this one we want to convert from boolean to numeric value. Let's actually even look at all these values first. And we can see that job ID is in fact int, work from home is boolean, job posted date is a timestamp, and salary year average is double. Anyway, we want to convert job work from home from boolean to numeric value. Well, anytime you're doing any conversions, I'd recommend consulting with the source documentation linked here. And we just want to go to a numeric value. Specifically, I'm thinking we want to go to an integer. And for this, it has the name but also there's other aliases, so I can use any one of these to get a an integer. So, I might use cast here and then inside of here I'm going to specify that we're going to cast it as an int. Put in closing parentheses. Now, running all this, I can see that it is now int and all these are zeros and that's because previously they are all false values. False values correlate to zero, true values would correlate to one. Anyway, I'm also give this an alias so that way we keep the same name of job work from home. All right, next one. We want to convert that job posted date from a timestamp, which it has a date and time, to actually reflect its name of job posted date. So, I run the cast function on this. Specify we're going to do this as a date and then give it the alias job posted date. Running this, we now have only dates in those column. Salary your average, we want to convert from double to no decimal places. Well, going back to our documentation, we can see we have this one here for decimal and it takes the two arguments of precision and scale, which are the full amount of numbers for the precision and the scale is the number of decimal places after. So, we'll do cast on salary your average as decimal and we're going to specify 10 along with a digit of which will no digits or zero, so no decimal places. And I'm only going to do 10 because I would expect all these values to be less than, well, a million dollars and if they are, they're outrageous anyway, we'll probably need to get them out of here. So, this will be a good filtering mechanism. Running this, well, we can't see any values cuz they're all null right now. So, I'm going to add a where statement in here of where salary year average is not null. All right, let's go ahead run this bad boy again. All right, and now we're getting nice, clean values for this. I will give it an alias, too. Now, this top one and we're going to do job ID and we want a more unique identifier. With this, what I'm thinking is we're going to combine it with company ID cuz it's a pretty unique identifier as well. Let's go ahead and run this to show you what I mean. Anyway, what I want to do is I want to combine these two this job ID and this company ID column to one. I don't want to add them together cuz right now they're both integers. I want to basically append one to the other. In order to do that though, I'll need to convert them to strings and then from there append them. So I'm going to cast that job ID as varchar and I'm going to do the same thing with that company ID. I'm going to run this real quick just to see what I get. Okay. I have both of these columns now and they're basically they're varchar. So I can see that they're a text value because they're offset to the left side of the column whereas numerical values like the salary average are to the right side of the column and just like this job work from home. That's how you can quickly check whether it's a text or a numeric value. Well, I'm going to use a concatenator operator and this concatenates two strings list or blobs. In our case, we have two strings. They're basically put them together. Could also use the concat function, but I prefer the operators that makes it more readable I feel. So I'm going to take this one up here, put on those two concat operators, not two concat operators, just one, but I'm putting them right next to each other and now running this bad boy. All right, I'm getting them combined. Looks a little strange to me. Now I could wrap this all in another cast and then convert it as an int to get it back to an integer, but actually what I'm going to do is I'm going to put in between these is a hyphen and I'm actually going to have to stick another concat operator in there as well. And now whenever running this, we get this

### [6:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=23100s) Segment 78 (385:00 - 390:00)

nice little hyphenated unique identifier. Pretty cool. Now besides using the cast function, DuckDB supports a popular option that's used in other databases like Postgres and uses an operator. So I can use this double colon to cast this I here as a double. So, I went through and updated what our previous query was using cast and just replace it out with this double colon. Personally, I feel that the latter option is a little bit more readable and it doesn't have cast all over the place. You can just follow along and read it through of what's going through. Anyway, whenever I go to run this bad boy, this provides the same exact results that we saw before. So, I've just as efficient. Now, one last thing, you should be explicit when you're casting anything that there may be ambiguity between two columns when we're combining them. Say in this case I was adding something like three, which is an integer, to something like 5. 5. Whenever I run this, I can see that I do get back a decimal of 8. 5, but remember three is an integer, 5. 5 is a well, in this case a decimal of 12. 1. If for some reason that you wanted to maintain it as an integer, that would this would be a good use of that casting operator in that you could get that out of this. But, it really depends on what you're trying to do and what your end goal is. So, casting is going to save your butt in data pipelines. It's not going to necessarily hurt you now, it's going to hurt you down the road later because some junior data engineer is probably going to put in some values that they shouldn't into a column and you need to have the code in place to check for this. All right, it's your turn to give it a shot. We got some practice problems for you to now go through and get more familiar with not only using that cast function, but also the cast operator. And now that we have this level of knowledge on different data types, we're going to be shifting next into using a different type of SQL, DDL and DML, which we'll go into. But, this uses our knowledge on data types to build out future tables and pipelines. With that, I'll see you in the next one. — Welcome to this lesson on DML and DDL commands. DML stands for data manipulation language and DDL stands for data definition language. Now, these type of commands are imperative in order to build out and create different databases data warehouses and even a data mart, which we're going to do. So, over the course of these next three lessons that we're going to be doing on DDL and DML, we're going to be building out this job mart and for it, we're going to be exploring all the different commands or the most important commands that you need to know within DDL and DML. Now, in this lesson, we're going to focus on just the basics. We're going to start out simple and look at how to use things like a create and drop command to create and drop databases, schemas, and tables. After we know how to create these objects, we're then going to move into inserting data into a database and specifically into a table and updating records within a table. Merge is a more of an advanced concept and so, we'll be covering that in our third lesson on this topic. Finally, we'll get into altering tables, looking at things like adding columns, dropping columns, renaming them, renaming tables, and some other popular features with it. Now, this lesson does cover quite a bit of different keywords, but I don't think it's anything beyond your capacity. None of them are that complicated that you can't really wrap your mind around what it's actually doing. Now, up to this point in this course, we've really only covered one type of SQL command and that is DQL or data query language. These are commands that query and retrieve data from tables. Things like using select and from and then filtering as necessary. We're going to be using where and then maybe even aggregating it using something like a group by. Now, what we'll be jumping into well, as first is DML or data manipulation language. This is used to well, manipulate data within tables. We're going to be using keywords like insert, update, delete, and merge in order to change or update data within an existing table. But, that's for existing tables. What the heck do we do to actually even create a table? Well, that's where DDL commands come in or data definition language. These define and modify database structure and schema. These use keywords like create, alter, drop, or truncate. We're creating and destroying complete objects with this language. Now, I will add a disclaimer. Technically, there's five different types of SQL language. We're not going to be covering the two other ones of DCL, data control language

### [6:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=23400s) Segment 79 (390:00 - 395:00)

which is used to control and access permissions within databases. That's for more database engineers. And then also TCL or transaction control language, which manages transactions and ensure data integrity. That's even more complex than we're going to need to get into for this point. Now, what the heck are we going to be doing with this DML and DQL language? Well, we're going to be working with creating our own database. As you know, this is our existing data warehouse that we have and this is the data jobs where we have the main schema and our four main tables that relate to each other and has all of our job posting data. This is a database that is a data warehouse. Well, we're going to build something that would be downstream of a data warehouse and going to be based off of our data warehouse that is a job mart. Now, this is a data mart and it's subject specific. So, it's curated subset of data designed for a specific business function or analytical use case. If you had something like a sales team, they want to look at something like revenue or orders. And then if you have like a marketing team, they'd want a mart that would look at campaigns, clicks, and conversions. So, over the course of these next three lessons, we're going to be building out different components inside of here to better understand the DML and DQL language. Now, just a sneak peek, we will be building out not only the data warehouse, but also a few other data marts in our final project at the end of this course. So, if your mind's not wrapping around completely what data marts is, don't worry, we're going to have plenty of practice coming up even after this. But, the main thing to understand is that data marts are downstream of a data warehouse, and they have a curated set of data that's coming from a data warehouse designed for a specific business function or analytical use case. They're typically smaller, that's why I made them smaller in this picture. Now, where are we going to be executing these SQL queries? Are we going to be running this locally with local databases or in MotherDuck? Well, we're going to be using the latter, in MotherDuck. And this is going to give you practice with creating and setting up different databases in the cloud. Now, if you want to, you can do all of this locally as well. Some of the commands will change from time to time, and I won't necessarily be going in-depth about how the differences are to execute this locally. I'm going to be focusing on specifically MotherDuck executing it in the cloud. All right, our first set of commands that we're going to get into are create and drop. For this, we're going to focus on creating our database, and then inside that database, create a schema, and then inside a schema, create a table. Now, drop is a very dangerous keyword. You can ruin people's lives if you run drop on the wrong database schema or table. So, make sure anytime you're running it, you know exactly what you're doing and think twice. Now, let's jump in and start programming. For this, I'm going to be operating all of this on a Mac, but I've done this before on a Windows as well. Everything's completely the same. Anyway, I went ahead and created a file called 1. 21 DDL DML part one, and this is where we're going to be executing queries from. In the terminal, I'm going to go ahead and create uh connect to Motherduck. Note, if you're running this locally, you just go out and list that duck. db file that you want to create. So, enter run that, that would be the create warehouse or create database that you're creating for this. That's uh the extent of the support I'm going to give if you're going to do this locally. Anyway, I'm going to connect. I'm going to connect specifically to the Jada Jobs uh warehouse in Motherduck. First thing I'm going to do is just see what databases we have listed currently inside of here. Now, right now, it has four different databases, information schema to get informations about the different databases, sample data, data jobs, and also my DB. Anyway, none of these are what we want to create. We want to create another one. So, we're going to use this syntax of create database and then list the name of the database. If not exist, we'll get to that in a bit. I'm going to add in create database and then the name we want to use for this, and I'm going to call this jobsmart. Don't forget that semicolon. Now, I'm going to go ahead and run this. Bam! You just created your first database. Now, let's see if it exists. I'm not going to run dot databases. Instead, I'm going to run something that's more dialect friendly. Preferably, command show databases. Okay, we got five now. We can see that jobsmart has been added to this. Too cool. Now, back to that if not exists portion that can be added to this. Let's say for some reason we had a script we were trying to run an automation script to create databases, tables, and schemas. Really common for data engineers to do. If I were to try to

### [6:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=23700s) Segment 80 (395:00 - 400:00)

rerun this command right here, I'm going to go ahead and do it. I'm going to get this error right here, binder error. It says, "Failed to create database, database with the name job mart jobs mart already exist. " We can add this condition of if not exist, therefore it can attempt to create a database that already exists without throwing an error and stopping execution. So, I'm going to add in if not exists. Now, whenever we run this, we don't get an error. And checking the show databases, it still exist. Now that we have created a database, let's drop it. We got to be very careful with this one though. For this, I'm going to use the command or keywords drop database. And then be very careful we're listing the actual one. So, job smart is the one we want to drop. I'm going to go ahead and run this. Okay, and now I'm going to select show databases to see what's available, and it's deleted. Now, once again, if we were running some sort of automated script with this, if we tried to run this again now that we've dropped it, we're going to get catalog error. And that's a database with the name jobs mart does not exist. Do you mean my DB? It tries to relate to maybe a similar one. Anyway, with this one, instead of writing if not exists, we want to check if it exist, then you can try to drop the the database. So, I'm going to add in if exist into here. Now, let's try to run this bad boy. Boom, don't get an error. We do want this database though, so I'm going to go ahead and comment out this line. I'll just keep it there for the time being. And we'll rerun this create database so that way we have it. And just double checking it, running show databases, it's in there. All right, next up. Now that we have our database created, we need to now create a schema. For this, we're going to be using the keywords create schema, and it has very similar in that you can use an if not exist optional keyword inside of there for if it exists or not. And then you list the name of it. Now, for the data mart that we're going to be building, we're going to have two different schemas. a staging schema where we basically stage our data before we send it into the other schema of the main schema. So, before we actually even create the schema, let's check what schemas are inside of this database. For this, we're going to be using information schema, specifically checking the schemata. This is a database agnostic. Basically, you can run this in most all databases. Let's go ahead and check this. I had a typo in information schema. Let's try this again. And inside of here, we can see that we have our jobs mart, and specifically there's only one schema of main. By default, DuckDB creates your database with one schema of main already. So, this main one's complete. We just need create the staging schema. Oh, also with this main, you can't delete it. It's there by default. It's there permanently. So, let's create that schema by using that create schema keyword. And then now, we can't just specify staging because what database are we connected to? Well, if I scroll up and see, we connected initially via DuckDB to the data jobs schema or the data jobs database. So, it would try to create the schema there. One option is we can list the database name in front of this. So, in our case, we're going to list that jobs mart in front of it using that period notation. And then I can go ahead, and then putting that semicolon in there, running this, and then checking it the information schema, we can see now that jobs mart has this staging inside of it. Now, we're going to continue to use jobs mart throughout the remainder of this lesson and the next few lessons. So, I don't want to have to keep on writing JobSmart every single time. It gets annoying. So, above this, I'm going to add the command of use and then specify the data the database I want to use of JobSmart. This use keyword is only allowed in some other databases like SQL Server and MySQL and also DuckDB obviously. But, anyway, let's go ahead and run this and show what I mean by this. So, we got was able to use that. Then, I can go ahead and remove that now I create the schema of staging, well, it's a schema with name staging already exist. So, I can add in if not exist and then run this again. And so, now one, I don't have to list JobSmart because it knows this and two, it created our schema without any issues. I can just double check it by going here and checking the information schema and then I see, okay, staging is inside of JobSmart. All right, we've created a schema. Now, we need to delete it or better said, drop. And we're going to be dropping the schema, specifically that of staging. Okay, let's go ahead and run this bad boy. Looks like the command run just ran just fine. Now, let's go ahead and check information schema and inside of here for JobSmart, it's no longer there. Like drop database, you can also include the optional command of if exists. Think

### [6:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=24000s) Segment 81 (400:00 - 405:00)

you get the point of that. We're not going to run that again. So, I'm going to actually just comment this out cuz I want to be very careful not to drop the database or drop schema. All right, next up is tables. We're going to be creating and dropping a table. Creating table and dropping are they are the same commands that we've seen before. But, for this, we're going to be creating a table called priority roles. And this is going to be used by the business to track which job postings they are marking as a priority to track. Now, we're going to end up renaming this and let's say we accidentally started out with creating this that we wanted to name it preferred roles, not priority roles. Now, this keyword is a little bit more involved than that for schema and database. Sort of glossed over that. But, we need to focus more on what is the syntax of this. Well, first of all, we start with that create table. If not exist is an optional command. So, I'll put that square brackets to make that more clear. Next after that is the table name that you're going to give to it. And then underneath that in parentheses or actually you can do it off to the side. This is the once again, indentation doesn't matter in SQL. It's just if it's more readable. But, underneath this, what do we have on each different line is the column. Now, that first column is optional, but we want to build nice databases that have a lot of similarities to how data warehouses are built. And people expect an ID column. This is typically of the data type integer. So, we'd specify that data type of integer. And then next to it because we want this to be our primary key, we'd put that out right next to integer. Then from there, underneath this, you would list any associated column names that you want in there and their associated data types. If you had any foreign key columns, you'd list their data type. So, for this, we're going to start with that create or the create table command. And this is going to be called preferred roles. Note, we're going to be changing the name of this preferred roles later to priority roles with the main name of the table, but I'm just doing this for demonstration purposes that we can change the name later. Anyway, inside of here, I'm going to go ahead and open parentheses and then indent down underneath it. And we're going to keep it simple with only two columns. We're going to give one of role ID cuz we're going to have also roles associated with this. Anyway, we'll get to that. Then we're going to give it the data type of integer. And then the role name. We'll be putting in things here like data scientist, data analyst, data engineers. So, this is going to be of that data type varchar. Once again, I don't know if varchar is approved to say for this, but it's fun to say. Now, I used integer and varchar, but remember there's a bunch of other different data types that we can use to assign the data type of a column. And we covered that in the last lesson. Anyway, let's go ahead and run this bad boy. I don't know if anybody knows this, but we haven't We're going to have an issue with this. I'll go ahead and run it. Well, the first is that there's a comma that was missing in here. So, I'll go ahead and run this again. All right, it ran with no problems, but let's go inspect it. Specifically, we're going to be using information schema again. We're going to be coming from, well, information schema. tables. And I'm going to just filter this down for the table catalogs, that way it we can make it a lot easier to read, and that of job mart, cuz that's what we created. Put a semicolon on the end, go ahead and run this. And nothing returned because I'm silly. I have job mart, it's actually jobs mart. Let's try to run it again. And okay, it's inside of here. Preferred roles inside of here, but what the heck, it's inside the main schema. Well, the problem with this is anytime you run this or run to create a table, you have to specify the schema. In our case, we want it to go into the staging [snorts] schema. I'm going to go ahead and run this now again. And it ran into an error there actually. I don't know why, but my schema Maybe I did run drop the staging schema. Anyway, I had to recreate the staging schema. But this time, I ran it, boom. And we created preferred roles in staging. We can go ahead and check it by, once again, checking information schema, filtering for the table catalog for jobs mart. Okay, now we have preferred roles, but we have it in two places, main and staging. So, we need to drop it. Now, this one doesn't have any additional syntax to memorize with this or to learn with this. It's just drop table, and you can use that additional application if it exists. So, we want to drop table, and specifically I want to delete the one inside of the main. I can just write preferred roles and drop this one. So, running like this, and then inspecting inside of there to see if it in fact delete you looking at information schema, I can see that only staging is inside there. If I wanted to write this drop table with main. preferred_roles, that would also be acceptable and probably more preferred just to make sure you're more precise. In our case, this is already deleted. If I were to run this again, I'd have to include if exist, run this bad boy, and it would actually work. So, now that is the basic keywords of create and drop

### [6:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=24300s) Segment 82 (405:00 - 410:00)

whether using with database, schema, or table. Now, as a data engineer, you're going to be assigned to build scripts that do item potent operations. Item potent as defined by Google in here is denoting an element of a set which is unchanged value when multiplied or otherwise operated on by itself. Basically, if we build some sort of script that we want to be safely rerun without any side effects or errors, we want to build it in a way to do this. So, right now, this is our script. I'm going to go ahead and save it by pressing command S or control S on Windows. Let's say we're building this and we want to maybe put it somewhere in the cloud to rerun to build this from scratch. And by running the whole script, what I mean is well, I have everything highlighted here. I'm going to go ahead and press shift enter to send that into the terminal. But anyway, as it tried to go through this whenever I gave it the command, it starts up here, I ended up getting an error halfway through here because of preferred roles, that table already exist. So, in our case, I would have to add if not exist. Okay? Now, when actually running this entire script right here, I don't get any errors with it. So, this is pretty good that we built this. Now, I will say an easier way to do this, instead of having to highlight this entire file, DuckDB has a dot command for this. Specifically, it's read. And for this, you have to remember we're right now for our terminal inside the SQL data engineering's project folder. We want to run this folder right here. So, what I can do is right-click it. I can go copy relative path. And then inside of here, next to read, I can go ahead and paste this in here. And so, it will navigate down into the where I want it to run the script. Okay, now press enter. And as we can see underneath here, it went through and ran the script. Notice uh name of preferred role already exist. Uh I guess I was silly earlier when I said it worked just fine. I guess I missed it. I didn't save it. Anyway, let's try to run that again using read. Okay. Now, when I'm going through this, I can see it listed databases, listed the catalog. Another thing, there's no error messages anywhere in there. So, it worked just fine. It's item potent. Anyway, I'm going to want to do this again. So, I'm going to copy this command, and I'm going to go ahead and actually insert it up at the top. But, key thing here is I'm not going to cuz that'd be really strange. I'm going to comment it out. It's just going to be up here for if I want to run that. All I have to do is just highlight this. I don't want it to run this command, then run that command, and it just be in this never-ending loop. That'd be crazy. One last tweak that I'm going to make to this script as we're going through this is I do want it to start from scratch whenever it's doing this. I don't want to Every one of these has this if not exist in it for the create schema, database, and table. Basically, it's checks that it exists, and it doesn't run it. I do want it to do it during this. And that's just the operation that I want that I'm just particular about. So, this drop database, I'm going to go ahead and actually stick this up at the top. So, I know it's going to drop the database and these items then shouldn't exist anyway and we're going to be creating them from scratch. We'll go ahead and save this and then copy this command up at the top that the one then paste into the terminal to do this entire script including dropping. And I get this error, "Cannot detach JobMart because it is the default database. Select a different database using use to allow detaching this data database. " Basically, I'm trying to drop a database that in our script here we have use JobMart. So, I need to use a different database first before I actually drop it. So, I'm going to use data jobs, which is a different database. Then, I'm going to drop JobMart and then go through the creation script, which will then switch back to JobMart. Saving this and then pressing up to refresh that last command, we're going to try it again. Hopefully, we're right. And we did it with no errors. Everything happened just fine and we can see at the end that we did create it. Pretty cool. All right, next up is insert into. And this is grouped all I put it into this diagram with two other ones of insert, update, and merge. Merge is just a combination of insert and update and it's more advanced. We'll be covering it in the third lesson when we go through DDL and DML. Anyway, in this lesson we're only going to be focusing on insert and update. Insert is pretty simple. It's just a row-level addition. If the row already exists, it will fail on duplication. And this is best for things like new data ingestion into a table. So, we just

### [6:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=24600s) Segment 83 (410:00 - 415:00)

created that preferred roles table right here. I'm going to go ahead and actually remove this drop table if exists main preferred roles cuz that's not a problem anymore. That shouldn't be in our script. Anyway, we need to fill in these preferred roles with a role ID and role name. For this, use the keyword insert into, you list the table name, and then you specify in parentheses the columns you want to fill in and the particular order it in. And then underneath this, we would list the values and what values we want to put into those specific columns. So, let's go ahead and try this out. I'm going to do an insert into and we want to go into in the staging schema into preferred roles. Now, inside of here, I would list the first column of role ID and then the second column have of role name. Then underneath here, I would insert in values. I'm going to create a new line for this and then each set we want to put inside its own set of parentheses. So, we'll do the job or the role ID of one and we'll put in that first one of just data engineer. Then outside the parentheses, I'll put a comma, I'll enter the next row that we're going to insert into here, and it will be number two of senior data engineer. That's all we're going to insert for right now. I'll go ahead and close this with the semicolon. Now, let's go ahead and run this. Okay, no issues. Let's actually query this table to make sure that it got inserted into there and I'll do a select star from staging. preferred_roles. Running this, we can see we have inside of here that role ID of one and two of data engineer and senior data engineer. Worked. Now, we can insert in more roles. I'm going to just go ahead and copy this for brevity. And let's say in this case we wanted to add a third role of software engineer. Now, real quick, remember I didn't add it up here and that's because remember if we try to run this with these roles already made, well, whenever we run this, yeah, we get no errors and if we do this, look into it, well, we have actually duplicates now. What the heck is going on? Oh, I made an error. It shouldn't have allowed those to do this and I realized what I did now. We never specified that this role ID was a primary key. That was the issue because then it wouldn't prevent the duplicate. So, good thing we're testing this. So, let's go ahead and add this keywordy here of primary key. And that would go after the integer in this case. Now, we've already created this table. Just for simplicity, we're going to go ahead and just drop this table. So, I'm going to type this down here because I don't want to save this cuz this was just sort of a an impromptu thing that we had to do. And this will be drop table staging preferred roles. Okay? And now we're going to recreate this time assigning that primary key. Good. And then querying it to make sure that the table exists. Yep, it exists in there. And then actually querying inside of the preferred roles table. The values didn't get inserted because, well, we haven't inserted values yet, right? So, I'll go ahead and insert them in. And then now checking it. All right, we have those two da- two that are already in there. If I try to re- insert it, I'm going to get an error because we have duplicate key error cuz we already have a one and two and it's checking based on assigning this primary key that we don't have any repeating. So, just rechecking this of querying that preferred roles table. Okay, we have data engineer and senior data engineer. Okay, and now getting to that final point that I was trying to get to. Now, we can add a third role if we wanted to in here. And querying this table, we can see we have all three of these roles inside of here. So, a little bit of a learning experience going through that. Now, I want this script to be, like I said, idempotent. So, what we're going to do I'm going to just add this up here. I know I want these three values. I'm going to take that. And I'm going to insert it right here. Make sure you have that comma for that second line. And then I'm going to go ahead and delete this right here. I'm going to go ahead and save this file, command S or control S. Double check it real quick, make sure everything looks right. Yep. I'm going to copy this up top here. Paste it in. Let's try to run this entire script. Bam. And for our final output, we can see that we have the correct table. So, our script's working great. Now that we've created our table and inserted data into it, let's go through a few alter table commands that we can do. And this is These commands are stacked on one top of the other. Basically, we have alter table and then you specify an additional thing that you want to do to alter the table. Specifically, you can do something like an add column, drop column, rename a column or a table, or you can alter the column itself such as the data type it is. So, for our table, we want to add an additional column. And this column is going to be called preferred roles. Basically, we want to give it a boolean value of true or false if it is preferred. In our case, data engineers and senior data engineers preferred, then this will be true. Software

### [6:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=24900s) Segment 84 (415:00 - 420:00)

engineers, not so preferred, we're going to give it false. So, the syntax for this is we list first alter table. We specify the name of the table, so staging. preferred_roles. And then underneath this, I'm going to do the next part of the command of something like add column. In this case, we want it to be preferred_roles. And note for this column name, it's preferred role, not preferred_roles like the name of the table. It's very similar. We're going to end up changing both of these, but the just for demonstration purposes. Now, anytime you create a column just like we did above whenever we did that create table, we need to specify the data type. In our case, we want it to be boolean. So, inside of here, I'll add boolean. Let's go ahead and run this. So, we added our column. We're going to now run our query to check it, and we can see that our preferred role column is created and that there are null values or no values inside of it. Now, along with add column, I'm going to go ahead and just copy this right here, paste it underneath. We also have drop column, and it's just changing add to drop. And I can remove that Boolean value cuz I don't need to specify that when I'm dropping the column. Now, whenever I run that to drop the column, querying the database, I can see that it's gone. But we want it in there, right? So, I'm going to go ahead and add it back. Check to make sure that it went into there, and then from there remove this because we don't want to repeat this in our script. Now that we've created this new column, we need to add data to the column. Now, insert is a row-level addition. Update is a row-level modification, so we can pinpoint the new column and add data specifically to the null values in that new column. Now, it needs a where condition, or it will update all the rows particular to whatever column we're calling out for this. This is best for corrections and fixes, and not inserting an entire row. For this, we need to use at minimum three keywords. The first one is update to say what the table name we want to update is. Next is set, and it defines the new value that we're going to be putting or inserting into here or updating in here. And then finally, the where condition. What do we need to meet for that particular row that we want to modify? So, we'll start with that update command, specifying our table of preferred roles with an S. For this, we'll next use set, and we're going to be setting that preferred role. Specifically, remember we want to set data engineer and senior data engineer to true and software engineer to false. So, I'll specify in here that we want the preferred role to be true where role ID equals to one or role ID equal to two. Okay, let's go ahead and run this, and it looks like it we didn't have any errors. So now I'm going to go ahead and query the table to make sure they got inserted in. All right, we got two true values in there. For our software engineer one, I'm going to go ahead and just copy this cuz I'm a little lazy. Change the role ID to three and remove the rest of this and then also update this instead of true to be false. Running this query, I've got no errors and now querying the table, we should have it all filled in. Yep, true false. Nice. Now one last note on this, we just went through insert and update. We will in the third lesson cover merge and that's going to combine both insert and update all within one keyword command. All right, last portion of this, let's wrap this up. We're going to be finishing up with the alter table. Specifically, we're going to be covering how to rename a table and a column and then also how to alter a column. Now as I hinted towards before, the preferred roles, that's not the preferable name of this. Also, I'm not a fan of this preferred roles column that we're using. I want to change not only the name of the table but also the name of the column and what's inside that column. What I want it to be instead is a priority level. So instead of true or false, I want to like rank it on a scale of one to three where one is the highest priority and three is the lowest priority. So let's first start by changing the name of this from preferred roles to priority roles for the table. So we're going to use that alter table command and specify the table of preferred roles. In this, we're going to rename to, that's the keyword. So we're renaming to what we want to now name it. In this case, we don't need to specify the schema. We can just specify what we want the name of the table to be and we're changing this to priority roles. So running this, it looks like it runs quite correctly. If I try to query our previous preferred roles, I'm going to get an error saying, "Hey, that doesn't exist anymore. " So, I'm going to go ahead and copy this and then put this

### [7:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=25200s) Segment 85 (420:00 - 425:00)

underneath right here and then change this to be priority roles instead. Let's go ahead and query this and we get it back. Okay, we rechanged the name of the table. Now, let's rename column. For this, we're still using that alter table command as the top one, specifying now that staging. priorityroles cuz we renamed the table, and then we're going to use the keyword of rename column, and we need to specify the old column first of preferred role, and we're changing this to priority level. I like this better, too, this rename because now priority roles and priority levels, they're not as close as was preferred role and preferred roles, the name of the table. Super confusing. Anyway, let's go ahead and put a semicolon in, run this bad boy, and then query the table to see if it's updated. Yep, priority level is updated. So, we got the table changed, column changed. Let's actually change the values in here. Now, we're going to be using the alter column command for this. In addition, well, alter table, alter column. And you can do multiple different things in altering a column. We can specify to either change the type, such as if we want to change the data type, we can specify what we want to set default values to whenever we add new roles, so you don't have to necessarily automatically set it to null. We can also drop default values from a particular column, and we can also add or drop specific constraints of whether it's null or not. The last two are really beyond this the scope of this. We're just going to focus on changing the type of data that's inside of that column. Specifically, right, we want to be a priority level, and it's going to be either one, two, or three. So, that's an integer. So, let's start by typing this out. We're going to start with alter table, and we're doing this on the staging. priorityrules table. And we're going to be altering the column. We need to specify the column, which is priority level, and then what we're doing to it. We're changing the type. So, we're going to put type in now as the keyword, and we need to specify the new data type for this. Now, anytime you select the data type you're going to, you have to be very specific. Not all data types can be transferred to other data types. In our case, boolean, which is true or false, can be transferred to a numeric value. Yes, it has true or false in here, but behind the scenes it's encoded as a zero or a one. So, transferring it over to an integer, so I'm going to type in integer, is not going to be a problem. We'll actually see what it gets. So, if I run this, I've transferred it to an integer, and now whenever querying that table, we can see inside of here the trues were replaced with a one, and the false was replaced with a zero. But, going back to that point I was trying to make, there are some things you can't transfer to, such as if it was a character value, you can't transfer it to numeric. For example, this is just an example, you don't have to necessarily run this. I'm going to paste this underneath here. We're going to do the priorities role. Let's say I wanted to change the role name, which is of the type varchar, to an integer. If I tried to run this, I'm going to get a conversion error. Could not convert string to type int32. So, there's some data types you can't do this to. Anyway, we need to now go in and fill out what our priorities levels are for this. Honestly, data engineer and senior data engineer, I was going to make a priority one, so they're for perfectly fine the way they are. However, role three I need to update. So, I'll use the update command on staging. priorityroles. We're going to be setting priority level equal to three, and we need to live a where condition of the role ID equal to three. Okay, let's go ahead and run this. Looks like no issues. I'm going to take this query that we have right here and I'm going to insert it at the very end and now run this. Boom! It's updated. All right, so not bad at all. Just taking a roll scroll through this script. It's looking like it's working pretty good. Let's actually I'm going to save this by pressing command S, control S if you're on Windows. And then from there taking this command and running the entire creation script seeing if we have any errors. And inspecting this, it goes through, tells us the schemas, the tables, everything like that and gives us our final updated table. No error message. This is an item potent script. So this was a little bit of a longer lesson, but I don't feel any of the keywords and commands that we went over really that intense or tough. So we now have some practice problems for you to go through and get familiar with all of these different keywords. In the next lesson, we're going to be jumping into using these DML and DDL languages to create different types of tables and also get into some more scary stuff of how you can delete or truncate tables. With that, I'll see you in the next one. All right, welcome to part two of three

### [7:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=25500s) Segment 86 (425:00 - 430:00)

where we're going into DDL and DDML. This one's going to be heavy on the DDL. In the first half, we're going to go over three very common commands used to create different types of tables. We're going to go over CTAs or create tables as and you can provide a query after that and create whatever you want. A view, which technically not a table, it's a stored query definition, and then finally a temporary table, something that only exists for a session. Now besides just table creation, we're also going to be going into how to remove rows from a table. We're going over the delete keyword to just remove one or more rows, or truncate to remove all rows. Just a quick refresher on DDL or data definition language and DML, data manipulation language. The majority of what we're covering today or in this lesson is DDL and it defines and modifies the database structure. Specifically, we're going to be creating different types of tables and this affects the schema. Now, we do have one command today that is DML or data manipulation language and this manages and manipulates data within the tables. And delete is a is this because it actually affects records in a table. Delete is the opposite of insert in that it can delete rows based on a condition. Now, there's one other way that I like to think about how to separate these two. It's in their usage frequency. So, DDL ones that you use to, you know, affect the schema, create tables, create databases, that's used during database setup, whereas DML is more used on a daily basis with working and interacting with data to insert or maybe remove rows. Now, for this first section, we're going to be doing or creating CTAs, a view, and a temporary table. But, before we actually dive into that, like what are we going to actually creating inside of our data mart? Well, for this, we've already created inside of our data mart, we've created that main schema and the staging schema along with priority roles table in our staging schema. The first example, we're going to be using CTAs to create the job postings flat table, which is just a flat version of our job postings, the list of job postings and the company associated with it. From there, we're going to use that table combined with our priority roles to create a view inside of our main schema called our priority jobs flat view. And then finally, we're going to wrap up this table section by creating a temporary table where we're only looking at the senior jobs and we're going to create a senior jobs flat temporary table. Anyway, you don't need to memorize this. I'm going to remind you. But first, let's get a general idea about these three different objects that we can create. CTAs means create table as select. This is a physical table that we create inside of a schema. Because we actually create it, it allows for really fast reads. Next up are views and these aren't technically tables. Instead, what it is a virtual table. It's a query and then whenever a user goes to this view, the query executes and then provides this virtual table. No data is stored with this and if you use this in a query, it's more likely not going to be slower. And then finally, we have a temp table. This is just materialized data that is stored in a table but only for you the session that you're logged into. So, as soon as you log out of Mother Duck and you're no longer in there, temporary table doesn't exist anymore. All of these have a very simple syntax that follows the same pattern where we're going to be writing something like create or replace table and then the absolute name that you're going to give it and then from there as. And then following this as, usually we include some sort of DQL language that queries and provides what we want to put into that table view or temp table. All right, so let's jump in and start coding this CTAs. For this, I've created a new file of 1. 22 DDL DML part two SQL file. Similar to the last lesson, we're going to be building it out on here and we're going to be creating it to be item potent so that way we can execute the script as a whole. And we could rerun it over again, and it doesn't cause errors to happen. First thing I need to do is connect to MotherDuck, and I'm going to connect right to our jobs mart. So, for this example, we're going to be creating a job postings flat table. What this is going to do is it's just going to combine our job postings fact table with our company's dim table, put it into one table. We're not going to join the skills for now. So, the first thing I'm going to do because it's a create table as select, I'm going to first just build the query, make sure it's correct, and

### [7:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=25800s) Segment 87 (430:00 - 435:00)

then use it to create the table. So, with our select statement, I'm going to just select all for right now, well, specifically from our job postings fact table and from our company dim table. We're going to fine-tune it down after this. And so, I'm going to do from job postings fact as JPF, and we're going to do a left join because we want to maintain all of those job postings in that job postings fact table to the company dim table as CD. We're going to do this on the company ID of both tables. All right, let's just go ahead and run this and see well Well, you know me, I like to make sure we limit this first. I'm going to just limit it to 10 values and going ahead and run this. Got an error in here. I forgot to put a comma after our first portion in the select. We'll try to run it again. Oh, silly me. It says, "Hey, table with name job postings fact does not exist. Did you mean data jobs? " I did. Remember, when we logged into DuckDB, we specified that jobs mart. So, anytime we're connecting to anything, we need to specify Well, anything outside that database, it. So, on both these, I'm going to add data jobs. All right. Let's hope there's no more embarrassments with this. All right, not bad. I want to see what columns are going on here. As we learned previously, we can just do This is DuckDB specific, but I can run describe above this, and then running this, I can see all the different column names in here. Now, I want to pick out specifically what company name or what column names I want to use in this. And conveniently, it's in this list right here, so I can copy this and then just read that read uh reuse this as necessary. Now, I could paste this inside of here and then try to clean it up. This is a mess. This is honestly a job for AI, and I can highly encourage you to use it, especially for this kind of case. I can pop open the AI pane over on the right-hand side, and I can see then here that it has the file as context, including lines 1 through 27. I can say, "Hey, clean up this query. " And it look like it fixed it. I'm going to close out of this chat so we can see what it we did. And it has uh red for what it removed and green for what it included, and it did all those different column names that I want to do. This is really good. I'm going to go ahead and keep all this. Now, if you don't want to log in to GitHub in order to use that, probably fine. Feel free to just use ChatGPT. Anyway, that's cleaned up, and now I can clean up this list of what's going on here. Specifically, I'm going to remove this company ID, along with down here for company, I'm going to remove the link, link Google, and the thumbnail. Everything else can stay. But, the main point is we wanted to join in to get that company name in there. So, let's go ahead and make sure that this query works. I ran shift enter. And it looks like it's working well. Okay, I'm liking it. So, now that we have this operating query, we can use this to create a CTAs. So, I'm going to remove this limit 10 on the end here first, and then up at the top where they the AI input this comment, I'm going to do create table, and then we're going to give it the name, and we're going to put this into the staging schema. We're going to give it the name job postings flat. And we're going to do this as this entire query underneath here. Now, it's a very important, right? Remember, we, whenever we logged in, I logged into the Motherduck connected to the job smart, so I know I'm connected to that. So, whenever I go to run this query, it's going to insert it into there. Right now, it's running through the query. It took about 4 seconds in Motherduck to do. Now, it's done. So, now let's query it to make sure that it's there. Remember, we need to specify it using select star from the staging job postings flat, and I'm just going to limit it to 10 values. Running this bad boy. Boom, it's all there. And something that [snorts] I actually should have checked before is what is the count of this, probably the most important thing. Make sure we got it all, 1. 62 million rows in there. So, running this, whoop, it's in there. 1. 62 million rows. Now that we created the CTAs, we're going to move on to creating a view. Once again, not a physical table, it's a virtual table, and it's going to be a query stored, so that way other users can go to it and see a fresh set of data because whenever they query it, it's in real time pulling whatever data is in those associated tables. That's like the one key benefit of views is that it's always updating, whereas if you use CTAs to create a table, whatever that snapshot is that you use of the table, that's locked in place of whatever that time it was. Now, for this, we listed previously, we created previously that pro- priority roles table, and now we just created that job postings flat table. We're going to combine these two

### [7:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=26100s) Segment 88 (435:00 - 440:00)

to create a priority jobs flat view, and we're going to put this inside of our main schema. So, basically, it's our job postings data warehouse that is filtered for job postings that the business defined as priority in the priority rules table. Now, similar to last time, we need to build up the query first that we're going to then be putting into the view. So, for this, we're going to select right now, we're just going to select star, select all, and we're going to be coming from the staging of that's job postings flat. We're going to give this the alias of JPF. Slightly confusing with job postings fact. And we're going to join this to our priority rules in staging, and we'll give it the alias of just R. And we're going to do this on that job title short column, cuz technically, we don't have all the different job IDs in our priority rules. So, we're going to do that job title short and match it up to the priority rules role name. Now, let's go ahead and execute this. We'd have to fine-tune this query a little bit more, and came back, and here's what we need to fine-tune. Mainly, I don't really care about including well, the role anything from the priority rules table. So, role ID, role name, or priority level. Because of that, I'm going to just specify JPF. star. Now, we do have one more condition, and that's that we only want those with a priority level of one, only the most important data. So, we're going to put a where condition on this. So, we'll add in a where priority level is equal to one. All right, let's go ahead and run this bad boy. And it looks like we got about 480,000 rows back, not too bad. And it looks like also all the columns are filtered as necessary. All right, now we can go ahead and build out our view. So, above this select statement, we're going to go ahead and do create view, and then we need to give it a name, with the name staging. priority_jobs_flat_view. And then we always need to leave that or as at the end. All right, now let's go ahead and create this view. Run shift enter. All right, and like before, we need to query it, so I'm going to copy this query above here that we queried for the other table, and then change the table name to priority_jobs_flat_view. I'm going to go ahead and run this. Okay, looking good. There's less counts in there, but that's not the actual verification I want to do. Let's take this query of this view a step further. Specifically, I want to filter this for the job title short column, and then with each one of these counts, we can label this as the job count. Now, because we did an aggregation, we need to do a group by, specifying that job title short column, and also order it. And that's by the job count in descending order. Okay, let's go ahead and run this. Make sure it's running correctly. All right, good. So, we have senior data engineers and data engineers. The priority level ones in there, and those values for the amount of jobs that are associated with it matches to what I expect. Now, I'm going to go ahead and save this. Remember, we wanted to create this script to be idempotent, so that way we could run this again. But, if we were to try to run this again by doing that dot read command and then putting in the name of where this is located, I'm going to right click this and go to copy relative path, and then paste that into here. Now, if we were to do this, we're going to get an error. two errors with this, and that's because when we went to create it, it says, "Hey, table with the name job posting flat already exists, and the priority flat view job flat view already exist. " Now, for both of these, they have an optional parameter we can add where after the create statement, you can say or replace. Remember in when we did create or drop a database, we could have if exist or if not exist, well this is very similar. In that you can either create this new object or replace it if you're running this again. So, for our first query, I'll go ahead and put or replace in here. And then for our view, I'll do the same thing and do replace or replace. Now, saving this and now running this again, this takes a little bit more cuz now it has to create every single time. Created the table at 3. 44 seconds. We see the count of the table and then we visualize that view. All right, so now moving into the last table that we can create is a temp table or temporary table. This is a session scope table, so as soon as we log out of Motherduck, it's going to disappear forever. It's really just should be used if you need to basically have some complex query that you need to get some data and you just want to stick with that data to further manipulate more. I do it all the time when I'm running analytics on a table and I have this complex query that I need to go a little bit further, well I can save it as a temp table and then do analytics on that

### [7:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=26400s) Segment 89 (440:00 - 445:00)

temp table. So, for our example, we're going to create a senior jobs flat temp table, which is just our senior data engineering role inside of a temporary table that is aggregated or has all the results of the job postings flat. So, let's first start by just creating our query. For this, we're going to be doing select star of all of our data. And this one, since we already filter down for our priority jobs flat view of just the priority jobs, I'm going to use that view. So, I'm going to say from and then I'm going to paste it into here the staging priority jobs flat view and then it's going to be where that job title short equals to senior data engineer. I know this query isn't that complex, but use your imagination here in that you may have even more complex queries that you want to save into a table to use. All right, let's just run this bad boy to see if we get the correct results, and looks like I have some errors in here. And had a little bit of issues. I was using double quotes around senior data engineer when I should have been using senior single quotes. Anyway, this was throwing issues. Changed it to single quote and it's working just fine now. When you use double quotes in DuckDB, it treats it like column or table names. So, you don't want to confuse it. So, anytime it's text values, use single quotes. Anyway, this is populating just fine. We want to now create this as a temporary table. So, we're going to do create, and you can do temp table, but it's more acceptable in more databases to just do temporary. I'll type it in all caps. Now, this one we're not going to specify a schema because it doesn't live in any sort of schema. Actually, you know what? Just to show it, I'm going to put staging in here and put an as on here. If I try to run this with staging on there, it's going to give me an error. Actually, it tells me that schema doesn't exist, but that's not the real issue here. The real issue is we don't list a schema for temporary tables cuz it doesn't go into any schema. So, now running this, boom. Now, that's available. Now, what I can do, I'm going to copy this query above and I'm going to take that name of that temporary table cuz that's what we want to use instead for the from and I'll replace that in here. So, now whenever I run this, I should only see senior data engineers, which I do, and it's the correct count. So, real quick refresher on this, CTAs or create tables as select store rows. Views don't, they store the query, and a temporary does table does store the rows, but this temporary is only for that session. CTAs and views can be viewed by everyone, whereas the temp table is for you and only in your session. Views are really great because they are refreshed by default in that whenever they run the view, it automatically runs the query and connects to the appropriate tables and queries it. However, that also makes it slower. Compared to something like a CTAs that is not fresh by default, but hopefully your tables are built in a way that they are maintaining that freshness into it. And this is the same for temp tables. So, here's how I like to think about it. If you need the latest data for every query, use a view. If you need fast reads and stable results, you CTAs. If you're just testing or debugging, then use a temp table. In the last half of this, we're going to cover two commands of delete and truncate. Delete is I like to think of it the opposite of insert in that it removes specific rows. This can be slower on bigger tables, but it's best for whenever we want to do targeted deletes within a table. The other command is truncate, and this would be in a case where we want to keep that schema of the data of the table and then maybe refresh it with some new data. This is a little bit of a faster way to empty a table. Now, I put this also along drop table because drop table is another way to delete data and that is just removing the schema and table and data all together. So, you have three different options depending on what you need to do. The syntax for this is pretty simple. For delete at least, it's delete from, you list the table name, and then where as the condition. What condition are we trying to meet with this? Well, the business unit came back to us and they told us inside of our job mart, they only want data from 2024 and beyond. Anything before 2024, they don't want to have inside of their job mart. Not my choice, it was their choice. So, for this we need to modify our source table or that staging table of job postings flat that we brought in and we just want to remove that from any values before 2024. Now, it's important to monitor what's going

### [7:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=26700s) Segment 90 (445:00 - 450:00)

on before and after this. So, I'm going to run some queries on looking at the row count for each of the three tables that we just made. Specifically, we're going to look at the count from that job postings flat table, which is the one we're going to alter, also from our priority jobs flat view, and then finally from our senior jobs flat temp. I'm going to put semicolons all this. Let's go ahead and run these, make sure they're running correctly. All right, it looks like they are and that we have 1. 62 million in our fact table or the flat table, 483,000 in our view, and then finally 91,000 in our temporary table that's only senior roles. Let's get into deleting this data. So, we're going to do delete from and that's the staging job postings flat table. And that is where the job posted date is less than put in parentheses, single parentheses, January 1st, 2024. And I'll put a semicolon after this. Okay, and then after this I want to run this as well. I'm going to just put this here. Now, anytime we're about to run a delete, need double-check it. Looks like it's correct. All right, I'm going to run all of this at once. So, with this initially we had 1. 62 million in that fact table or in that flat table, and then going down after this it went to 828,000 jobs. With our view, we had originally 483,000, after this right of view is a fresh snapshot, so it went down to 251. Now, however, for our temp table right here, right? We started 900 91,000 and we ended at 91,000 because it's a temporary table and we didn't update it. Now, let's get into truncate. For this, if the business unit came back to us in one of those jobs that were, you know, more fresh, that were 2024 and greater, but they had concerns that maybe some of the job postings in the original data warehouse had updated, this would be a good use case of truncate in that we can wipe the entire table of this and then rebuild it back up with the contents because we know the schema is going to remain the same. This all depends on delete or truncate depends on how many rows you're moving, how long this is taken, is the data upstream fresh or not. You have to take this all into account. It's really going to depend on the scenario you're in. Anyway, let's demo this. To do that, we need to get back in that 2024 data and pretend that it's all it's in there. So, what I'm going to do is I'm going to just come up to the top and run that CTAs that we used to create that job postings flat table. It loaded, took about 4 seconds, and now running the count on it, I can see that it's 1. 62 million. Okay, we have a good starting point for this. Now, for this, the syntax is pretty simple of truncate table in that you write truncate table and then the name of the table, so job postings flat. Now, whenever you run this, this is going to take everything out of the table. Let's just actually demo it. I'm going to go ahead and run this, shift enter, and now whenever we query this staging. jobpostingsflat table, there is Let's make this a little bigger. There's zero rows inside of it. So, now we need to fill it in. And we know the structure is there cuz if I do a select star from that job postings flat, all of the columns are still in there. So, we just need to insert into it. So, for this, we're going to use the insert into command. We're going into that job postings flat table. Now, previously whenever we did this in the last lesson, we went through and we specified all the different columns and then the values that went into it. And especially and like I said, these names of these columns are can be optional, but you have to make sure that they're lined up properly. Anyway, we're going to be a little lazy with this and we're going to insert into this entire select statement that we used for the CTAs up above it. So, we'll go ahead and select this all. I'm going to close this off to the side. Put this into here. And remember why we even wanted to do this from the first place in that we wanted job postings that were more fresh, that were 2024 and greater. So, for this, I'm going to put a where statement filtering our job posted date for those that are greater than or equal to January 1st, 2024. And then after this, I'm going to also post this down below underneath it to make sure that we have everything updated. All right, let's go ahead and run this. Up from truncate table, let's find a run truncate table again. And boom, we have that job postings flat updated that 828,000 like we expected. That view back to 251,000 and like usual, the temp table not affected by this. Now, delete or truncate are also going to depend on how many rows you're going to removing

### [7:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=27000s) Segment 91 (450:00 - 455:00)

or rows you're going to be adding. If you're removing almost all the rows, you're probably going to want to truncate. If you're only removing a few rows, you're probably going to want to delete. Really going to be situation dependent. All right, it's your shot to give it a try now. We got some practice problems for you to go through creating different tables, views using delete and truncate. We only have one more lesson next on DDL and DML and it's going to get even more spicy from here. All right, with that, I'll see you in the next one. All right, in this lesson we're going to be going over subqueries and CTEs or common table expressions. For this, we're going to do a brief intro to explain what those are, subqueries and CTEs, and then do an example of each along with a more complicated final example. And one quick note, we're taking a short little break from our DDL and DML lessons. We have three lessons for that, we've completed two. The third one will be following that. We have a concept in here of CTEs that we need to master before we can actually go to that third part. So, what the heck are subqueries and common table expressions? Well, they're used for organizing and simplifying complex queries. Here in this image, we're showing the syntax differences between a subquery on your left and a CTE on the right. Subqueries, as the name implies, are queries nested inside a larger query. It's subquery. And this is done by putting the query inside of parentheses, and you can also provide it an alias specifying it as in this case data engineer jobs. Common table expressions or CTEs define a temporary result set that you can reference. And is just a fancy way of saying that it's a intermediate table of the rows of data that we're going to use for a query and then get rid of it after. This is the same general concept also behind CTEs as well, is that they're a temporary result set. Anyway, we're going to quickly demonstrate each using a simple example. First thing you need to do is start a new file, and we need to connect to our database in Motherduck. All right, we're connected to that data jobs data warehouse in Motherduck. All right, first up is our subquery. Remember, that's a query nested inside a larger query. For this, we're going to do a very simple example. We just want to filter down our job postings fact table to only provide job postings with salary data. So, we're going to select star of all the columns, and then in our from clause, we're going to open up parentheses, and this is where our seat our subquery is going to be. In here, we'll do a select star from job postings fact. And we'll filter down so in a second, but I just want to make sure that this query first works. So, I'm going to do a limit of 10, and then I'm going to run this query. All right, so looking good, right? We can see that we did this. Now, we want to inside the subquery filter to where we can actually see values for salary average and salary hour average. So, for this inner subquery, we're going to add a where clause, where salary average is not null or salary hour average is not null, either. All right, let's go ahead and run this bad boy, and we get this back. We get all this where we have actually salary data filtered down in our subquery. Now, real quick, this is just a simple demo. Obviously, you probably can just Well, you can. You can just rewrite it with the query inside of it, but I wanted to demonstrate how simple a subquery is. Anyway, let's now rewrite this using a CTE. Now, for a CTE, we define that temporary result set at the beginning of the query using the with keyword. After that, we need to give it an alias, so we'll say for this valid salaries. And then we use as, and then in parentheses, similar to a subquery, we're going to define the query itself. It's the same one from up here up above, so I'm going to go ahead and just copy this all above and paste it in here. Now, we need to actually display this temporary result set, so we can now provide it in the name of that temporary result set of valid salaries. We'll say select star from valid salaries. Put a semicolon in the end. Copy this all, and then go ahead and run this. I didn't do a limit 10 on this one. Probably should have as well. But, looking at the results from this query, we can see that we get it back, and we've actually filtered down to include only job postings with an associate salary. Okay. So, that was a quick demo of CTEs and salaries. One minor note, in that CTEs, I used this valid salaries. I defined this alias of the CTE right here. You can also in the subquery give

### [7:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=27300s) Segment 92 (455:00 - 460:00)

it an alias, such as valid salaries. And then, whenever I go ahead and run this, still provides the same results. Doesn't really change this one in any way, but it's typically good practice to provide an alias, so people understand what's going on. We're now going to dive deeper into subqueries. Previously, we're just demonstrating using the subquery of a temporary result set inside of the from clause, but we can actually use it in multiple different locations, such as the select statement, the from statement, where and also have it. So, we're going to tackle this first one once first one first of using a subquery in the select statement to display what is the overall market median salary based on well, all the job postings, and display it next to each job title short. So, we'll start with the select statement. We'll query that job title short column, and then we'll next provide our subquery in here. We're going to open parentheses. And in here, we're going to select our median salary year average. All right, we're only going to select one column for this. That's the key to making it work. And then, we're going to be doing it from that job postings fact table. For this, we're going to give it an alias of market median salary. And now getting back into that outer query, where is the outer query from? Well, it's from that same table of job postings fact. Right now, we're going to just limit the results to 10. Let's go ahead and run this and see it in here. All right, so pretty cool. We have here the results. Now, this isn't really that helpful. What I do want to display also with this is yes, we have the market median salary. We can see that all of the median salaries are the same as we would expect. So, that's pretty cool. But I want next to this the actual salary of what it is for each one of those job postings. So, we'll add in that salary year average column. And for this, I don't want to see any null values. So, I'm going to add in a where clause of where the salary year average is not null. All right, let's go ahead and run this. All right, not too bad. This is pretty cool cuz we now can see next to each one of these job postings and their correlated salary for that posting, what the median salary is for the entire data set. Now, let's move into scenario two, where we're going to build on this query further in that now we want to only stage jobs that are remote before aggregating it. So, we're basically going to use a subquery in the from clause to do this. So, I'm going to start with that previous query that we had and go ahead and paste it into here. And sorry, I think I should have been more clear. Stage only jobs that are remote before aggregating and that's to determine the remote median salary per job. So, instead of just looking at that salary average column, we're going to instead now do a median. And since we're doing this aggregation, right, we need to now do a group by. And we're doing that by that job title short. This market median salary is technically an aggregation, so we don't need to do it for that. So, I'm going to go ahead and run this. All right, so So, based on the query that we placed in, we're getting based on the job title short column, we're getting their associated median salaries. And next to it, we have the market median salary. This is even better at what I would probably want compared to that last query. Also, I noticed here I actually I didn't alias that median salary, so we need to do that. So, I'll add an alias of median salary. And also, we need to get back to that main thing that we wanted to do of adding a subquery in our from to ultimately to determine that remote median salary per job. So, we're going to replace this job postings fact with parenthesis uh parentheses to put our subquery in. And I'm going to minimize this so we can actually see this all. We need to make sure that we query all the relevant columns that we need to provide to this upper query up here. So, you could do a select star, but I don't really like that. We're going to do job title short and that salary or average is that's better practice to name the specific columns. And we're going to be doing this from the job postings fact table. And we want to do it where the jobs are remote, so we're going to do the job work from home has a true value. We're going to give this the name of as clean jobs. But, as you can see, this alias isn't really getting used anywhere else in here. All right, this should be working as is. Let's go ahead and run this and then taking a look at the query results, looking good. They are different, right? Cuz initially senior data engineers was 147,500, but now senior data engineers are 145 uh well, thousand. I don't know why I said hundred uh previously. But, look at this. This market median salary is still 116950. And when I scroll up here, it's 116950. So, let's go back to that query. And the problem is this. Is this why I wanted to

### [7:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=27600s) Segment 93 (460:00 - 465:00)

do this example where we combine subqueries? This query, this subquery inside of here that we're using the select statement, is its own query and it's not affected by any of the filters for the outer query. So, if I want this query to also provide the market, or in our case the market remote median salary, well, I'd have to add a where clause in here as well for where a job work from home is equal to true. Okay, let's go ahead and run this. See, now our market remote median salary is 130,000. Where previously it was that 116950. So, the market remote median salary is slightly higher. Pretty interesting. Now, one quick note on this original query that we have here, we're doing aggregations of Well, we're doing aggregations inside here. So, this whole where of salary or average is not null is not necessary anymore. I'm going to go ahead and remove this. And then if I go ahead and check it by running it, I'm still getting those same results that I got above. So, anytime you can simplify queries, do it. So, now let's get into the third scenario in where we want to use a subquery in our having clause. And now for this, we're going to use the having clause to filter to ensure we keep only job titles where the median salary is above the overall median. This will be for all that remote jobs that we did previously. This is going to be building on our previous query. So, I'm going to just go ahead and copy this from above, paste it down in here. And then as we know, the having clause goes after that group by. And we're going to be using this for to make sure that we're above the market median or the market median remote salary. So, we're going to run the median function again on salary average. And we want those results provided to be greater than what's inside of our sub query. And conveniently, we provided it up here as the market remote uh market remote median salary. So, what I'm going to do is I'm going to just go ahead and copy this all, command C, and then paste it in. All right, let's go ahead and operate on this to make sure we're operating correctly. Okay, we got the results. And as we can see now, that market remote median salary was 130,000 and only results greater than that are being shown now. So, not bad. We've demonstrated how to use a sub query in the select statement, from, and also having. We'll go into it the final example, how to use a sub query within a where clause. Now, I do want to point out something real quick with all of these different scenarios that we did. Whenever you're building these out, I like to sometimes even start with the base query or that sub query, if you will, first. So, if I were to just run this first one, right, where we were going through and we wanted to get the market median salary within the select statement, I can just run it by doing shift enter. And then it puts it down into the terminal, add a semicolon, and then run it. Anyway, it provides only one value as a result, this 116950. And that is what is being put into this entire query, which I'm going to go ahead and select it all, and then run it. That's what's getting put into here. So, any of these queries, if you'd like to dive into more about thinking about how to build it, it's as simple as making it into a state that is being uh that's able to be used. And what do I mean by that? Well, for that select statement, well, moving into scenario three, remember we only had one value for that select statement because we just want to put one value per row. Whereas for our from, right, We had an entire table with the twitch of I run this, right? Adding a semicolon onto this and running it. This output more of a table or a multiple values that you would expect to use as a table in a from statement. And then the having statement in this case was what we used previously, right? Whenever I go to run this, putting a semicolon on the end, it's just a single value. So, that's what I mean. It has to be in the required state for example, let's actually make an error in here. Let's say for this select statement, right? We had it needs to be a single value to be put into the column. If I were to get rid of this aggregation function and basically provided a list of values or so when we're going to run this and output the results below. When I run this, right? It's a it's multiple values that it had that will be providing this query with in the select statement. So, if we tried to run this in this form within here, it's going to give me an invalid input error because

### [7:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=27900s) Segment 94 (465:00 - 470:00)

as it explains, more than one row turned by subquery you're used as an expression. Scalar subqueries can only return a single row. They have an option down here to change the behavior by setting a certain variable some we're not doing that. Basically, you need to just write the proper query for what our intended uses. So, I'm going to go ahead and add median back from here and hope you take a learning lesson out of that. I sort of glossed over checking these, but anytime you're building these subqueries, feel free to execute it first and then put it within the main query. All right, let's move into CTEs now or common table expressions. For this is a similar in that we're creating a temporary result set in parentheses, but it's defined above the query that we're going to use it with a well, with the with keyword defining the alias and then what's next in parentheses. Now, a few short notes on this. Similar to the last subquery, it's used in a lot of different places, right? So, so can CTEs. What is defined as the alias? We can use it in a from, in joins, you other CTEs and that's actually pretty common as you can have multiple CTEs on stacked on top of each other. And then you can use it inside any main statement such as select, insert, update, or delete. It's used with that with keyword at the beginning to define that CTE and it exists only during the execution of the query. I don't think I mentioned it with subqueries, but you can also use subqueries, the CTEs. You can use subqueries, you can use join uh with a subquery as well. So, moving into our CTE example. For this, we're going to compare how much more or less remote paying roles compared to on-site role on-site roles for each job title. So, for this, we're going to use a CTE to first create this temporary result set where we have a job title, median salary, and whether they are job work from home or not. And that's for all the different job postings. Then with those CTEs, we're going to be able to make it into a table that compares one line by line how it looks per job title. Well, let me just show you the solution. In this, we have the job title short column that we aggregated it by and then we're going to have the remote median salary right next to the on-site median salary. And then because they're on the same line, we can also get what is the difference between the two that we're going to call the remote premium. And this is all possible, we can do this because of CTEs. So, we're going to be building a CTE, but I'd like to typically just write the query that's going to go into the CTE and then wrap it in a CTE. So, remember, we need three different columns. We need the job title short, job work from home, and then that median salary. We're going to give it the alias of median salary. This is all from that job postings fact table. And then because we did aggregation, we need to do a group by of job title short and of job work from home. I'm going to put a semicolon on this. Let's go ahead and execute this. Make sure it executes. And it does. And notice for this, right, it's aggregated, so it has each of the job title shorts, and then it has for a false value and the true value the market median salary. Now, I'm going to modify this question slightly. I'd like to narrow it down to a region. So, I'm going to change this or I'm going to add on to this a where statement. In that specifying that the job country is specific to me of the United States. You can also modify it to your home country as well. And then going and doing this one. All right, not too bad. This is looking good. Comparing it to that table above, I can see there are some slight differences with this. It looks like, well, for the United States, it's slightly higher. Anyway, one thing I don't like about this is this has some crazy Look at I mean, look at that long decimal place. I'm not a fan of that. So, what I'm going to do is I'm actually going to cast this market median salary as an integer to just make this a little bit easier to read. And then after running this, I can see, okay, everything's cleaned up. It's looking a lot more readable. I don't need down to the decimal place with a yearly salary. All right, so going back to that table that we're working towards getting. We now need to wrap what we just did into a CTE, so we can get this displayed in this manner where we have the remote median salary right next to the on-site median salary and it's broken down by job title for each line. And then what we'll do after that we'll focus on that remote premium. So I'm going to do first I'm going to indent this over by highlighting it all and pressing tab and then giving a with keyword. I'm going to call this the tidy title median. We'll do as and then I'll open parentheses and then we'll wrap this all within parentheses. Now I'm not going to put a semicolon on the end of this cuz I want the query to continue. All right, so the first thing I'm going to do just to make sure that I'm

### [7:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=28200s) Segment 95 (470:00 - 475:00)

building correctly, I'm going to just do a select and we're going to look at the job title short and the median salary. This is going to be from our CTE which that title median. Okay, I'm going to put a semicolon on the end of this. All right, let's go ahead and run this all. And it has a I have a parser syntax error at or near that parentheses. I never took off that semicolon on the end of here as I should have. You can't have a semicolon in that CTE cuz then it tries to end it. All right, let's try it again. I also have an issue. Goodness gracious, I have a lot of issues today. I named this incorrectly. Specifically I named it the market median salary. I'm actually not sure why I named it market median salary. It should just be median salary. So that was a little bit of a goof up. Um okay, let's try to run it again. All right, we got it. Okay, so this good, but remember we want to aggregate this or just have, you know, let's do first we're going to focus on just remote median salaries. How are we going to do this? Well, here's what we're going to do. do instead is we're going to provide a where clause and where job work from home is equal to true, so the remote. Now, what we're going to do is you follow along with me for a second. We're going to assign the alias for this table of title median as R, basically remote, short for remote. Okay, I'm going to define this as R dot. This is R dot and then this as R dot because this is all alias here. If I go ahead and run this query, I'm going to have an error because I misspelled where. But when I run it, it runs just fine and it's now filtering just for those remote jobs. Well, how can we bring in now those on-site jobs? Well, we can do a join. So, what I'll do underneath the from is add an inner join and we'll add join on that same table, but we'll give it in this case a different alias of O for on-site and we're just going to be matching it up on that job title short column because that's what we have to similar between this the same table, basically. So, we add in that job title short and we want to do it and we need to filter those now that we're bringing in with O because we want to um bring in O. median_salary. So, these median salaries we need to filter that are coming from the O table. So, I'm going to go ahead and put an or in here for O. job_work_from_home equal to false. And I've been really bad about giving aliases here. I need to give aliases. Specifically, this is the remote median salary and this is the on-site median salary. All right, so now we have both of these defined now of what this is. We're filtering the appropriate table. Let's go ahead and run this bad boy. Well, we got to run all of it including the CTE. And I have a synta- a parse error syntax error at or near R. Basically, I forgot to do specify on for the keyword. I said inner join title median as O and then I need to say on this. All right, let's go ahead and try to rerun this. Okay, getting a little bit of an error here and that's well, no, not an error. Getting an issue with the table, right? We're seeing that these job titles are duplicated multiple different times. Specifically, the problem is with our filtering on remote and on-site. Right now, I messed up that query. I did where the remote jobs are true or the on-site are false. And this basically allows it to do every different combination because everything's going to be true or false. Anyway, I need to change this or to an and. Now, going ahead and re-running this, bam, we got it. All right, so looking good. We now have it in the same table. Now, what we can do is we can calculate that remote premium by subtracting the on-site median salary from the remote median salary. So, let's go ahead and add another line in here, and we're going to do the r. median_salary from the remote table and subtract from the on-site table median salary, and we're going to give this the alias of remote premium. Also, I want to order it by this, so I'll add an order by and add in remote premium and put it in descending order. All right, let's go ahead and do execute this final query. Not too bad. All right, looks like things like machine learning engineers have the biggest premium along with data scientists, data engineers, and senior data scientists. Not too bad. Okay, so this is a great example of how you can use CTEs to manipulate data and basically compare it within a single table. All right, let's move into a final example, and for this, we're going to be

### [7:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=28500s) Segment 96 (475:00 - 480:00)

focusing on subqueries. Specifically, we're going to be using existence filtering in order to use a subquery within a where clause to filter the data in a table that we want when comparing two tables. Now, for this, we're going to be looking at two tables. Previously, when we did joins, we like an A and B table. For this, we're going to use the nomenclature of a source table and then a target table. This is very common in data engineering to have a source table that you're taking data from and a target table that you want to, well, target and get it into. And there's two different existing existence filtering that we're going to go through. One The first one or both of them are one's where exist and that's going to be filtering where the data in the source table is also in the target table. And then we'll also look at, conversely, where not exist, where the data's in the source table but not in the target table. The thing about this is just has it uses the not keyword in front of exist. Not that special. So, let's break this concept down. Let's pretend we have a source table with three job postings in it and a target table with two job postings in it. The source table has a job ID of 101, 102, 103 and the target has 101 and 104. So, we can see that 101 exist in both of these tables. So, how do we get this result via a query? Well, we first query the source table. With this, we're doing a select star from the source table and we want to filter it with a where clause where exist and we put our subquery inside of here. Now, this subquery has a common syntax that you'll see with anybody that uses existence filtering in that it starts with a select one and that really that one value can be any value. It's just going to return a value if, while going through it, if the target table has where a job ID matches up. It's going to go through each one of those job IDs in that target table, compare it to that source table, and if, in the case of 101, it returns a match, the select one will actually return something. Whereas, if there's no match, like in the case of 104, it returns nothing and we don't get anything for 104 in that return statement for the where statement. So, therefore, we just get our return table that has 101 in it. Conversely, we can use something like where not exist to keep the rows with no match in the target. In this case, we see from our source table, we have 102 and 103 in here that do not exist in the target table. So, with this new query of select star from the source table where not exist, that subquery will go through and first in that target table will say, "Hey, it does match up with a 101 job ID in both tables. " But, we're using a where not exist, so it will tell it not to include it in that final query. And then, 104 doesn't have an associated job posting, so therefore, there's no match or anything returned with that. So, our final results table only has 102 and 103. So, I want to just walk through a very simple example using a new function that we haven't seen before, specifically, uh select and then the well, not select, but range. And then, I'm just going to put a number in here of range of 10. I'm going to go ahead and run this. I forgot to add a semicolon. I did. Anyway, what it returns is a list of values, all 10 values, and it doesn't return 10 because the number 10 is exclusive. It's not going to do it, but there are 10 values there. And really, this could be anything. I can make this even range three. Put a semicolon on this time and run it. And this will be 0 1 2 and not 3. Anyway, what I can also do with this, I can manipulate like this a little bit further to make this into a table that we can, you know, is used as a source and a target table. So, I'll say select star right now from range three. I'll go ahead and run this. Okay, now it's providing me a table of 0 1 2 3. I can then give it an alias. I'm going to give this the alias as uh source cuz this will be our source table. However, when I run this, we can see that okay, you don't really see anything uh that the table's named source, but I do want to name this column right here because we're going to do this existence filtering right? We need to be able to compare the values in here. So, what you can do is you then if you want to name the values of a table you've created through something like this function, you put it in parentheses and I'll give it the column name of key. All right, let's go ahead and run this bad boy. All right, we can see it replaced up at the top. Now it is key for that column name. And so we'll say this is our source table and then I'll go ahead and create just copy it right below this and I'm going to name source for this one too. We'll make this our target table and this one will just be of the range two. Go ahead and run this just for safekeeping. Yep, 201 note two. So, let's use existing filters now to keep

### [8:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=28800s) Segment 97 (480:00 - 485:00)

rows with a match that are in the target that were in both the source table and in the target table. For this we're going to use where exist. So, start with I'm going to just copy this query up at the top, paste it down underneath. So, we're selecting star from um the source table and now we want to apply a filter for it, the where filter where exist. Then we're going to start our subquery in parentheses. For this one this one's going to start with select one and we're going to try different values in a little bit. We're going from this target table right here. So, I'm going to just go ahead and copy it right here. Remove that semicolon and we're going to do this where and we had to name those columns right cuz now we're going to compare where the key of the target is equal to source. Okay. Now, what we should see right as we compared previously, this should keep the zero and one from the both tables whenever we go to compare this. So, let's run it. Oops, let me I got to select all of this. Okay, it does. Returns that zero and one. Conversely, we can use where not exist to keep rows with no matches in the target. So, in our case, that source table has 0 1 2, the target has 0 1. Two is the value that doesn't appear in the target. So, if I change this to where not exist, and then go ahead and run this bad boy, two is what is returned. Like I mentioned before, that one in the select one is a common way of sharing what or what symbolizing this is existence filtering. You could do any other number. I could do something like 1 2 3. Whenever I run this, still get the resame same results. Heck, I could even do Data Nerd, or I can even do something like select star. And that works as well. So, doesn't really matter what's in there. One is the simplest query to do, and then when we have that matching key return result. And so, we're using the least amount of data, so I'd use select one and not something like select star. All right, so now that we've gone through that entire thing, let's actually get into the final example of identifying the job postings that have no associated skills before loading them into, let's say, a data mart. This is very common thing to do using this existence filter filtering. Remember from our entity relationship diagram, we have that job postings fact table, which has all those different job postings, and then related through the skills job dim, and then to the skills dim table, they have the associated skills in there. So, if I were to just query it, this is just a simple query. I'm querying the job postings fact table, and then ordered it by the job ID to show the first 10. And we get back this table. Notice that the numbers here, what I'm trying to prove with this is the numbers are all sequential. There's no missing values in between here. For it starts at 4 5 9 3, and in this case just goes to 4 0 6 0 2. Now, also with this, let's then now look at what I want to show is look at the skills job dim table. I also order by job ID. I'm going to go ahead and query it. And the thing I want to show with this that query that's now returned is if there's not an associated or if there's no skills associated with a job ID, they're just going to skip over that job ID. So, if I scroll down looking through this, what I can see is we get to 4601 and then skip over 4602 because there's no skills associated with that and it goes to 4603. Similarly, 4605 also doesn't have any skills. So, things like 4602 and 4605 are jobs that we want to basically capture in jobs that don't have associated skill. So, we're going to use existence filtering specifically where not exist to keep the rows with no match in the target. Where in our case, the source table is our job postings fact table and our target table is our skills job dim table. For this, I'm going to take this query that we used up here and just paste underneath the start with. We'll put a where clause. And for naming sake, I'm going to just keep this as and we normally didn't do in the alias of as JPF, we're going to do target. So, TGT. We're just doing it for this example just to make sure that we wrap our hot hands Yeah. Wrap our minds around target and source. And then we're going to do a where not exist, start our subquery. And for this, we're going to do the select star from skills job dim. So, I'm going to paste this in underneath here, making sure that it's indented over. We're going to give this skills job dim the alias of source. And we're going to be doing this where the target job ID matches up with the sources job ID. Also, so let me I have select star in here. Remember that's not good practice. We're going to do select one. And I'm going to keep that order by in there still cuz I want to be able to find those two jobs that we already identified should not have any associated skills with it. Let's go

### [8:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=29100s) Segment 98 (485:00 - 490:00)

ahead and run this bad boy. And this query ran, it took about 2 seconds to run. And the first two postings on it are the 4602 and 4605, which we previously identified. And it has a total of 285,000. Not bad. Now, remember we have I think it's what, 1. 6 million job postings totally. So, I could, just to double check, I could remove this not from here and also see how many job postings are returned with this. And in that case, [snorts] they have 1. 33 million rows. So, this plus what we got last of about 300,000 rows gets us to that 1. 6 million jobs. Okay, this makes sense and it's checking out that our query was built correctly. All right, you got some practice problems. Now, go through and get more familiar with CTEs, subqueries, and also threw in some existence filterings. With that, we're going to be moving back into DDL and DML part three to do a pretty interesting and fun data pipeline that I'm excited to share with you. All right, with that, I'll see you in the next one. And this lesson, we're in our third and final lesson on DDL and DML. And specifically in this one, we're only going to be covering really one new keyword, and that is of merge. This DML or data manipulation language keyword allows us to wrap up within one statement an ability to perform an insert, update, and delete. Now, before we move into this and actually build the query or the table that we're going to get into, which I'll get into the table in a second, we're going to go through a brief refresher to build out that same query that we're going to do with merge, but instead using just insert, update, and delete. And we'll be doing this using that existence filtering that we learned about in the last lesson. So, what the heck are we even going to be performing this merge operation on, and why the heck do we even need it? Well, let's get into it. Before this, we need to understand that there's two major types of processing. That's batch processing and continuous processing. Batch processing is by far the most common type of processing, so that's what we're going to be going over and using merge for. It's used for incremental loads or ETL or even with DBT models. It's scheduled to run on fixed intervals, processes data as in groups, and only updates or changes new rows. Now, all the hype nowadays you'll hear about this continuous processing, but to be honest, it's not as frequently used. This is for streaming and real-time data pipelines. And this processes data in real time as it arrives. Each record that triggers this new logic, it's event-driven, so every time there's a new record it processes in to go into where it needs to go. And overall, it's built for live systems. And the reason why I say continuous processing isn't that common is as most times we're working in data teams where you don't need to see analytics in a real time. You could have some sort of day delay and it's going to work just fine with management to make their decisions. So, for our problem today, we're going to be focusing on batch processing. So, what the heck are we building? Well, as a quick refresher, remember we have our data jobs data warehouse. And that's the same data warehouse we've been using the same course. Now, over the course of our DDL and DML lessons, we've been building out this job mart. Inside of it, we've had two schemas of a main and staging. And we've built things like temporary tables, regular tables, and even views. Now, we're going to create one last table, and this is going to be built with our merge statement, and that's the priority jobs snapshot table. Now, this priority jobs snapshot table is a combination of multiple tables. Specifically, we're going to use that job postings fact table and company dim table from our original data warehouse, and this is an engineer owned table. We're also going to be using priority roles. Remember, priority roles has what the business designates as a priority and a level of one to three. We're going to combine these tables into one called the priority job snapshot that has key information from our data warehouse along with priority jobs and their priority level. Now, the issue or what we're going to be demonstrating in this is that the business owned priority rules tables is going to be updated by the business. You know, it could be updated as frequently as daily. And so, there's going to be updates to this, and that's going to affect our down stream table of our priority jobs snapshot. So, if they do things like change a priority level of something like software engineer or add a new job like data scientist, we'll need to be able to update that in that priority jobs

### [8:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=29400s) Segment 99 (490:00 - 495:00)

snapshot table. So, what's going to be the data pipelines for these tables that can get changed or updated? Well, for priority roles, we're just going to assume that every single time the business updated, they're updating the entire table all at once. It's a pretty small table, so creating a new table isn't a big deal. Because of this, we're going to assume that the business just has a simple script for creating or replacing a table and inserting into it the values they want to change. If they change those values, they're going to just adjust that simple SQL script. It's pretty simple for them to handle on their own and do. I would assume a business has somebody trained enough in SQL to do this. Now, the priority job snapshot is a little bit more complex, so this is going to be engineering owned. The first thing we're going to do is create a script for an initial load. So, it's going to create our table for that snapshot table, and then therefore load all that data into it that needs to be there. This is only going to be one run once. Now, on a daily basis, we'll then need to go through and perform a batch load. And for our first version of this, we're going to be using what we previously learned of an update, insert, and a delete statement in order to maintain this final snapshots table up to date. We're then going to update this batch loading into our version two that relies on using that merge keyword that will combine all of those different things we've done with update, insert, and delete into one statement alone. And this will do exactly the same thing as our version one. All right. So, the first thing we're going to do is create the SQL script that is going to be used to automatically perform that initial load and update of the priority roles table. I create a new folder of 1. 24, and I'm going to go ahead and add a script inside of here and call it priority roles. sql. I'm just going to go ahead and create this from scratch. You could go back to lesson 1. 22 if you want to. Anyway, we start with create table, and we specify it's in the staging schema of priority roles for the table name. And inside parentheses, I specify the role ID, the role name, and the priority level. Role ID is an integer, role name is a varchar, and priority level is an integer. We'll also establish the role ID as the primary key. Now that we have this table created, we need to insert into it in that staging schema of priority roles those three columns of role ID, role name, and priority level. We're going to insert into it the values, and specifically that of one of data engineer level one, two of senior data engineer of one, and three of software engineer of level three. Now, this is the script we're going to be running. So, at the end of this, I'm going to put a select statement in here to just see what's inside of here anytime we run this script. Okay, I'm going to go ahead and save it by pressing command S. And then I'm going to log into DuckDB, specifically connecting to mother duck, and then going into our jobs mart directly. So, I want to run this script directly. I don't want to have to select it all and then press shift enter. So, we're going to be using for this is the dot command of read, and then we need to specify this file. So, what we can do is we can right-click priority roles, and we can do copy relative path, and then I'll go ahead and paste that in here. And it says it's under the lessons 1. 24 priority role sequel. Yep, seems right. Let's run it. Now, we got an error with it. Let's inspect it. And that's a catalog error. Table with name priority roles already exist. If we go back and inspect our script, right now we didn't design it to be item potent. Instead, we need to add into here create or replace table. So, it knows to do that. Okay, anytime I update my script, I have to press command S. And then opening up the terminal, I can just press up, and it runs that last command. That ran pretty quick. No errors. That fixed it. Our priority roles table is up to date. If we need to make any changes to it, I would expect the business unit, which they have control of this script will say, they can just come in here and they could say, "Hey, update software engineer to one. " Save it, and then inside of terminal, they can rerun it by running it, and in it we can see that software engineer is one. Once again, this is such a small script and small table that I'm not going to do some sort of batch processing to update the table. It's just simpler for me to create or replace it from scratch. Let's get into now initial load of getting that table built for our priority job snapshot table. For this, we need to do something very simpler simple as in we need to first create our table and then from there insert into it the values we want. As a reminder, this takes use of or is built using three other tables, job postings fact, company dim, and that priority roles uh joined in a way that provides us that

### [8:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=29700s) Segment 100 (495:00 - 500:00)

job snapshot. And that priority job snapshot table only includes roles listed in that priority roles table. So, under 1. 24, I'm going to create a new table and call it priority jobs snapshot initial and then name it a SQL file. So, we'll start out with this of a create table statement specifying the main priority jobs snapshot. Now, I would don't expect to run this again, so you don't necessarily need to conclude or replace, but uh for good measure, I'm just going to include it. Inside of here, I'm going to include seven different columns. We're going to use the job ID, job title short, job posted date, and salary average that's from the job postings fact table, the company name, which is from our company dim table, and then the priority level, which is from our priority roles table, and finally an updated at column that has a timestamp of when the row was last updated. We're going to be able to control this updated at timestamp as we go along and build out that update script. Anyway, I want to keep it as simple as possible with the amount of columns. You can include more from the job postings fact, but I want to keep it simple because it's going to make our other columns a little bit or keywords even more robust. Next, we're going to insert into that main priority job snapshot. We're going to specify that job ID, job title short, company name, job posted date, salary average, priority level, and updated at. Now, we're going to build this using a query, so I'm going to begin with a select statement and then we're going to go ahead and add in from what we know we're going to give an alias of job posting fact table for job ID and job title short. Oops, looks like I got a duplicate. Next from the company dim table, which we'll alias as CD, we need to get the name or what we're going to alias as company name. We'll then get that job posted date and salary average from that job postings fact table and then finally from our priority roles table, we're going to get the priority level, which we're going to give the alias for that table of just R. And we have one more attribute we need to apply for this and this is the updated at. For this, we're going to use a formula to fill this in, specifically of current timestamp. Now, if you haven't seen this before, it's actually pretty simple. Inside my terminal, I'm just going to go ahead and type this in and I'll run select current timestamp, put a semicolon at the end and bam, it provides me the current timestamp. So we're going to mark all the rows with the current timestamp and then if we update, insert or delete it, that timestamp will get updated appropriately. Now, where we getting this from? Well, we're getting it from the data jobs data warehouse, where the job postings fact table is. We're going to alias as JPF and we're going to left join from the data jobs data warehouse, that company dim table as CD. And we're going to do this on the company ID of both tables. Now we have one more we got to bring in, right? We need to get that priority level from that priority roles table. Now previously, we just did a left join with job postings facts and company dim because we want to preserve all those job postings that are in that job postings fact table or that company or that table A. However, for our priority roles, we only want to keep those roles that are in the priority roles table. So in this case, we want to do an inner join. So I specify inner join and we're connected to our jobs mart database, so I don't need to specify the table. So, I'll specify the schema of staging dot priority roles. Just as a reminder, we didn't have to put the schema in these up here because they're in the main schema, and main is the default one, so you don't have to specify it. But here, we're using a separate schema, or not the main schema, so that's why we have to specify staging. Okay, we're going to give it the alias of just R. And we're doing this on the job title short of the job posting spec table and that of role name. Going back to that priority roles table, right? We don't have a like a job ID with this, so we're not going to link on the job ID. But the most common attribute is the role name itself. All right, so with this, I'll put a semicolon at the end. Let's go ahead and save this. Oh, and there is one more thing. Anytime I run this initial load, I do want to see what is on what is the final table results. So, what I'm going to do is I'm going to query our final table, and I want to aggregate it by that job title short column. So, we can see to make sure did those have an update for what I would expect to have updated. One thing I want to see is did the count get updated? Did the numbers go up or did it go down? And then the other thing I want to check is the priority level and the updated at time. For this, right? Anytime we're doing some sort of aggregation, we need to wrap it, so we're going to wrap priority level and also updated at in a min function, and I would expect them well, they are going to be all the same, so it's going to provide a single value back using that min function. This is going to be from the priority job snapshot table, which we created right above, and we're going to group by the

### [8:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=30000s) Segment 101 (500:00 - 505:00)

job title short column as expected if I'm doing all this aggregation. And then finally, I want to put the most important up at top, so I'll put the order by job count in descending order. All right, this is looking good. Let's save this now. Same thing, I'm going to come over to that table, I'm going to right click it, and I'm going to copy the relative path. And then inside my terminal, I'm going to do dot read command, and then paste in the location of that script. Let's run it. Hopefully we don't have any errors. And I got an error cuz I had a typo. It says, "Hey, table name job postings facts does not exist. Did you mean job postings fact? " I did. I'm going to go ahead and update it up here. Save it by pressing command S. Going back in the terminal, pressing up, and then running it. All right, that took a total of 3 seconds, and I can see that the final table that we have has, looks like, almost 500,000 jobs in it of data engineer, software engineer, and senior data engineer. It's what I expect. And the priority level of one, and it was updated, well, as of looking at this, yep, well, just a few seconds ago. So, that is our initial load script. We're only running that for the initial load, so we're not going to be using any more. We're going to be transitioning to the batch load process now, and that'll be using the update, insert, and delete statements. Because of that, inside of here, just to keep things simple, I'm going to close out of the priority job snapshot initial script cuz we technically shouldn't have to run this again. I am going to keep that priority rules open because we'll be updating this to make sure that our batch processing actually is working correctly. So, let's get into building that data pipeline for updating our snapshots final table. In this, you know I love Venn diagrams. And so, with this Venn diagram, what we have is we have a source table, which is represented by the blue color, and then our target table. And what I want to go through this is what we're going to be doing four different sets of data that may be in the source and not in the target or in the target source. First up, let's talk about those right here in the center where our source and our target table overlapped. And that's where we have matched rows. If they're matched rows, they're either likely okay and we don't need to update them, or they maybe have a change within the row and in that case, we need to update them. Next up are those that are unmatched in the incoming rows of that source table. In that case, if they're in the source table and not in our target table, we need to insert them in. And conversely, if they are unmatched existing rows in our target table but are no longer in the source, we need to delete them. So, we're going to be walking through how to do this update, insert, and delete with our tables. Specifically, we already know that our target table is going to be that priority jobs snapshot. Our source table, which we've shown previously in the initial build, if you will, that is just right now it's only a SQL query for what we can get. So, what we're going to do is we're going to build a temporary table within our script to create this source table and then as necessary, load it or update it, insert it, or delete it into our target table. So, inside of 1. 24, I'm going to go ahead and create a new file. And for simplicity, we already have the other one called the prior jobs snapshot, we're just going to call this one priority jobs snapshot. sql. Now, inside of this script to start with for this data pipeline, we're going to create a temporary table that has our query that creates our source table. And the reason why we're creating this temporary source table is because we need to then reference it in our other statements below. Specifically, we have to reference it in our update, insert, and delete. So, let's get into creating this source table. First thing we're going to do is use create or replace temp table or temporary. We're going to give it the alias of source priority jobs. And I have that backwards. The as needs to go at the end of that cuz we're assigning that name as and we now need to use that query that we previously did. Now, I said we could close out the initial. I'm going to go ahead and open it up. And in here, we're going to reuse this insert into query right here with the select statement. So, I'm going to go ahead to paste that into here. Now, I want to test this as I go along. So, let's just run this cuz it's going to do no harm just to run this and create this temporary table. So, I'll right-click that priority job snapshot in order to get that copy that relative path. And then inside of our terminal, I'll use a {dot} read command pasting in that snapshot SQL file and it ran with no issues. Now, the first thing we're going to write a query for is an update statement to capture those matched rows that may have chains that are different between the source and

### [8:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=30300s) Segment 102 (505:00 - 510:00)

target table. Now, as a quick refresher on update, this is used to in situations where maybe a value within a column changes and we need to update it. It can change with either a single column or multiple columns. In our case, what we're trying to update for or the only situation we tried to update for is if that priority level changes for a role. Not if a new role gets added to that priority roles column, but if a priority level changes such as software engineer changes from like three to one. Now, the update syntax is pretty simple. We're going to use update, specify the table, you set to set what column we want to the new value, and then we use a where condition. The where condition we're going to be using existence filtering for this. So, we'll start with that keyword of update and we're going to be using the main. priority_jobs_snapshot as our target table. We're going to give it the alias to keep it simple and our mind in the right spot of target. In this, we want to set from the target the priority level equal to our sources priority level. Which our source is up here is that temporary table that we previously established. So, we'll define it in our from statement that source priority jobs as source. That's the alias of source, SRC. Okay, we don't only want to just update the priority level. Remember, we also created that column that has the current timestamp added. Specifically, it's noted as the updated at. And I'm noticing this temporary table doesn't have that even specified yet. So, I'll do updated at. So, here I'll add in a line now to have the updated at equal to the sources updated at. Which the script's going to run all at once, so it's fine if we want to do it like this. I could also replace the source that updated at with just that current timestamp formula. So, now we need to include a where statement to how we're going to match up this priority level or we'll know what's updated on. Specifically, we need to do it to where when the target job ID is equal to that sources job ID. Now, currently as this is written, this will go through and update every single priority level in the target table whether it updates or not. So, we need to include an and condition to also check if it actually changed. Specifically, we want to check if that target priority level is distinct from our source priority level. And this is distinct from is a new type of way. We've seen distinct before and using distinct for columns. This is distinct from is a new way to check line by line or row by row if it actually changed. So, let's go ahead and test this now to make sure that our update statement is correct. Remember, previously we ran this priority roles already and we saved software engineer as a one. So, I want to see what happens if we were to change that software engineer back to something like a three. I'm going to go ahead and save this from any time we do anything to this file, we save it. I'm also save this snapshots file. So, I'm going to press up to get back to that one {dot} read statement that looks at that priority roles SQL file cuz we updated now software engineer to three. I'm going to go ahead and run this. Okay, it's updated to three. Now, I want to run the priority jobs snapshot. So, I'm going to press up twice cuz we just ran it. And nothing is shown because we didn't put a SQL query at the end. If you remember from our initial build of priority job snapshot, we include the SQL query at the end. I'm going to go ahead and just copy this. And then underneath this update statement all the way at the very end, I'm going to go ahead and paste this in. Press command S. All right, let's see if it did update this time. I'll press up in terminal and then run this. And bam, it updated. Software engineer is now three. And what we can also see from this is that it only updated that software engineer because we can check that updated at time stamp. And we can see that software engineer is updated an hour later. It was an hour later because I took a little bit of a lunch break when between filming this. So, we've now handled those matched rows or those that update with priority level. Now, let's move on. For this, we're going to be looking at unmatched incoming rows. Basically, did we add or did the business add jobs to that priority roles table? If so, we need to insert those new rows into our target table. So, let's start by building this by putting in our insert into statement specifying our main target table of that priority job snapshot. In it, we want all seven columns of that replaced or inserted into. That's typical of insert And we're going to insert into it using a query underneath it. And for this, just for consistency, I'm going to go through instead of doing a select star, I want to make sure that the column names do line up properly. So, I'm going

### [8:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=30600s) Segment 103 (510:00 - 515:00)

to list out every single column name, job ID, job title short, company name, job posted date, salary average, priority level, and updated at. And we're going to be getting this from that source priority jobs, the temporary table. We're going to alias this as source. Now, there's a reason why we assigned it an alias, and that comes to the fact that well, we don't want to insert every single value from the source table into our target table. Remember, we only want to include those unmatched incoming rows that aren't in our target table. So, this one we're actually going to do existence filtering. I think I said on the last one we're going to do existence filtering. This is the one we're actually going to do existence filtering for. Specifically, if you recall back, we want to do where not exist to find those values in the source tables and insert them into the target. In our case, or in case of the where not exist, keep rows with no match in the target. So, we'll do a where not exist, open in parentheses, then start with that select one. We're going to be coming from our main priority job snapshot, which is our target table, and that's where our target job ID is equal to our source job ID. All right, so that's this query. I'm going to go ahead and now save this. And now we need to test it. So, we're going to go back to our priority roles, and we're going to insert a new line here, and we're going to add in a number four of data scientist with a level of three. I'm going to go ahead and save this as well. So, let's first run this priority roles script. So, I pressed up with the arrow twice. I have a syntax error because I forgot to put a comma after this. Make sure you have the correct syntax. Let's try again. All right, so it's updated. Have data scientist in there. Now, let's run the priority job snapshot specifically testing our insert statement to see if data scientist gets inserted in. Press up twice to get to the priority job snapshot and then run this. And it updated. We see that data scientist is added to the list now. It previously, if I scroll back up here, we can see that we previously had three values in it. Well, now we have all four. We have the data scientist three and we can see that updated up updated at time was updated for it about 16 minutes last after we did that insert into. I mean, 16 minutes after we just did the insert into after that last update. So, now we only have one last statement to create. And that is to delete unmatch existing rows in our target table. Basically, if we have jobs that were removed from our priority roles table and affects that snapshot table. For this, we're going to use the basic syntax of a delete from statement and then specify a where condition. Once again, we're going to do existence filtering for this as well to get the correct rows we need to delete. So, let's get in into building this delete statement. The first thing we're going to do is use delete from specifying the main. priorityjobsnapshot as the target table. And then we're going to use where not exist. Now, previously for this where not exist, the main table is source, but if you've noticed, we're actually using in our query, we're using target. So, that source and target are flipped here in the diagram. And we can see that delete from the target table is the main table here, whereas up above here for our insert into, right, our main table is the source and our target is that other one that we're trying to filter. Anyway, I want to clarify that on why we can still write this where not exist because we've sort of flip-flopped it. Anyway, we're going to then put the query or a subquery inside of here a select one from our source priority jobs table as source and that is where our source job ID is equal to our target job ID. All right, so now let's test this. I'm going to go ahead and save this file inside of our priority roles. I'm going to go ahead and just delete out this data scientist and save this. Now let's run our priority roles by pressing up twice. Okay, we only have three roles in here. Then I'll press up twice again to get that priority jobs snapshot and we're now going to run this and bam, we can see that previously we had data scientist in this table, but now in this new one we have data scientist removed. So this priority jobs snapshot SQL file is a full data pipeline in order to perform batch processing and we do this by loading in a temporary table, performing an update, then an insert, and a delete where necessary, and then having that final check query at the end to double-check our results. So technically we could stop right here. We have our initial load script that we built for our snapshots table and then our batch loading V1, but I want to now

### [8:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=30900s) Segment 104 (515:00 - 520:00)

go further. Specifically, I want to combine these three different update, insert, and delete statements into one single merge statement. So let's quickly go over the syntax for merge and this starts by using merge into specifying the target and then what you're using as the source. And then from there you specify what you're doing it on similar to like a join, in this case a key column. In our case we'll be using that job ID. The first thing in merge into what we're doing is an update. For those matched rows, we want to update them and we'll set them appropriate to their columns based on what's in the target to what's in the source. Next up, we'll do those rows or unmatched incoming rows in the source table that we need to put in the target, so we need to insert into. We'll use the keyword in our syntax of when not matched, then and then from there insert syntax to insert it in. And then finally, we need to account for that last condition. Those unmatched existing rows in our target table. This is a single line we can just put in there of when not matched by source, then delete. So, let's go ahead and do this. For this, I'm going to go ahead and select all the way here from the update down to the delete and I'm going to press command {slash} to then comment then all out. And then underneath this all, we'll put our merge into to where we'll be running this now, this new query that we're building out for our priority job snapshot SQL file. First thing we're doing is build out that merge into using and on. So, we'll type merge into our main. priorityjobsnapshot table and we'll specify this as our target table. Next, we're going to be using our source jobs or source priority jobs table as our source and we're doing this on the target job ID equal to the source job ID. Now, we need to include our update statement using that when matched then to include those matched rows where the priority level may have changed. So, I'm going to start with a when matched then, we want to update and set the priority level of our target table equal to source table. Similarly, we also want to update that updated at for the updated at time in our source table. Now, we did up above if we scroll up here, whenever we did that previous updated at, we added this condition in here and that was when the target priority level is distinct from the source priority level. Basically, only updating it if the priority level changed. And that's also what we want to do here. So, I'm going to go ahead and just copy this. And then inside of here, we can go when match Well, it should be when matched. When matched and this condition of the target priority level is distinct from the source priority level, then we want to update the set. So, it's not only matched, but it also it needs to meet this condition to be also distinct. Match being the job IDs are equal. The next condition we now need to get to is doing our insert into. Specifically, unmatched incoming rows in our source table that we need to insert into our target table. This starts with the keyword when not matched then and then with our insert statement. So, in our merge statement, I'm going to do when not matched then we want to insert into and I'm going to come up here. I'm going to cheat a little bit. I'm going to copy this portion of this insert into. We're going to modify it slightly. I'm going to select all the way from the insert into all the way to right before the from. Then I'm going to paste this into here. I'm going to do command {slash} to remove the comments and then tab it over. Now, what we want to remove is we don't want to say insert into cuz we don't need to specify the snapshots table again because we're already doing this in this merge statement right above here. So, we do want insert with those columns and then from there, we want to specify not using a select but inserting into using values and we put an opening and then a closing parentheses around this all. Specifying we're inserting into that target table these values from the source. All right, the last thing now need to match up on is the target tables unmatch existing rows. Basically, we need to delete from it. And we only need to include this key keyword phrase right here of when not matched by source then delete. So, include this. So, I've now now let's go ahead and test if this works. Remember, this updated or this update insert and delete that we created before is commented out. So, we're actually testing the merge into. I'm going to go ahead and save this. Now, the first thing we want to check or test is a change in this. So, I'm going to change this data engineer to a level or a

### [8:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=31200s) Segment 105 (520:00 - 525:00)

priority level of two and go ahead and save this. And then I'm going to first start by running the priority rules to make sure that it got updated to a two. Yep, that engineer is two. Now, I'm going to run our priority job snapshot. This is the first time. Hopefully, there's no errors. All right. So, I've been sitting here for a second. This query didn't execute. That's because I got an error. I quickly uh went through this. I went through this and saw that what I did an on and I had target. jobID up at the top of that merge into the very beginning. It should say equal to source. jobID. I thought I typed that, but I guess I didn't. Anyway, if you didn't do it, uh need to fix that. Anyway, this query is stuck up because it can't do that just yet because the query is wrong. So, a good way to exit out of this, I'm going to just go on press control C and I'm going to interrupt. Now, we're going to try this again, this priority job snapshot. I'm going to go ahead and save it, command S now for those new changes and we're going to try again. And bam, I did it. So, now we have this data engineer and it uh updated it for that engineer of priority level two. And we can see that's the most recent timestamp of 18:42. So, an hour after our last addition. All right. The next thing to test for is let's test adding a line. I'll test in adding data scientist with a level of three. I'll run first our priority rules script to make sure data scientist added. It is. And then from there, test the priority jobs snapshot script to see if it will update or insert those data scientist things. And it does. We now have data scientist added to this and we can see that it's done a minute after our data engineer. All right, last one to test for is a delete. What happens if we delete data scientist from this? Go ahead and save that priority roles. Run our priority roles again. Okay, I see data scientist removed. And then from there run our priority job snapshot. And bam, data scientist is removed. So this merge into statement captures it all. Compared to what we were previously using with our update insert and delete along with some existence filtering, I feel this merge into is slightly easier to read and simpler to use. But as we saw, it took a little bit of an understanding of why this is necessary. So that's why we walked through you building that update insert and delete to start with. Now as a quick reminder, right, we went through this example to demonstrate with both the update insert delete and then also the merge to demonstrate how to go through and perform batch processing. And typically something like we are doing would probably run on a daily basis, maybe even as quickly as hourly. Now you may be wondering this or at least I did after building this. Why wouldn't we, instead of building out that complex merge into that we did, why wouldn't we use something like a CTA or create table as and just create that priority job snapshot table from our original SQL or original SQL query where we queried all those different tables? And that would be a very viable option, especially in our case with how small our tables are. But the problem runs into when you start getting into bigger and bigger tables. Anyway, throw together this table that compares some different scenarios on when you would want to use a CTA over something like a merge. This is very general, and don't take it as hard rules. But, if it is a small table that you're building, less than 10 million rows, a CTA would be perfectly fine. And this would accomplish that batch loading that we would need to do by just rebuilding the entire table instead. You may also want to do a CTA when it's a very large table and more than 50% of the table needs to change. Basically, you maybe have some deletes or inserts, and it's going to be a lot of rows, then you may need to build it from scratch. But, I would say in most situations where you have a very large data set, greater than 10 million rows, and you're just doing incremental changes, so less than like 10% or so, that's when a merge would be possible or when you'd want to use a merge. Other times you'd want to do it these last three where you need atomicity, some live traffic, or even incremental or streaming data. These are a little bit more nuanced, and I wouldn't necessarily think that you need to commit those to memory, though. All right, your turn now to go through, and we have some practice problems for you to now get more familiar with this merge into. In the next lesson, we're going to be going deep as you saw we did uh some intro into data modeling, but we're going to be continuing this with data modeling in our part two lesson. With that, I'll see you in the next one. Welcome to the second lesson on data modeling. We covered the first part

### [8:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=31500s) Segment 106 (525:00 - 530:00)

before project one, and this one we're going to build on it further. Now, don't worry if you don't remember much from that part one. I'm going to do a quick refresher at the beginning of this. But, the majority of this lesson is going to be focusing on understanding design behaviors and building different databases, specifically data warehouses. We're going to go through seven core design patterns with the key one being a star schema as it's most common seen in data warehouses. Now, we're not going to just jump straight into design patterns because we first need to understand where are we building this design pattern? Are we building it in a source system or an analytical system? Previously, we've been working a lot in analytical systems, specifically data warehouses, but we have to be able to get this data from somewhere, a source system, and put them in analytical system. Don't worry, we'll cover more on that when we get there. Also, because we're going to be covering the source and analytical systems that will support the different design patterns, I'd be remiss if I didn't cover the different types of databases that you can use for these source and analytical systems. Anyway, I'm getting ahead of myself. Let's actually get into this with a quick refresh of what we covered in the last lesson. Previously, we went over the database hierarchy, where the database is the overall container that keeps all of our data inside of it. Specifically, we have different schemas, and then under that, we have the sub-level of tables. We then transitioned into understanding how our data warehouse, specifically that data jobs data warehouse, was modeled. And this was modeled after a star schema using fact tables and associated dim tables. And with that, we learned the importance of inside of tables, the importance of things like primary and former foreign keys in order to relate the different tables, and then the relationships that you could have between tables, specifically the most common of either a one-to-one relationship or a one to many relationship. And then finally, we wrapped it all up in that part one by using actual SQL to query that information schema to look at things like tables, columns, views, and table constraints. Now, in that lesson, that lesson had some actual SQL exercises. This lesson is going to be mostly all theoretical. And by mostly, I mean entirely. Now, why the heck does data modeling even matter? Well, in order to understand that, we probably should look first at the problem. And here's the problem I had whenever I was first trying to collect these job postings in order to make a data warehouse that had all these postings throughout the internet. So, the first website that I targeted was LinkedIn. Now, LinkedIn probably uses some database. We'll say in this case they'll you they use SQL Server. And they probably have some sort of design pattern to keep all of their job postings organized in a data model that they like for their own data team. And with that, they have their own schema, so therefore table designs as well. Now, let's say I did get access to that LinkedIn SQL Server database and I was able to get that in. That's great. But what happens when I go to the next company? For example, let's say I go to Indeed next. They may use a different database such as MySQL. completely different schema design as well along with table design. Oh, and then I go to say even one more thing like Upwork, and they're using Postgres with a different schema and table design as well. Well, how the heck if I have this data with from these three different sources, how as a data analyst could I go in and actually query this data? It would be a mess. So, data modeling helps with this by bringing in structure and consistency. If we design our schema to have a known design pattern that our analysts are going to be using, they can go into it and access the data a lot quicker. Additionally, we could use a table design as well with certain column attributes that makes it a lot easier to analyze and aggregate. Well, for the solution of what we could do for this is we could combine these three different databases of SQL Server, MySQL, and Postgres from LinkedIn, Upwork, and Indeed and combine this data into a singular location, specifically a data warehouse. And in our case, we're using DuckDB in order to drill in and analyze this data from all the different sources. This is why data modeling is so important. As all sources are standardized into one analytical model, an analyst can query data from a single

### [8:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=31800s) Segment 107 (530:00 - 535:00)

consistent place. So, this was my use case. What would be a real-world use case of this? Well, typically, as a data engineer, what we're going to be doing is taking data from some sort of source system. In my case, that was the example that was given with LinkedIn, Indeed, and Upwork. Then from there, perform some sort of ELT, like an extract, load, and transform. These are just example scripts right here. And then from there, loading it into analytical systems. In our case, our data warehouse of that job postings fact table and the other dimensional tables. Now, analytical systems, we've seen data warehouses, and we've also seen data marts that we just built in that last lesson. Another common one is data lakehouses, which is basically a combination of a data a and a data warehouse. That's going to be out of the scope of this right now. Understanding is more of an advanced concept, but it is an analytical system where we want to store data in. Anyway, so what are some common source systems? Now, that would be things like ERP systems or enterprise resource planning, CRM or customer resource management systems, app back ends or normalized tables. Now, let's quickly break each one of these down because I don't think you may be familiar with them. Enterprise resource planning is a system that is used by an entire company and it's used to manage, well, everything within the company. You can use it to manage the supply chain, your shipping, your manufacturing. HR can work in it, finance, everybody can work in it. Some common enterprise resource planning solutions that are out there right now are things like Oracle, Odoo, or even SAP HANA. Next up is CRM or customer relationship management. This is a solution that allows businesses to, well, talk to other businesses or customers. It's basically the front office interaction. This is very common for teams like sales, marketing, or support. Popular CRM solutions would be things like Salesforce, HubSpot, or even Zoho. Now, there's two other types of common source systems and they'd be more custom-built. Imagine if you have some sort of app, you could have the app back end which with its own database that would be the source or system of truth. Additionally, you may have a robust data team that builds out your own internal, you know, ERP or CRM systems and they have their own normalized tables. We'll go over more into normalized tables in a bit. So, just hold on to that for a second. Now, all of these solutions have to store their data in some sort of database. For example, Oracle's NetSuite is going to be using an Oracle database. Hadoop uses something like Postgres. And SAP HANA uses, well, their custom solution of SAP HANA. Similarly for CRM solutions, something like Salesforce uses also an Oracle database. HubSpot uses Postgres. And Zoho uses MySQL. Anyway, there's actually a commonality between all of these different databases. And when we're saying that, we need to understand that databases are made with a certain design pattern in mind. Specifically, there's two major types of relational databases that we can have. One is OLTP, which is online transaction processing. And the other is OLAP or OLAP, online analytical processing. And I'm going to say OLAP. All of these are spoken out for be via their abbreviation. Anyway, OLTP are for transactions. And they operate apps and capture live transactions. This is optimized for high velocity writes. So, it's very common, like if you're in an ERP or CRM system where we're getting lots of data updated all the time, it's constantly having to write to the database. Now, the structure of this is more normalized and once again, we'll get into a normalized data schema in a little bit. The other type is that OLAP or online analytical processing and it's our analytical type databases. This is optimized for large scans, aggregations, filters, basically any type of data analysis. And this is typically stored in a structure that we saw previously of fact and dimension tables or let's star schema. So, let's look at some common OLTP and OLAP or transactional and analytical databases. We're going to start with the transactional first. Some popular options include things like SQLite, Postgres, MySQL, also other ones like Microsoft SQL Server, Oracle's own database, or MariaDB. If you recall, when we looked at the databases of ERP

### [8:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=32100s) Segment 108 (535:00 - 540:00)

systems, all of those, well we didn't look at SAP HANA, but that Oracle and Postgres were transactional processing databases. Additionally, for those CRM solutions, they were also using transactional processing databases. And that's what's typically used in any type of source system. Now, moving on to those OLAP or analytical databases, we've already been working with one, which is DuckDB, but they also have other popular options like Google's BigQuery or Amazon's Redshift. Then you also have popular options you've probably heard of Databricks, Snowflake, and even something like ClickHouse. So, for our pipeline of what you would typically see of a data analyst from those source systems to analytical systems, you're typically going to see those source systems are typically OLTP and analytical systems are typically OLAP. These aren't hard and fast rules that you have to use these type of databases. Like you could see a Postgres database as an analytical system, but it's just not common. Also, it's just not designed for it. Although, Postgres is getting a lot faster. All right, so we're now going to shift into going into the seven core design patterns. And the main one that you're going to take from this, that I want you to take from, is the star schema that we're going over first. From there, we're going to see that a lot of these different design patterns are based off of the star schema or a modification of the star schema. All right, first up is star schema in that it has one central fact table and it's surrounded by dimension tables. The fact table is a central table that stores measurable, quantitative data about business events or transactions. The dimension tables are supporting tables that provide descriptive context about the facts. Facts answer how many or how much and the dimension dimensions answer who, what, when, and where. For example, an e-commerce company may have a sales fact table and then as dimensions around it has things like the customer, product, store, and date dimensional tables. Our job posting table or a job posting data warehouse really has two dimensions in that we have a job postings fact table and then a company dim and a skills dimension table. Now, this star schema is the most popular design pattern used within analytical systems. And because we typically use OLAP databases for this, you're going to see star schemas in those OLAP databases. So, what is the most common design pattern in source systems which are typically OLTP? Well, that's where we get into a normalized schema. And this is a database design where data is split into multiple related tables. The goal here is to reduce duplication and to maintain data integrity by only in storing the instance exactly once. In the real world in ERP systems, I've seen this where it's hundreds of different tables linked together. It's a nightmare as a data analyst trying to go through trying to find the correct table. But, it's great for maintaining transactions and actually storing all of our results. So, when we compare this normalized schema that's usually a source system to something like a star schema that would be in a data warehouse or mart, the star schema is a lot less intimidating as we have the data in a known form, the star schema that an analyst would expect for them to go to and analyze and extract data from by having to go into a normal schema and try to figure out where in the hundreds of tables something may be. So, in a real-world scenario, we would have to develop either some sort of ELT or ETL pipeline to get maybe some order data into and created into a star schema within a sales fact table along with its dimensional tables. And building each one of the tables would be as simple as for the customers just moving the customers table over, but then maybe for the sales table it would be extracting out and joining different tables such as the orders, payments, and order items. So, this normalized schema to a star schema would be exactly what we would be trying to do in this analogy here that I have with this source to analysis of pulling something from a source system and getting it into an analytical system. All right, next up is a constellation schema, also formally known as a galaxy schema. Constellation is more frequently used nowadays. Anyway, if we can see from this, this is a rendition of a star schema in that it's a collection of multiple different stars. As you can see, you can have multiple fact tables connected by multiple different dimension tables. With how it's arranged, you can see how it sort of looks like a constellation. This would be used in very large company

### [9:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=32400s) Segment 109 (540:00 - 545:00)

like enterprises that have maybe different teams such as a sales team, an orders team, a marketing team, or whatnot. They would have their associated tables and it would be easy for them each to get into the data they need. So, this is a typical schema you would see in a real enterprise data warehouse. Next up is a snowflake schema. And this one is also a rendition of a star schema in that you have your fact table surrounded by a dimensional table and your dimensional table could have an additional things that spun off of it to have like sub dimension tables. This is more typically used in legacy data warehouses. You're not finding this used as often, but you could find yourself on a data team that has this implemented still. A simple example of this would be say we had a sales fact table and then from there we had our product dimensional table and then each product may go into a category and then each category subcategory. So this is going into a snowflake type schema. You could also see it broken down to something like this. We're not going to go through every attribute, but maybe in the upper left-hand corner you could see sales broken down by customers and then customers broken down further by their customer type and maybe region. Next up is a factless fact table, which once again is a rendition of a star schema. But in this, pay close attention to this fact table. In it has what's factless. The only thing inside of here are IDs to associate it to its associated dimension table. So no numeric measures are recorded in the fact table, rendering it a factless fact table. Our job posting's fact table does not fall into this for our data warehouse. We actually have numerical attributes in there like salary, whether work from home and whatnot. However, we do have this design pattern of a bridge table. A bridge table is a table that resolves a many-to-many relationship between two other tables by acting as an intermediary. So between our job posting's fact table and our skills dim table, we have well, what's actually named as skill jobs dim, but it would be more appropriate to name it skill jobs bridge because technically it's a bridge table. That skills job bridge table includes those foreign keys of the job ID from the job postings fact table and the foreign key of that skill ID from the skills dim table. If that table didn't exist, we'd have to shift what not only the skill ID into the job postings fact table, but also the job ID into the skills dim table. And then we'd have to join them using the job ID and skill ID, and that's going to be creating a many-to-many relationship. That'd be awful. Bridge tables help solve that. All right, the last design pattern that we're going to cover is a flat or known as a wide table. In this, we would take something or data that's in a data warehouse in a star schema, perform some sort of ETL or ELT, and get it into a single table with all of our column and attributes in it in one. Flat tables are sort of what I'm more used to it as an analyst, and that's historically used in reporting in Excel or BI tools. You'll probably more commonly here in the data engineering world referred to as wide tables. And this wide name emphasizes that all the relevant attributes are joined into one wide table with many columns. I would expect to see a flat or wide table in a scenario where maybe you have a more junior analytical team and they're not used to using a star schema database or a data warehouse to query out their data. Now, there's one last thing that I want to cover, and it revolves around maintaining and updating data models. And that comes through a very well-done well-known practice of slowly changing dimensions or SCD. Now, if you think about it, if you make a data warehouse and it maintains for over 10 years without updating, you're probably doing something wrong. SCD is a method used to track and update changes in a star schema or data warehouse. So, like in the case of where I said, "Hey, skills job dim probably needs to be updated to skills job bridge. " I could use this type of method to update this table so that way I have it tracked properly and changed properly. Now, slowly changing dimensions is only one method to actually track changes. There are a few others, but SCDs are by far the most common to track changes. And typically, you could see anywhere from zero to type six. In no way and we're going to go through a few of these and no way do we need to memorize them at all. They're mainly just there for you to understand what the purpose of this is. Now, for implementing these

### [9:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=32700s) Segment 110 (545:00 - 550:00)

you could use any one of these types or in the case of something like type five, it's a hybrid of type one and two. Or in the case of type six, it's a hybrid of type one, two, three. But, what the heck are these types? Let's look at some examples. Well, let's say in our company dim table, we needed to update a record and we had a company called Facebook, but it got renamed something like Meta. Oh, and also the link for it changed as well. Well, with type zero, it's fixed. So, you can never change the value and it's immutable. It has to stay what it is. That's actually an SCD. Kind of strange. The next type or type one is an overwrite. Up above, we have dim company before the update where it's Facebook and we have Facebook's link. And then after the update, you would find that it got updated to the name of Meta and then the link. Now, one thing to point out in this is you would typically find in this type of environment that you would have a company SK or surrogate key and the company ID. The surrogate key would be technically the primary key and it would be assigned by the data warehouse and the company ID would be the unique ID inside of our maybe normalized schema inside of our source system. So, a little bit different from the company ID that we have in our table. Next up is type two where we would just add a new row. In that we would create another line where we'd create that new surrogate key for that company and we'd add in the information of meta, the new link, and when it was valid from or valid to and anything else we need to associate with it. Type three would be just be to add a new attribute or attributes. We'd add new columns. Previously we had name and link. We probably change them to previous name, previous link, and then add the new names and new link under current name and current link. In this case, this only tracks the last change, not the full history. The last one we'll cover is type four of a history table. So, our current or new table would be that dim company table with our meta information in it. And then we'd probably save our old previous one as something like dim company history. In it we could also not only have that previous name of Facebook, but we may store in it as well that history version what our new current name is as well. The key thing here is that main table stays clean and the history moves elsewhere. And then just to wrap this up, like I mentioned, type five is a hybrid of type one, two, and type six is a hybrid of one, two, and three. Anyway, the key thing to under stand here is not to memorize these different types, but to keep in the back of your mind what would be a way to keep track of design patterns if they need to change. All right, so I have some practice problems for you now. Go through and test your knowledge on all of these different design patterns that you could have within the different types of databases. With that, I'll see you in the next one. We'll be getting into case expressions. All right, see you there. All right, in this lesson we're going to be going over case expressions. These apply conditional logic in SQL transformations in order to categorize, clean, and standardize data. Really powerful tool for data engineers. So, after we go through the basics of the syntax of case expressions, we're going to then go in through some multiple quick use cases along with a final example. So, getting straight into the syntax, case expressions begin with well, case as the expression and it they end with end. Inside of there, we can specify multiple conditions that we want to meet or when we meet a condition, then we do what that condition is. There's no limit on how many when and thens we can then perform underneath this. That's why the when and then in the second one is in square brackets. Now, additionally, we can do a catch-all at the end of an else, which is optional cuz it's in square brackets, and it provides an output if none of the when conditions are met. Now, today, we'll be going through examples using case expressions inside of a select statement, but really they can be used anywhere where a value is expected in SQL. So, you could use them in wheres, group buys, or even order buys. So, let's go through a simple explanation or example to showcase how this case expression would be used in a scenario like this. In this, we're going to be using the salary hour average column, and we want to basically say, "Hey, if there are any values that are greater than $40, we want to classify it as high salary. Otherwise, classify it as low salary. " This low salary uses an else condition as a catch-all for any values that are not greater than 40, so basically less than 40. Anyway, we wrap

### [9:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=33000s) Segment 111 (550:00 - 555:00)

this all in a case and then also end. And as you can see, we provided an alias using as and specified to be salary category. So, let's walk through this step-by-step. We're going to have in our job title short column, we have data analyst and we have our first value of 10 for salary hour average. It's going to check whether that 10 is greater than 40, basically that first condition. In our case, this is false, so we're going to now meet the else condition to classify this as a low salary. Similarly, we'd have something like data scientist, maybe that's a 30 as well. The 30's going to be less than that 40, so it returns a false and therefore once again we meet that else condition of low salary. Finally though, we will get a result like data engineers at 50, conveniently greater than our threshold. In that case, the returns it true and therefore we will meet our condition to be marked as high salary. Now, we're going to be walking through using case expressions in four popular use cases that I've found for data engineers. This first one of bucketing data will build off that example that we just went through using salary data in order to bucket it. We're just going to make it slightly more complex. After this, we'll move into our second case of handling null data, specifically with the that salary data. Then, we'll shift gears and we'll perform categorization of values using text analysis. And then finally, we'll wrap it up with our fourth case using conditional aggregation and this is more of an advanced case in order to perform analytical analysis using case expressions. We're going to start with this first one first of categorizing data or more specifically bucketing data. We're going to bucket salaries based on the following bins. And this is all for that salary hour average column from our job postings fact table. So, let's start writing this query. For this, we're going to write a select statement. I'm going to add in [snorts] that job title short. Also, I want to check as we go through with our case expression. So, I'm going to list also the salary hour average column. Next, we're going to start our case expression. I'm going to write a case. Then, also an end. Inside of here, I'm going to put our when. And we want to meet our first condition of salary hour average less than 25. So, that's the entire condition right there. Then, we want to categorize this as low. For this, make sure we're using single quotes for this. The next condition we want to meet is when salary hour average is Well, we can do between 25 and 50, but I'm actually not a fan of this. Instead, I'm more of a fan of just explicitly saying, "Hey, less than 50. " because you've already once you go through that first condition of filtering out all those that are less than 25, you can then go to the next one of less than 50, and it's understood that it's between 25 and 50. In this case, we're going to classify this with a then statement of medium. Now, I could do another [clears throat] when statement, but we'll just right now we'll just wrap it up with an else, and all others are high. We'll give this an alias as salary category. And we're going to be getting this from the job postings fact table. Now, one thing, right? We don't all not all values in our table are filled. There's a lot of null values in here. So, I'm going to filter them out by giving a where statement salary hour average is not null. Also, I don't want to do this to the entire data set. So, we're just going to put a limit 10 on there and close this out with a semicolon. Okay, let's go ahead and run this. For this, I'm going to open up Turnable terminal and connect to our data jobs database. Selecting our query and throwing it in here. Bam, we get back these values, and I can now go through this and see that, okay, it is matching our logic. Everything's matching as expected. All right, the next type of method or that is used for case expressions is in handling missing data, specifically those nulls. We're going to build on that previous example, and instead of just filtering out null data using a where, we're actually going to use this within our case expression to handle this. So, I'm going to come back to our previous query and copy it, and then paste it into here. Now, let's see how null is handled if I were just to remove this where statement right here, and then run this. Basically, those nulls get captured by the else statement. And I'm pointing this out because this first when of salary average less than 25, like null is not a value, so it's not getting bucketed into that low of less than 25. Instead, it's going in getting captured by that catchall of the else high. So, I like to be explicit with if I want to capture or handle null value. So, I'm going to add a statement or a when statement to capture this. Specifically specifying when salary average is null, then I want

### [9:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=33300s) Segment 112 (555:00 - 560:00)

to categorize it as missing. Let's go ahead and run this, and now those null values are marked as missing. The next thing we can use case expressions for are standardizing categorical values. Let's look at this use case where we want to use the job title column, not the job title short column, but just the job title column and classify jobs as either data analyst, data engineer, and data scientist. Let's just take a look at these values first, specifically looking at the job title and job title short. And that's from job postings fact, and we'll limit this to Well, we'll limit it to 20. We'll go ahead and run this. All right, looking at this, this is not necessarily providing what I want as most of these are Well, data analyst or senior data analyst, and I got this one business analyst. I want to look at some that are data scientist, engineers, and data analyst to see if we can categorize them. So, I want to randomize this sample a little bit more. So, I'm going to modify this query that we have and I'm going to put an order by and then throw in here a random function. Random function it is. It's not just DuckDB specific, you can also use it in Postgres and SQLite, but it's not used in every single uh SQL dialect. Anyway, let's go ahead and run this. All right, so this is getting more random results of what I want. All right, so looking at this, that I think the plan of attack I'm going to do for this is so if like things like data engineer, I can see that there's data and engineer in there. In this case, it has data engineers with an S, so we'll need to make sure we capture for that. But even the other ones like data analyst, they do have I mean data and analyst in it and data scientist has things like data and scientist in it along with other things. So, I think if we're filtering for those specific two words in each, we could maybe get, you know, maybe 90 80% of it. All right, so after the job title, I'm going to throw in that case and then automatically put in that end and I'm going to give it an alias now of job title category. All right, so let's go for the first condition and that's when job title contains data. We can't do that equal sign, remember we're you talking with text character right now, so we're going to use like and then inside of single quotes, I'm going to use two wild card characters of that percent sign and then data. And so this is helping those percent sign, if you remember, like a match zero or more characters. So, data could be at the front or it very end, it'll match no matter what. Now, this is just the first of one condition. We can actually we can plug multiple conditions into this and I can do this by putting an and in here and then specifying that job title. Now, we want to add in that portion. Let's we're going to go for analyst next. So, we'll use like and then inside of single quotes, those two wild cards surrounding analyst. All right, so now we have these two conditions in here. If we meet those two conditions, then we want to categorize it as data analyst. Okay, this is actually enough for now to go ahead and just run this, right? Cuz remember, you only need one when statement inside of a case expression. So, I'm going to go ahead and run this, and I have an issue because I forgot to include a comma at the end. Specifically, we say, "Hey, end as job title category. " I need to include a comma right here. All right, let's try this again. All right, not bad. We can see our first one, it does match on that data analyst and specifies the analyst. Also, this list doesn't really have a lot of data analyst in here. And this one right here does have data analyst in it, but this case sensitivity messes us up in that the analyst is lower case. Luckily, we're going to be covering different text functions to help with these type of situations, but that's for an upcoming lesson. Right now, we're not going to be able to handle it. So, let's build out the remaining of those scenarios we want to meet. So, I'm going to go ahead and just copy this, paste it twice, and then update that line for data engineer, and then the third line for data scientist. Then we'll also include a catch-all using an else specifying it as other. All right, let's go ahead and run this bad boy. And bam, not bad. This goes through and as we can see, I mean, this doesn't get 80-90% like I thought. This gets about probably about 70%, but pretty good in categorizing these data analyst, data scientist, and data engineers just using that job title column. It's not really related to this, but just as background, if you're curious, that job title short column I used a machine learning model. Specifically, it's called BERT. You can Google it. It went through and read all these job titles and then categorized it into these 10 different ones here. So, I didn't necessarily use a case expression for this, but this is a situation where you could use a case expression if you didn't have access to machine learning models. Of the last use case we're going to look at is more specific to those data analyst and I do cover this more in depth in my intermediate SQL course.

### [9:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=33600s) Segment 113 (560:00 - 565:00)

This is going to be a brief intro. You don't even have to follow along if you don't want to for this specific example. Anyway, it's on conditional aggregation. Let's say we want to calculate the median salaries for different buckets. Specifically, we'll say hey, one of the buckets is a yearly salary of less than $100,000 and greater than $100,000. What is the median salary for each of those buckets? Once again, this is a more advanced case, so don't expect that engineers to have to do this all the time, but you should be aware of it. So, for this we're going to do a select and we're going to aggregate this based on that job title short column. So, I'm going to use that first. And we're going to just do a simple count first just to get the hey, what are the number of job postings there. This is not part of the conditional aggregation using case. This is just a simple total postings that we're trying to find. So, let's get into this actual conditional aggregation. For this, we're going to be using the median function and inside of here, we're going to use that case expression. And we're going to do that first bucket first of when salary year average is less than 100,000. And in that case, well, we want to provide that salary year average. Now, technically, the else basically if it's not any of those conditions, we want to provide it back as null. And so, you could do else null, but that's sort of that's not necessary. It's redundant. It's already assumed. So, I'm not even going to include that. Anyway, we'll give this all the alias of median low salary. Put a comma. All right. So, I'm going to just copy this now and paste this underneath here and change this to greater than or equal to 100,000. And the alias to median high salary. Remove this comma from here. And then we're going to be from the job postings fact table. Specifically, where that salary year average is not null. Obviously, I just don't want to see them in results. And then from there put a limit 10 in there for to see only those first 10 results. And actually, limit 10's not necessary. I don't know why I said that. We're doing aggregation [snorts] here, right? Counts, medians. So, I need to do a group by that job title short column. Now, we'll go ahead and put a semicolon. A limit's not necessary cuz we only have job 10 job postings. We'll go ahead and run this. And bam, we get this back to where we see for each of our job postings, what are the different total postings they have associated with it? And then for salaries less than 100,000, what is the overall median salaries for that bucket? Compared to those greater than 100,000, what are the different What's the median salary for those? Once again, this is an advanced concept. I don't necessarily expect that engineers to know this. You should just recognize when this is being used in case you find it in queries out there. All right, let's move into our final example. And for this, we're going to look at one other common method used for case expressions. And that is for conditional calculations. Specifically, that first part of this problem, where we're going to compute a standardized salary using our yearly salary and then adjusting also our hourly salary. Basically, getting our hours hourly salary to a yearly data and then making it all into one column. From there, we're going to go also and then bucket this data into tiers of low or medium or high based on where it falls into the provided ranges. So, let's build a query first to handle that first condition of getting that standardized salary. For this, I want that job title short column, and then we want to check the work as we go. So, I'm going to put in that salary hour average along with salary year average. Then, we're going to get into building our case expression, which we're going to give the alias of standardized salary. So, for this, we're making this column, right? And it's using yearly salary when yearly salary is available, or then hourly salary adjusted whenever that's available. So, the first thing we need to check for is well, is yearly salary available? And we can do this by checking if it is not null. So, I'll specify salary average is not null. In that case, I do want to provide it. Oops, sorry. I'll write this hour. I don't know why I write hour. We're talking about year like right now. I change that to salary year average. So, in that case, then we want to provide that salary year average column because it's not null. Conversely, when salary hour average is not null, then we want to provide salary hour average, but with a twist, we want to multiply this times 2,080. Basically, there's 2,080 or typically 2,080 work hours in a year. So, I'm going to multiply this times 2,080. Now, in either of those cases, if we don't meet this, I want to provide it back as null. But, as I discussed earlier, that's sort of by default anyway, so I'm not going to include that. All right, let's wrap

### [9:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=33900s) Segment 114 (565:00 - 570:00)

up this query now. We're going to be doing this from the job postings fact table. And I just want to see the salary year average and salary hour average data right now. So, we're going to say where salary year average is not null and that salary hour average is not null. I'm going to go ahead and limit this to 10. Let's go ahead and run this bad boy. And I have a syntax error. Basically, I included a comma at the end of my case expression or my when. That's not right. You don't need to include a comma at the end of that. I went ahead and removed it. And we got zero rollbacks cuz I'm silly and didn't even think about this. This shouldn't be an and for this condition of where. This should be an or condition to actually get something back cuz it's not going to be never both a salary hour average and salary year average in a single column or in a single row. Okay, we finally got results back. Anyway, pretty cool. We have back this standardized column where we have that adjusted salary average provided in a yearly format. And this is all within a single column that we now can aggregate with. Or like we want to do, actually bucket. So what we're going to do now is we have this that we have this available. We need to move on to categorizing these salaries into tiers or buckets of low, medium, high. So because of this, I'm going to wrap this all into a CTE. So I'm going to indent this over, remove the semicolon, and I'm going to go and start with that statement of with and we'll give it the alias salaries as and then open the parentheses. And then at the end of this, close the parentheses. Now, I need to also remove this limit 10 at the end cuz we're wanting to well, bucket all the salaries. I can If I want to, I'll put the limit on the next one. So now, we're going to add a select and I want to get Well, I want all of these columns from that salaries. In addition to, we're going to be bucketing, so I'm going to throw in that case expression. And we're going to give this case expression that we're doing as a salary bucket. Now, our first condition we need to meet right up here of less than 75,000 of low. So coming in here in that case expression, I'll do when standardized salary is less than 75,000 then low. Then 150,000, we'll do medium. Else, we'll do high. Now, remember we're filtering out null values up here in the CTE. So, if we didn't have this, actually I'm going to just go ahead and remove it and say we don't have that there to remove out those nulls, we would need to have a condition to handle those nulls. So, we'll say, "Hey, when standardized salary is null, then it's missing. " All right, so not bad. Let's go ahead and wrap this up by doing from of our CTE of salaries. And for this one, I do want to see the values, right? Remember we have null values in there. So, I'm just going to do an order by of standardized salaries and do descending. We'll just do limit 10 for right now. Okay, let's go ahead and run this. Hopefully we don't have any errors in this. All right, not too bad. We're seeing, well, all of these are marked as high because, yeah, they are greater than 150,000. Conversely, I can come in here in this order by and remove this descending portion and then get it well, ordered in ascending order. And now we can see that the low ones are actually marked. I can also go as far as just to remove this order by altogether and then run this and we can see that the null values are all marked as missing. So, looks like our calculations or case expressions are working out just fine. All right, you have some practice problems and now go through and get more familiar with using these case expressions. In the next lesson, we're going to be jumping into functions, specifically date functions. With that, I'll see you there. — Welcome to this lesson on date functions. And for this, there's a whole host of date functions we go over, but we're going to be focusing on the three most common that I find that I use on a daily basis when working with SQL. First is extract and this is used to pull out specific parts from a date such as like a year, month, or day. Next is date trunk or date truncate, and we can do things like well, truncate a date. So, our date time data we can make into maybe just only a date. And then finally, we'll look at time zone in converting date time data into different time zones. And before we get into all of that, let's do a quick refresher on what we know up to this point on dates. We're going to focus on that job posted date column. So, let's query that column in that job postings fact table. For this, we only need 10 results. In the terminal, I'm going to go ahead and connect to my Motherduck to that data jobs data warehouse. All right, so let's inspect these columns. And silly me

### [9:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=34200s) Segment 115 (570:00 - 575:00)

misspelled job postings fact, not facts. All right, so inspecting this, we can see that the job posted date column is of the data type timestamp, which means it has not only a date part, but also a time part. Now, let's use the casting operator to convert this to a few different data types that we saw previously in our data types lesson of date, time, timestamp, and timestamp Z, where that includes the zone of the timestamp. Or more specifically, timestamp with a time zone. So, I'm going to go ahead and create a new column with job posted date. I'm going to use that casting operator to first convert it to a date, and then give it the alias. I'm just going to keep it simple for right now of date. Next, we'll cast it as time and provide it the alias of time. Then, cast it as timestamp, name it as timestamp, and then finally, cast it as timestamp with time zone as timestamp Z. All right, let's go ahead and run this all. All right, so inspecting these, these are all pretty self-explanatory, right? The date is a date only, the time is well, time only. Timestamp, there was no change because it's already timestamp. And then timestamp with time zone. For this one, look it kept the same time and then it added this negative 06. What the heck's going on here? So typically what databases do whatever you convert it to a timestamp with a time zone, it converts it to your local time zone. Right now, I'm in Central Time Zone the United States and that is -6. And what this means as this is 6 hours behind well where there's a zero and that's known as UTC time or Universal Time Coordinated or more commonly spoken Coordinated Universal Time. This is strange enough defined as the time at the longitude zero or the prime meridian. Strangely enough not with the international dateline. I'm not sure why, but it's no it's located or based off of this time and it's a central location that most people base their clocks off of. And by most people, I mean all. So why did I provide you that history lesson? Well, up above whenever we converted this timestamp Z or timestamp zone, it converted it to this -6 or my time zone. And actually this I know whenever I collected this data I actually stored this as UTC. So it's right now it's converted to the wrong time zone of what it's keeping track of. Anyway, that's a future problem we're going to be fixing with the at time zone function. So little spoiler alert. So let's get into our first function and that is extract. It follows this syntax. For using in a select statement, we can use another statement as well, but for using a select for simplicity, we would use the extract and then we're specifying a part from the column name which the column name would be some sort of date time or date object. So, going back to that base query that we had where we're selecting from the job posted date from our job postings fact table, we're going to go ahead and extract out we're just going to give an example first of extracting out the year from the job posted date column. And then we can also provide it as a column of job posted year. Let's go ahead and run this bad boy. All right. So, for all of these we can see that we got this new column called job posted year. We got it in an integer data type format because it's just well, integers for the year itself. So, now you're probably wondering, "Well, Luke, what are different parts that I can put into here besides just year? " Well, jumping into the documentation in DuckDB on date part functions, we can see first of all they can be used for timestamps, timestamp Z, date, and also interval. Intervals real quick are periods of time that can be added or subtracted from date, timestamps, or timestamps with zones. We're not going to go into intervals at all with this, but it is important that you're aware that you could do intervals such as like 1 year or a month and 1 day or whatnot. Anyway, we can use the parts in the extract function in this list right here. And you can see we could use century, day, decade, hour, microseconds, and whatnot. And this is for date part specifiers and in intervals along with this down here for date part specifiers exactly. And so, you could go as far as saying things like day of the week, day of the year, or even something like week number. Anyway, common ones that I find myself doing are specifying things like the month. And then in this case I gave it the alias job posted month. Or even something like day. In this case giving it the alias job posted day. Running this query, we can see that these also are converted to their appropriate columns and each of these are represented as integers as well. Now, I find this highly useful especially

### [9:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=34500s) Segment 116 (575:00 - 580:00)

whenever I want to do aggregation. Let's say in our case we want to aggregate the total number of job postings for data engineers on a monthly basis. So, let's modify this. I only need the month and year. I don't need this day, so I'm going to go ahead and remove that. I also don't need that job posted date. And remember, we want to aggregate on a monthly basis, so I'm going to throw an aggregation in here. Specifically, we'll count the job ID and we'll give it the alias job count. For this, we'll do this where job title short is equal to data engineer. And then we'll need to remember we're doing aggregation, so anytime we do aggregation, we got to do a group by. And I'm going to just use the actual names themselves. Remember, DuckDB doesn't allow you or DuckDB allows you to use the alias, but some other databases like Postgres don't allow you use the aliases. So, just for continuity, I'm going to use this. I'm going to remove this limit 10 in here. Let's go ahead and run this and see what we got so far. Selecting it all, pressing shift enter. Okay, so this executed the query inside the terminal. Right now, it's a humbug jumble mess, but it looks like it's pretty good except for Okay, it's I want to get it in actual year and month order. So, I'm going to do an order by. And in this case, we can use the alias because that order by is done after the select statement in our order of execution. So, we'll organize by the job posted year first and then job posted month. Now, let's go ahead and run this. Looks like I misspelled order. I don't know what's going on here. — I don't know why it highlighted it blue still. Oh. So, I didn't misspell order. I put a semicolon at the end of this. That's why it got confused. All right, let's go ahead and run this. All right, so pretty cool. It's now plotting it in order of year and month. And now this is much more beneficial and taking it so I can analyze it. Anyway, I threw this into chat GPT to do the dirty work for me and we can actually see how those job postings are going down over time. Specifically, there's a dip back in October of last year and actually I somewhat suspect that towards the end of a year, they sort of slow down on hiring and then ramp it back up at the beginning of the year. Anyway, pretty cool insight with this. Next up is date trunk or date trunk date. For this, inside of our formula for date trunk, we're going to specify a precision and then from there inside of there provide the column of a date time column. And more precisely, it can be date or timestamp with zone. Anyway, this one is precision because this returns the value rounded down to the start of the specified unit. What do I mean by this? So, let's go back to that core query that we were working with previously of job postings date from job postings back table for 10 entries. And here we're going to use that date trunk formula. We're going to specify the precision. Now, this time it has to be in single quotes and provide it. And I'm going to just make it month for right now. And then do a comma and we'll do job posted date. We'll give this the alias of job posted month. All right, let's inspect this see what we got here. And in this case, as we see it ran the formula, what it did was it rounded it down. So, these are all January 1st postings right here. And so, what you're not seeing is it actually rounded it down to all of the January. I think it'd be better if I actually inside of here do an order by and then do random so we have random dates in there. Now, whenever we run this Okay, this one's showing better, right? Here's an entry for June 18th of 2024, but it rounds down, so it's only June 1st, 2024. And look at this, the column is of the date of the data type date. Now, once again, what can we use for well, precision in this case? It's the same values in here. This just this time we do have to use them in single quotes when specifying them within this function. And they're typically written in our case in lower case. So, you can do anything from century, day, decade, hour, whatnot. So, I went into here and I just went and plugged it in. We're going to look at year, quarter, month, week, day, and hour, other common things that I find with date trunk. We're going to go ahead and run this. So, with the year, it rounds up to when this case 2023. All of these start with January 1st be the rounding. The next is quarter, and the same here, it's going to round to that first month of the quarter and that first date. Similarly, we saw month already where it rounds that first date. You could even round to a weekly basis, a daily basis, or even an hourly basis. So, let's simplify our previous query

### [9:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=34800s) Segment 117 (580:00 - 585:00)

that we were working with now using this date trunk form function. Remember, we were using extract and pulling out the year and then the month, then we're having to group by this and also order by that. It's a whole mess cuz we have to deal with two columns. Anyway, instead, we're going to delete this, use date trunk, specify that we're going to do the month as precision, and then we'll specify the job posted date column, and give it the alias job posted month. We're still going to go with the same best practice of listing this entirely inside of the group by, but in the order by, we only have to list job posted month. All right, let's go ahead and run this bad boy. And we're getting now the same results back, only with now two columns. Now, I like this, so what happens if we want to filter this data? Let's say we want to look specifically at 2024. Well, in that case, we can filter it. I'm going to add an and condition in our where. And to do this, I'm going to use the extract function. Specifically, we're going to extract the year from job posted date and set it equal to 2024. Now, whenever we go ahead and run this, we get only the 2024 values back. Now, I find this is very common use cases of combining both date trunk and also extract. And then I'm finding I use like date trunk in the select statement and then extract in the where. Now, I could have used instead of using extract here, date trunk specifying this as year and then of job posted date and setting it equal to 2024-01-01. I think that's right. Let's go ahead and run it. Yeah, that's right. So, I could do it this way, but it's a little bit more robust and it's not as easier to easy to read as using the extract. So, there's cost and benefits of when you would want to use either. All right, last concept to cover is using at time zone. And we're going to be converting between different time zones throughout the world. Now, for this, the syntax is pretty simple. Well, it's pretty simple if we're dealing with a time stamp with a time zone already associated with it. In that case, what we're going to do is with an associated column, we're going to use the keywords at time zone and then specify in single quotes the time zone we want to go to. Now, in order to know which time zone you can use for this, you need to go into the documentation itself. We're going to be using really simple examples in our case. Specifically, uh that UTC or Universal Time Coordinated and then my time zone that we're currently in of Central Standard Time or CST. And we're also use EST or Eastern Standard Time Zone. Anyway, just as a reminder UTC is that zero time zone. So it's right there at the prime meridian in the middle. Central is that of the United States at that -5 or that blue portion and then Eastern Standard Time Zone is that -5. So 1 hour ahead of Central or -5 behind UTC. All right, so let's start with a simple example first and we're not even going to query our database. We're just going to make a single data set. So in quotes I'm going to specify a time and date of January 1st, 2026 at exactly midnight with the time zone of UTC also 00. Now right now this is a string because this is in single quotes. So I'm going to cast this as timestamp and then add a TZ just to specify that it's time zone. Anyway, let's see what we get with this first. I'm going to go ahead and run this. We're going to add that semicolon in there. So what this did is it returns back well data type of timestamp with time zone. And if you notice with this, it actually once again it converts it to my local time zone. So it's going to convert it to your local time zone. And remember I'm in Central Standard Time Zone. So that's -6. And so we originally started with this at midnight on January 1st, 6 hours back would be 1800 at night but for at that -6 for that Central Standard Time Zone. So this is effectively of doing the same thing of at time zone and then specifying in this case Central Standard Time. you'll need to put in what your appropriate time zone is. And then running this, we get what it is of 1800 for that time zone. So, we just demonstrated how to convert from UTC to Central Standard Time Zone. What if we want to go 1 hour ahead to Eastern Standard Time Zone? Well, I would just adjust our query to be Eastern Standard

### [9:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=35100s) Segment 118 (585:00 - 590:00)

Time and then run this. And it has no problems with this. It adjusted 1 hour ahead to 1900. So, this is not that hard of a concept, especially when you're dealing with time zone data. But unfortunately for us, our job posted date data, well, let's actually just query it. And this I'm going to look at the job posted date column from our job postings fact table and just limit this to 10 results. Querying it. Okay, what I want to show with this is that this is just of the timestamp, well, it's of the data type timestamp only. So, the syntax we need to use for this is a little bit different. So, when you have timestamps without time zones, our situation that we're in, it's automatically treated as a local time in DuckDB and most major databases. So, because of that, we need to use at time zone twice. First, we need to use it to specify what is the time zone of the data itself. And then you use it a second time to convert it to whatever time zone you needed it in. So, let's get back to that job posted date. This data, I know as we recorded it, we recorded in real time and I specifically recorded it in UTC. So, what we need to do, let's say we want to convert it to Eastern Standard Time Zone. So, what I'll do is I'll add at time zone specifying UTC. And actually, let's just run it seeing what it's going to do with that. So, press shift enter. With that, we can see that it applies this -6. So, it provide it provides it to our local time It converts zone. But, if we scroll up, we can see that this didn't actually mess up even though I converted it into our local time zone for the results here. When comparing these entries, I can see that Okay, uh previously it was 000004, and that was UTC. So, I would expect whenever we go we do that negative six. Yeah, that's six hours behind that the day before on December 31st of 2022. So, the core of the data is correct, but it's not in the format that we want. Mainly, we want to go to Eastern Standard Time Zone. And so, going back to our query, we can just add at time zone EST. Now, whenever we go to run this, we get that first entry is at Well, one hour ahead, scrolling up here. Remember previously it was 1800 at 0004 for Central Standard Time Zone. Well, Eastern is one hour ahead of us, so that's 1900. So, it did do the conversion correctly. As I would expect it to be 1900 on uh December 31st for this value. But, I just want to show this it wouldn't be correct specifically for our data to not include this at UTC. If I were to take this away, and then now try to run this, this does not go to the right time at all. So, once again, it converts to that local time zone of back uh -6, we're in Central. But, this time it says that it's only one hour behind that initial entry, which scrolling back up, we can see what the initial entry was of January 1st, 2023 at midnight 004. So, this is converted completely wrong. Moral of the story is that we need to If you're dealing with date time data, you need to specify what the time zone is for that data, and then also what the time zone is you want to get to. So, this type of time conversion is really useful. Let's say that I was, you know, we did all this work to get it into Eastern Standard Time Zone. Well, we could filter data where jobs are located here. Specifically, we'll do it on the job location column, and we'll look at values that are from New York. So, we specify these values. I'll also go ahead inside of the select statement include the job title short and job location. Then from there, I'm going to remove that limit 10 from here. I want to just see everything that's there in New York. Now, inside of terminal, we can see we ran the terminal ran the command properly, and we start getting back all the different hours and times that we got all these different results back from New York job postings. And it looks like we got a total of round 20,000 rows. Now, this is actually pretty cool cuz we could do some analysis to see what times locally they are actually posting to sort of game when the most job postings are. So, what I'm going to do is copy this original query that we have cuz that's pretty useful information. And now I'm going to modify this one to where I want to extract out the hour from this job posted date for that Eastern Time Hour that it was posted. So, I do extract hour from and then that previous column that we had. And for this, I then want to do an aggregation, specifically a count of We'll do a count

### [9:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=35400s) Segment 119 (590:00 - 595:00)

of the job ID. We did an aggregation, so we need to do a group by, and I'll go ahead and just copy this all and paste it into the group by. Also, I realized up above I didn't give this an alias, so I'll call it as job posted hour. Okay. Let's go ahead Oh, we need an order by in here. So, I'll do an order by. In this case, we can list the alias of job posted hour. All right, there's a lot of stuff. I'm not sure if this is going to work. Oh, it did work. All right, so we have this. Not bad. I'm going to take this and post this into chat GPT. And what we can see from this is that this is plotted an hour per day. It looks like we have a very high spike at 4:00 in the morning and then it sort of slowly goes down throughout the day and then dies down after 8:00 until about midnight. So, this could be pretty useful in being able to identify trends to a specific region because we've converted it to the applicable time zone. All right, so now your turn to give it a try testing out those three different popular functions I've found for manipulating date time values. In the next lesson, we're going to be jumping into set operators. All right, with that, I'll see you next one. All right, in this lesson, we're going to be going over set operations. And this allows queries, you can have two or more queries, to be combined according to set operation semantics. And there's some mathematical terms that we'll actually go through each specifically for these six terms. Yeah, there's three listed here, but you have technically have union, union all, intersect, intersect all, except, and except all. Anyway, get into deep into it. Let's jump into the computer. So, before even break down those six different operations, let's first understand the problem that we would probably be trying to solve that we need these set operators for. Here I have two tables. One on the left is from 2023 and has job postings. And then also we have the next one to the right of it of two job postings from 2024. Now, between the two, they have some unique job postings that don't get repeated, but some of them do. Specifically highlighted in red is that of a data engineer in the United States for 100,000. That one got repeated between the years. Similarly, we have one for data analyst, and we notice in 2024 that one got repeated once, but got repeated twice. Well, there may be certain scenarios that we want certain values, such as maybe we want all the repeating values or non-repeating values, and set operations are going to solve it. We'll actually go through solving some different scenarios with this for the final example. Now, as mentioned, there are six different set operators that we're going to be walking through today. We're going to walk through each one of these with just simple examples first, and then we'll move on to actually applying it to our job posting data set. First up, we're going to start with the most common of union and union all. In this, what we're going to be working with is the same table, so table A and table B. They only have one column in our case, but as shown in that earlier example of job postings fact, you could imagine that you could also do tables that are, well, more than one column. Anyway, table A is represented by red, table B is blue. Where they intersect is in purple. In the case of union, this returns all rows from table A and B, and it removes any duplicates. Now, union all does something very similar to just union in that all values are preserved or duplicates are preserved. So, in the case of table A and table B, everything is just appended together. So, let's actually demonstrate this instead of just looking at a diagram, and we can recreate those diagrams with a SQL query. First thing I'm going to do is go ahead and connect to Motherduck to that data jobs data warehouse. Now, what we're going to do is we're going to recreate this table A and then also this table B right there. So, we're going to do something we haven't done before. We're going to first create a list of those values. You create a list in SQL by using square brackets. And then from there, we'll place the values inside of here. If I go and head and execute this query, I can see that I get back this main list values 1112 and it's of the data type int of a list. Anyway, it's inside of here. What I need to do now is I need to use a function to get these this list value into rows. So, for that, I'm going to use the unnest function. Running this, I can see that I get all the values in a an actual table with their different rows. I can also create another query for that table B. And this one now has table B's contents.

### [9:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=35700s) Segment 120 (595:00 - 600:00)

Anyway, what I can do is Well, I'm going to remove this semicolon here. We're going to now perform a union between these two. So, I'll do a union right the operator. And all it is is just right in between the two queries. Now, when I go ahead and run this, boom, I get back those values. Now, union doesn't do any type of sorting. It's just going to perform the union operation and somethings may get out of order. In this case, they did 213, but it's unique values. If I change this now to union all and then run it, I expect to get both values from both lists and yes, we get both of them. All seven values. Next up is intersect and this is going to return rows that exist in both queries. Basically, they're intersection. So, this returns rows common to A and B with duplicates removed. In our case, one is preserved in the results and the other duplicate one is removed. Similarly with intersect all, this returns rows common to A and B with duplicates preserved. Note that this one in this case, right? For table A, that has one three times and table B has one two times. So, this is a row-by-row comparison of maintaining those duplicates. So, that third one in table A is not preserved. Anyway, let's test it out. Here, I'm going to go ahead and just put in intersect. It uses pretty much the same syntax as union. Go ahead and run this and only one is returned. Similarly, I'll add intersect all. Running this, just one and one is preserved. Does that row-by-row comparison to only preserve those values? Next up is except. It's also known as minus in some databases like Oracle, but except is most common. This combined results from two queries to compare or filter data. In our case, this is going to return rows in A, but not in B, and duplicates are removed. So, for us, two is preserved from table A, as that's the only value not in table B. Now, except all is going to be slightly different than we've seen previously from other alls. In that, we're going to return rows in A minus rows in B, but duplix duplicates are removed one for one. So, in the case of table A containing three ones and table B containing two ones, it does a one-for-one duplicate removal and one of those ones, that third one in the A, is going to stay in the results along with that two that's not in table B. Anyway, let's demo it in VS code. So, trying out except first, I expect only two to be preserved. Yep, two is only preserved. And then, using except all, one and two are both preserved in that case. Now, these six different set operators are pretty simple, right? Union is just combining all of the rows from table A and B. Intersect is their intersection, and except is basically getting that first table rows that aren't in table B. But, applying this in a real-world scenario can get tricky. So, that's why we're going to move into our final example to actually put these to a real-world scenario. So, let's set up this final example. As I've demonstrated previously, we're going to be using the job postings from 2023 and 2024. We don't have any tables right now with that, so we're going to have to create some temporary tables for both 2023 and 2024. Once we do that, we're going to then walk through six different scenarios on maybe questions we would have about these two years and how we can use these set operators to handle it. So, let's first get into creating those temporary tables for 2023 and 2024 to do this analysis. So, for this, I'm going to start by creating temp table, and I'm going to call it jobs 2023. Then, I'll use that as then we're going to use a query. Basically, we want to select, well, most all the columns from job postings fact. Now, I don't want to compare all columns between 2023 and 2024. And here's why. In 2023, we're going to have unique job IDs that's just unique to basically the row that it's on. So, that's not going to matter. We don't need to compare that value to 2024. Also, our job posted date column is going to be unique to the date and time they posted it, and so that's not necessarily unique to the job posting itself. So, I need to just go ahead and remove those rows and get into a table without those rows. Now, I'm a little lazy, and I don't want to have to go through and write out all the different columns just to remove job ID and job posted date. Anyway, DuckDB and some other popular databases, but not all of them, have this feature where you

### [10:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=36000s) Segment 121 (600:00 - 605:00)

can write select star and then exclude the columns you want to exclude. Once again, this DuckDB specific, not necessarily all databases support this. Anyway, what I can do in our query here is I can do select star exclude and then in parentheses write job ID along with job posted date. If I went ahead and then ran this, I can see that the query executes properly and job ID is no longer in the front. Um I can't really see it and so actually the easiest method is what I'm going to do is I'm going to just put a describe up at the front of this and then run this with describe and this then up outputs what columns are there. Now I have these columns left. I've gone through it and I want all of these columns that are unique to a job posting and there's something that would trigger me in my mind that, okay, this job posting is duplicated by this company because it has this location and this amount of pay. Anyway, I need to get rid of this describe now and go up here. I know this query works just fine, so I can go ahead and well, I can't run it yet cuz this isn't filtered for 2023. So we need to do a where clause and we need to filter it for the year of 2023, so we can use the extract function specifying year from job posted date equal to 2023. All right, let's go ahead and run this bad boy. All right, not bad. I'm going to then just go ahead and query it, see what a sample the data is from this jobs 2023 data set. Looks like 787,000 rows. All right, this is looking good. Let's do the same thing now for 2024. I'm going to just copy this create temp table, paste it below, pit change 23 to 24 in two locations and then go ahead and run this. Then similar to 2023, I'm going to also make sure that it is, you know, available. So, I'm going to query it as well, and it looks like this one has 483,959. So, let's get into our first question, and it's this: What unique job postings appeared in either 2023 or 2024? What do you think? Feel free to pause after each one of these questions to give you time to think about it, but I'm going to jump right in. In this case, we'd want to use the union operator. This is going to return all those unique values with duplicates removed. In order to create this, I'm going to copy this query up above, remove the semicolon, add a union, and then copy this 2024 one, and paste it as well. Now, running this bad boy, we get a total of 1. 08 million rows. But, what was our starting rows to begin with for 2023 and 2024? Well, what I actually like doing with the union operator is what I can do is I can do a count of each of these. So, I add the count function to both. Whenever I run this with this union, what's pretty neat is it puts it into a table with the values of each. And I can actually modify this table further to make it a little bit easier to even read to know which one's 2023 and which one's 2024. So, I'm going to break this query down by formatting it more appropriately. Anyway, I'm going to create two columns. Now, these columns have to match. And for this, I'm going to give a text value of jobs 2023 for this first value, and we're going to give it the alias for the column name as the, well, table name. Now, real quick, I do want to demonstrate something, right? These are These two queries aren't matched up. I have two different types of I have two columns in this first one, one column in the second one. If I tried to run this, I'm going to get an error. Set operations can only apply to expressions with the same number of results columns. Also, they need to be the same data type. So, I can take this up from above, paste this down underneath here, change this 2023 to 2024, and then now whenever I run this, it works. And it has the table name as job 2023, job 2024 along with their associated values. Also, I probably want to give this count an alias specifying that it's maybe the record counts or the row count. Running this again, all right. Now we have it. It's nice and cleaned up. I can actually see this. So, originally we had 787,000 in 2023 and 483,000 in 2024. When we did the union operator with this, we got back 1. 80 million What did I say? 1. 08 million rows. So, now let's say we want to ask the question, what job postings appeared across both years in 2023 and 2024 counting the duplicates. If you said union all, you'd be correct. It returns all rows from tables A and B preserving the duplicates. So, I just copied the query above modifying union. We're going to change this to union all. Originally, we had 1 million job postings in this.

### [10:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=36300s) Segment 122 (605:00 - 610:00)

one, we have 1. 27 million job posts. So, this makes sense. There are more job postings because duplicates are preserved. Next question up, we're going to switch it up on you. Which job postings appeared in 2023 but not in 2024? This would be except in that it returns rows in A but not in B. So, those in 2023 but not in 2024. So, copying the query from above and changing union all to except. Let's go ahead and run this. I expect this to be one of the smallest query results. Well, smaller at least than union. And it is in the fact that it's down to 709,716. And going up to execute that query above that does the count of each one of these, how does that compare to that? We can see that 2023 has 787,000. So, in this case, there's almost 80,000 duplicates that aren't in this one. All right, next up, which job postings from 2023 remain after subtracting those that match from 2024, doing a one-for-one comparison? Well, that'd be except all. It returns rows in A's minus rows in B's with duplicates removed one by one. Copying the query above, adding an all. Previously, this query was at 709,000. Now, as expected, it's going to go up to 766,000. So, makes sense. All right, second-to-last question, which job postings appeared in both 2023 and 2024? In this case, it would be intersect, and that it returns rows common to A and B with duplicates removed. Making a little bit of an assumption in this case they want the duplicates removed. So, I copied the query above and replaced it with intersect. Let's go ahead and run this and expect this one to actually be the smallest of all the queries in that, yes, only 18,761 job postings are repeated. Now, that gets us into our final question of what job postings appeared in both years, preserving duplicate counts. In this case, we're using intersect all, and that returns rows common to A and B with duplicates preserved. So, I'm copying the query above, adding intersect, and then adding all. I do expect this to be greater than that 18,000 previously, and it goes up to 21,000. So, almost 3,000 postings were even duplicated on top of repeating between the years. All right, so that's a quick crash course on set operations. Once you understand the basics of what union, intersect, and except are doing, you can better apply it to different business situations that you need it for. All right, you have some practice problems and now go through and get more familiar with these set operators. In the next lesson, we're going to be moving into some functions, specifically some text and also functions to handle null values. With that, see you in there. All right, welcome to this lesson on text and null functions. And we're going to be going through, well, two sections. First, we're going to go through some common text functions that I find myself using with a final example, and then also with null functions with its associated own final example. Now, real quick, this lesson, and actually this entire course, is not, well, all-encompassing, right? So, like text functions, there's hundreds and, well, not hundreds, there's tons of different text functions that you can have and access. And then over here, looking here on the left-hand side inside of DuckDB's documentation of functions, there's a variety of different types of functions. We did, and we will, we'll have two more lessons on functions after this. We've covered the majority of this, so you're going to have a good understanding already of what functions should be used for what. And I really can't stress it enough that you should use some sort of AI like ChatGPT if you've ever get into a situation that you're curious about, well, is there a function for that or not? You can simply query this and it will provide you back on what you can do. So, actually moving into text functions, I find that there are six major types. We're going to first go through just some simple examples using only a select statement to basically test out each one of these, and then we'll move into that final example. For this, I'm going to be starting in a new SQL file. Additionally, I'm going to go ahead and connect to MotherDuck Data Jobs Data Warehouse. First up is length and count. And for this, we can count the number of characters in a string. Now, I have two listed here of length and car length or char length. But going to the documentation, what we can find is a lot of these that exist in other languages, they'll sometimes give them, in this case like car length right here, it has the alias for length. And they'll have it's the same function, it just alias differently, so that way if you

### [10:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=36600s) Segment 123 (610:00 - 615:00)

come from another language, you can just stick with your same syntax that you're used to. Anyway, in the select clause, I'm going to enter that length function and then just put any old text phrase in there. I'll stick with SQL. Whenever we're going to run this, we get, as expected, three. S Q L. Changing this to char length, this also works and gives us the three cuz it's the same function, it's just aliased. Next up are case conversions. We can use this to upper and lower case different letters within a text or all of them specifically. So, let's test out that lower. Running lower, we can see that gives us and then all lower case. And then I also want to try upper, but I want to provide it actual lower-case values. We'll do capital S and then Q L. Then running this bad boy, we get all S Q L in upper case. Next is substring extraction. And this is really good if we need to extract out maybe something at the beginning or left of our text, the right of our text, or if we need to pinpoint something specifically based on a number of characters, we can use substring. Anyway, we're going to start with the left function first. And this is the syntax as you provide it the string or the column, if you will, and then the number of characters you want to extract from the left side. So, in our case, we're just going to do two. And it provides S Q. Similarly, I can try out right. Whenever I run this, I get Q L. Next, we'll do substring. Now, with this, this is going to take multiple more than two arguments. It's going to take not only your the column or the text you're providing, then from there, the number of characters over that you want to start the extraction. So, in our case, we'll say, "Hey, we'll keep it at two. " So, it's going to start at Q, and how many characters do I want to extract? In this case, let's say we want to extract two more. So, whenever we run this, we'd expect to get QL. Okay, and we do. And we can even change it up if we want. I can extract just one character if I want starting in that second position. In that case, I'm only going to extract Q. Next up is concatenation, and we've seen this before using the concatenation operator. So, this should just be a simple review. In this, this is used to concatenate text functions and put them together. In this, I specify I concat, and then I could do like SQL, and then maybe do a dash, and then add in functions. Whenever I go to run this, I get SQL functions. Now, I personally like to operate this with the concat operator. Um no pun intended. And it basically I feel it makes it a little bit more readable, as you don't have to have this function, and you just see the symbols right there. Anyway, whenever we run this, we're going to get the same results of SQL functions. Next is trimming functions, and we can use Commonly, I would just stick to trim, as it trims white space that is on the left or right side of a statement. In this case, there's an extra space to the left of SQL, and also to the right. You could also use something like LTRIM or left trim and R right trim, and this will remove the extra white space on a certain side. But, typically for me, I find just trim is good enough. Anyway, first, let's just provide the select statement with the SQL in it itself, and it's just the string, so we can see where it is. All right, so now we have reference. I mainly do this because there's no quotes around here, so you can't really see the empty space that it's getting rid of. Whenever we go to do now, the trim function, we're just going to wrap that all in. Now, running this, we can see that it did trim it because SQL had this extra space right there, and now it's all the way over. Last up for text functions are replacement, and they have two options here. You have like your standard replace, and then you also have regex replace. Anyway, let's just go over replace first, then we'll get into regex. Anyway, in here I can run the replace function, and we run it on what we want to run it on. We're going to stick with SQL. And we'll say, "Hey, I want to replace the Q with an underscore. " So, when running this, we get S_L. Now, this is just this replace function. You give it a text, in this case to replace with, and it tells you or you tell it then what you want it to replace with. It's very standard. Now, that brings us to using regex now for replace, and this one's more dynamic. You give it the regex or the regular expression in order to pick up on, which is sort of a little bit of coding, if you will, and then what you want to replace it with. Now, regex is beyond the scope of this course, and if you don't know regex already, I recommend just use ChatGPT. Like, in this case, I could say, "Hey, give me the regex replace formula in SQL to extract out any email domains, such as gmail. com. " I also add this, "Assume the input column is valid email addresses. " So, it then goes through and provides this formula

### [10:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=36900s) Segment 124 (615:00 - 620:00)

for us. We can see this formula right here uses regular expression in order to pick up on that, well, usually an at symbol, that is the domain, and then it fills it in at the end with, well, a blank character. Anyway, let's test it out. And of here, I place it and I'll just replace email with an actual value of data. nerd@gmail. com. All right, let's go ahead and run this. Boom. And this extracted out the email domain I wanted was Ford. Anyway, you get the point. You can use ChatGPT for this. So, let's get into our final example. If you remember back to our lesson on case expressions, we got to this portion where we were going through and using case expressions in order to match up based on a job title looking for certain keywords and then providing whether it was data analyst, data scientist, data engineer. So, running this, this does do the same problem that we saw before. And that's that some of these are categorized as other, right? Because they don't have data or analyst or engineer scientist. That's fine. The one that does bother me though that I want to fix is like right here. We have data analyst, [clears throat] right? But it's marking it as other. Clearly that one is a data analyst role and that has to do with this text is doing a like more of a perfect match. So, what we want to do is a lower case of this job title. Now, I'm not a fan of going in and putting lower into what is this? 1 2 3 4 5 6 different times just to then try to detect this. Instead, what I'm going to recommend is or what I would do in practice is I would first start with a CTE. With this, I'll call this with title lower. And then inside of the CTE, I would extract out the columns that I'd want. Specifically, job title and then also job title but in lower case so we can do lower job title. Now, also for good practice, right? In order to make sure that we're providing the most succinct data out of possible to this, I would just throw in a trim function to remove any white space to hopefully speed up the computation if necessary. So, I'm going to wrap inside of the lower, job title in trim. And we'll alias this job title clean. This from [clears throat] job postings fact. Down here, I need to update this. We're no longer coming in our normal query from there, we're coming from title lower. And I now need to update this because I use job title here. It needs to be job title clean. So, I'm going to copy this and then paste it into there. The last thing to do is we don't want to detect any more of these uppercase like data and analyst. So, I need to go through and change all these to just lowercase all the way through. All right, it's updated. Let's now run this query. And this ended up working. And this one right here, right? We have this full-time role data analyst with SQL, New Jersey's locals only. We did match up on it and we're able to mark it as data analyst. So, these text functions can come in handy especially for bucketing and categorizing data. Our last type of functions to discuss are null functions. And we're going to be going over two of the most common that I find that I'm using for null functions, and that's null if and coalesce. Now, this category of null functions I do want to give a caveat of is under the bucket at least DuckDB gives it utility functions because these functions which coalesce and null if fall into below are difficult to categorize into specific function types and are broadly useful. Anyway, starting with the first one and it's null if. And this takes two arguments, expression one, expression two, which could be values or in our case that we'll get to columns. This returns null if the values are equal. So, in the case of putting in 10 or 10 gives null. In the case of 10 or 20 gives 10. So, let's go ahead and test it out. Inside of here, I'll put 10 and then run it, we can see that they are equal, so therefore it gives null. We make it not equal, they provide the value. With that value being that first expression. And it does say expression because you could technically put something like a mathematical or something in here. And then that case it's going to evaluate and still give us that 10. So, what's the use case for this null if function? Well, let's take a look at our first Well, let's just this data right here where we're querying our salary year average column and our salary hour average column from job postings fact where both of these are not null. Basically, I don't want to have all those null values pop up. Anyway, here it is below. And what we would want to do is say there was a case that we had zeros in these values. Meaning, let's say that we had some data in here such

### [10:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=37200s) Segment 125 (620:00 - 625:00)

as salary year average and they listed the salary as zero. Well, what we could do is we could do null if and then provide that column of salary year average along with then zero and then close it. And then we'll also do the same for the salary hour average column adding that null if checking if it has zero. And in any case, if there were I execute this below, in any case that it has zero in this column, it's going to replace it with null. And this is extremely valuable if say we were getting, you know, like the median of these columns and we wanted to make sure that they're all the zeros were removed from this. I can do the median from this column as well. In fact, I will so I don't have to do a group by. And then when I run it, I know that it's using values without zeros in it. Now, disclaimer on this example is that I know I rewrote this query to look at the lowest salary hour average. In this case, it's the lowest is eight. If I reword this to see the lowest salary year average, whenever I run this, the lowest is 15,000. So, there's no zero values in our data because can't help myself, but I cleaned up the data to remove any of those zero zeros. But, if you find yourself in cases with this, this is the great use of the function for that. Next up is coalesce, and this returns the first non-null value from Well, this is a list of expressions. So, in that first case, if we're providing a the null A or B, A is going to get returned first. In the second case, 100 will get returned because null nulls is a Well, nulls and then it'll be the 100. So, trying this out, I'm going to just put coalesce and then zero one two. Running this, we would expect, right, the first non-null value, so it'd be zero. Changing the zero to a null, I would expect now the one, and one it is. And then finally, changing that one to a null, I would expect just the two to return because it's the first non-null value. Now, how is this typically used in the real world? Well, we've seen this before, right? Here, I'm once again querying that salary year average and the salary hour average from job posting facts. Oh, and I added in a where clause to basically show me the data so that I can see the salary hour average and the salary year average um when it's not null. Anyway, let's go ahead and run this. What we can do [snorts] is we can use coalesce, and we could use maybe salary year average as that first expression to check, and then salary hour average as the second expression to check. And why we want to do this? Well, if you remember previously, we did a We were trying to convert to a standardized salary. Remember we wanted that salary hour average converted to a yearly salary. So, this is a great use case for this. So, in here, I'm going to put in that coalesce. I'll list salary year average first, and then salary hour average next, and just multiply this times 20 2080 or 2080 hours which are in a year. And then, right? Normally, you'd list a third expression if it's not if you wanted to replace it with something like you could do we could do missing salary. But, we want to do text because this is an integer column or this is a double column, so we wouldn't want to do that. We also wouldn't or it's not necessary in our case to just list null because null is default already. So, we're not even going to list a third expression in this case. So, whenever I go to run this, I get an error because I have a period right here and not a comma. So, I change that after salary hour average. Let's run it again. Okay. Pretty cool, right? Now, it's going through and it's doing this operation by getting the first non-null value from these columns and then doing either in the case of the 20 multiplying it times 2080 or in the case of the 110,000 just using that value first. Now, if you recall back to the case expression lesson, we ended up drafting or providing this for the final example. And what we did was in the CTE, we used a case expression to identify whether it's null or not and then in that case, make this into a single column with now using that formula for salary hour average to multiply times 2080. Anyway, basically what I'm trying to say is this entire case expression is now a coalesce function and we did this CTE because we didn't want to have to write this over and over again inside of this other case expression down below here where we then went through and bucketed the data into missing, low, mid, or high. But, scrolling up, we can see that this Okay, so previously, we were using this entire case expression. Now, this has all been limited to this single function right here of coalesce. So, I can just go

### [10:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=37500s) Segment 126 (625:00 - 630:00)

ahead and copy this, and then I'm going to remove this entire CTE because we're no longer going to need it. And instead, I'm going to replace standardized salary with this. I'm also going to replace it here, and here. I'll give this the alias though of standardized salary so we can use it inside the order by. And then I just need to update the from to job postings fact. Okay, I think I got everything we needed for this. Let's go ahead and run it. All right, not bad. It went through and did what we wanted it to do of combining the standardized salary. We can see that we did have one hourly right here and it converted it into that yearly salary, and then we got the appropriate buckets for each. And then, where necessary for the null values, it filled it in the bucket as missing. So, this simplified our query a lot further with this function, and makes it a lot more useful. All right, so that is null and also text functions. We now have some practice problems for you to go through and get more familiar with these functions. In the next lesson, we're going to be diving into some other functions, specifically window functions, a really popular topic when it comes to using SQL for aggregation. All right, with that, I'll see you in the next one. All right, welcome to this lesson on window functions. And window functions let you, well, the name's given because they let you look around like a window at other rows while you keep your current row intact. And we're going to get to a demo here shortly. And these functions are very important to data engineers because they allow us to keep our data modeling intact while performing or adding in advanced analysis. Anyway, enough of me yapping. Let's get into this. We're going to be looking at what window functions are, the different type of functions we can use with it, and some real-world use cases. So, let's go back to that analogy I was just given where window functions let you look around at other rows while keeping your current row intact. Previously, we learned about aggregate functions and that is using group by. But, the results of this aggregate the results into a single row typically. Or multiple, depending on what you're grouping by. Window functions, though, keep this all intact and provide this aggregation at that row level. And because of this, you can do a lot of powerful things with it. Let's go ahead and demonstrate this using a simple example where we're going to be performing aggregation only to count the rows and then doing a window function to count the rows. Open up terminal and I'm going to connect to our mother duck database for data jobs. So, the first thing I'm going to do is just a simple aggregation of counting the rows. We've seen this before, right? We're going to select then we're going to run a count star to count all the rows and this is going to be from the job postings fact table. Make sure to end it in a semicolon. All right, let's go ahead and run this. As expected, we got back 1. 62 million rows. Now, let's use this with a window function where we preserve all the different rows. Now, for this, I want to preserve all the different rows, so in order to see all the rows we're going to list the job ID. And this is from job postings fact table. Let's go ahead and just run this to see what's being shown. So, with this query, right, we're getting every single job ID. What I want now is to be able to next to on each of those rows provide that total count. Well, it's pretty simple. We'll enter this in. We're going to run a count and then star. And then from there list the over keyword with an opening and closing parentheses. This keyword right here is the most important as it signals that this is a window function. And whatever you put in parentheses, we're not going to do anything right now, defines the window we want to look at. Right now, we didn't define anything so we're going to look at the entire data set. So because that, let's go ahead and run this. And with this one right, we have the query up at the top. We can see that for each job ID we then have the count of 1. 6 million. And this repeats all the way down. And so this allows us to get any number of different aggregations, not just that count star, and preserve that row level data. So why are window functions important for data engineers? Well, they let you add context without destroying all those rows, which is basically your job. You're typically going to be building data models and these pipelines need this row level row level data. And window functions allow you to keep this along with adding additional insights. So let's break down the syntax and we're going to be using it where it's primarily used within that select statement. The first portion is the window function, what we want to actually use as the function for this.

### [10:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=37800s) Segment 127 (630:00 - 635:00)

Now we're going to dive in just a second into this, but there's three major types of buckets I find for window functions and that's aggregates, row and rank, and navigation. As far as the aggregate goes, you've seen a lot of aggregate we've done already whenever we did our group by lesson, we're just doing it in the context of window functions. Anyway, more on this in a bit. So after we define what is our window function, the function we're going to use, we then start by creating our window function by using this over keyword and then in parentheses after this and that makes it a window function that's all you need that's the bare bones for it. The partition by is an optional next parameter that divides the rows into groups or partitions and then this function runs separately within each of those groups. For this we're creating basically separate windows for each unique value. Next up is the order by and this one is also optional but often required in that defines the sequence of rows within each partition. This is especially important when we get into those ranking functions and running calculations that require to be in a certain order. Now more advanced window functions may even use this additional uh addition inside window functions where you can specify the rows between or the range basically selecting what number of rows up or down around a row. Anyway, that's going to get too complex. We're not getting that that's beyond the scope of this. Just know that it's available to modify window functions even further if you want to. So getting back to those three types of window functions. You're already like I said pretty familiar with aggregate functions. We've run through all of these of average max min sum and count within a group by context and applying to window functions up here I feel will be pretty intuitive. Now other areas where it shine are things like row and rank so we can use window functions give the rows row numbers or even rank values such as highest to lowest salary. And the third other type of function that's useful is navigation in that especially those lag and lead you can actually use window functions to pinpoint a row above or below it in order to maybe do some sort of comparative analysis to see how salary is changing over time. Anyway, getting ahead of myself, let's actually get into demonstrating this. So, let's get into breaking down partition by. We're going to start with this example. Let's say that we wanted to not only keep the value of what is the hour average or salary hour average for all job postings, but we also want to look at what was the average of that for all job title shorts. Well, partition by is going to help us solve this because it helps divide the rows into groups or partitions specifically here on the job title short column. And then the function will run separately with each within each of those job title shorts. So, what we can get at the results of this is we preserve our row level because they're window functions. And then it goes through and calculates that in this case data analyst has an average hour hourly of $50 an hour whereas data engineers have an average of around 100. And it preserves that the row level. So, let's do this. We're going to go ahead and add a select. I'll add in the job ID column, job title short, salary hour average. And then we'll get into defining our window function. We're going to be using the average function for this. Now, we want to perform an average on the specific column, so we can do that in this case. So, we add salary hour average into here. And then we go ahead and add an over. And over defines the window function itself. So, now it is a window function except we're going to keep this over in here to demonstrate what would you would get in this case. Then from there we'll go and add it a from and the job postings fact table. Okay, let's go ahead and run this. As I ran this, took almost 2 seconds. Probably should put a limit in there so it's not causing it to be so much. But anyway, what we can see is we do at least have one value here. Well, what's happening though is all these values are outputting the same thing of 47. 39. Anyway, just to prove that this is the average over the entire data set, I'm going to go ahead and just run a select statement on just the average, not using window functions on job postings fact. And as shown, it is that 47. 39. So, what that window function is doing specifically with this over and then parentheses, we're looking at the entire

### [10:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=38100s) Segment 128 (635:00 - 640:00)

data set. And then from there, it's performing the aggregation on the appropriate column. So, just over by itself is already powerful, but now let's actually get into getting that average salary, but on a job title short basis. And we're going to do this by adding in a partition by and specifying a job title short. All right, let's go ahead and run this now. All right, not bad. We're seeing that for the results for this for software engineers, their average is around $51. 17. And then scrolling down, we can see that senior data scientist is different at 56. 96. Anyway, I want to actually see a little bit more to see the variation and check out this data set more. So, I'm going to add in a group by and then do random to randomize this data. And then I'll also add in a limit 10. All right, running this now. And I got an error because I don't know why I was doing group by with random. That's not what I want to do. I want to do order by random. So, I'm going to just change this group to order. And then running this again. All right, sweet. This demonstrates it better. And we can actually see the difference between this where senior data engineers are at around $60 and 56. This makes sense. Now, I am going to take this one step further. I'm actually going to delete this query down here. We're not using this at all. Let's say we were also wanting to not only get the salary by the job title short, but also by companies. So, I could specify in here, just to demonstrate it better, the company ID, and then inside of the partition by add in that company ID. Now, whenever I run this, it's going to partition by the job title short and company ID. And in that case, some of these companies don't have any salaries associated by it. So, only this one here from company ID 6334 has an associated salary. Anyway, I want to just look at ones that do have salaries with this. So, I'm going to add in a where salary average uh or is not null. All right, with that added in, I'm going to go ahead and now run this. All right, so pretty cool here. We can now see, based on a job title short value, and the company ID, what is that average salary? And then we could compare it to what is the salary of a posting. Very powerful analytics that we can provide our end users with, all with just providing or with using a window function. Let's now get into order by. And we're going to be looking at order by just by itself, without in addition to partition by and then order by. And then we'll move on to a more complex case combined with. Anyway, order by I find most simply is best to explain how it's used with ranking functions. If you recall, there are three major types of window functions I found of aggregates, row and rank, and then also navigation. We're going to just focus on a one example real quick of row and rank. So, let's say we have this scenario. We have our job postings here with are job title short, and then the salary hour average. And we want to rank these values in order of their salary, where the highest salary, as one of job 105, is $105. And that would be the highest ranking one of one. So, in this case, we're going to use order by within our window function with the window function of rank in order to define the sequence of the rows with in this particular partition. Our case, the partition is the entire data set. And so, for the final results for this, we'll have that rank hour salary column, and it'll provide that numerical rank all the way through based on what it is on that salary hour average column. So, for this, we're going to go ahead and jump into VS Code. I'm going to copy our previous query that I had above. I'm just going to paste it down here for what is to work with. And I'll just go ahead and preemptively change this to salary hour average in descending order, cuz that's what we're going to be doing. We're going to be ranking this. So, let's go into here. We'll keep all the things of job ID, job title short. Company ID, we don't really need it, so I'm going to delete it. Keep that salary hour average. Then from there, we're going to redo this window function. In this case, we're using the rank function, and the rank function doesn't require any arguments inside parentheses because it's really specific to the window function that has everything in it to specify it. We then from there have our over, and inside of here, we're not going to do partition by just yet, right? We're going to do just order by, and we'll specify that salary hour average. We'll give this an alias of rank hourly salary. All right, so let's go ahead and run this bad boy.

### [10:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=38400s) Segment 129 (640:00 - 645:00)

All right, so inside of here, we did that order by, so we have the highest hour salaries up top. Notice here uh that our ranks though start at 26059, and that's because, right, we didn't specify to order by the salary hour average in descending order. So, inside here I'll update our query to do that in descending order, then we'll run it again. And bam, now we have it to where we have a rank based on the order of the salary hour average. Now, one quick note, this order by portion right here, technically, you could I'm going to comment it out real quick by pressing command {slash} and then running this without order by in there. In that case, it is still that window functions because as a order by in there, it is going to order it, but you can't always rely on it, especially when you have more complicated window functions or maybe you have certain limits in there. So, it's always good practice to still list that order by because we don't necessarily know how it's going to output. So, now let's put these two concepts together in order to look at how we can perform not only a partition by, but also order by. Now, for this example, what we're going to do is we're going to get a running average of our hourly salary. And that running average is going to be based on that order by that job posted date. However, we're going to go break it into a subset or partition by based on that job title short column. Just a reminder, right, partition by groups rows for these window calculations. And then order by defines the sequence of rows within each of those partitions. So, in this case, what will happen is we'll have, yes, the jobs put in an order for the window function to an order to have this average function run. Specifically, if we look at data analyst, that first entry is 60 uh dollars an hour. That is the average for that case cuz there's none before it. Now, when we moved into the second entry for data analyst, we now have $40 introduced. So, 60, the average of 60 and 40 is 50. And then moving on to the third for data analyst, it's 50. So, so therefore the average stays 50. Similarly, we can see for the data engineers, it starts at 95 with only one entry, so the average is 95. And then when we add that second in of 105, that average comes out to 100. So, it's partitioned by the job title short and then ordered by the date in order to perform this running average. So, let's go ahead and do this. I'm going to take what we have up here and copy this original partition by formula. I'm going to go ahead and paste it right into here. For this, I'm going to change out that job ID with the job posted date column. We'll keep that job title short. We don't need the company ID for this. And we'll also keep that salary hour average. So, we're still going to be performing that hour average of the salary hour average over our partition by, we're going to do the job title short. We're not going to do company ID in this case. We're just keeping it simple. And then now we'll add in that order by specifying for that we want to do job posted date. We'll give this the alias of running average hourly by title. And we're getting this from the job postings fact table. We're going to keep our where in here to filter by, uh, to remove any non-null values. And then for the order by we'll delete this random. And then I'm going to do it first by the partition of job title short. And then next for that job posted date of the final order by. All right. Let's go ahead and run this. All right. And first up we have business analyst cuz B is one of the first job titles. And we can see that they are in fact ordered in that job posted date column. Now, going through this just to double-check, right? That first entry of 17, I would expect Yeah, the average at that time is 17. Then, whenever we get a second entry in of 20, 17, the average of 17 and 20 is 18. 5. Then, when we get that third entry of 35, we now have three entries until the average of all three of those is 24. And it goes on so on for this. Now, if I wanted to, I could also add an and condition here where we look at just data engineer jobs by specifying the job title short equal to data engineer. Whenever I go ahead and run this, check that one. Similar thing here, starts at 64. 5, that's the average. Then, we add in 25. 50, and then the average of those two are 45 and so on. Now, I do also want to demonstrate this with our last example that we did with this order by ranking our salary. So, I'm going to go ahead and just copy this and then paste it in here. I think a really neat thing would be to not only rank this um in this case, right? We were ranking it

### [10:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=38700s) Segment 130 (645:00 - 650:00)

based on what was the hour salary, but we also maybe want to rank in different partitions based on that job title short. So, go ahead and add in a partition by specifying job title short. And then, I do want to order this correctly, so I'm going to go and go ahead and press uh command {slash} to uncomment that and then add in the job title short and a comma. All right, let's go ahead and run this bad boy. And it did do it correctly. I can see that based on business analyst, I can see the what it is for 1 through 10. Which is good, but I actually want to see it from what is the highest salary hour average to see different job postings. So, what I'm going to do is I'm going to change up this order by right here and put salary hour average descending order first and job title short second. Now, whenever we go ahead and run this, we can now see data analyst that its first one is at 391, and then data scientist 250. And then correspondingly, data analyst second one is at 242, and so on. So, it's pretty cool now because now we have not only ranked it based on this our salary, but also we've grouped them or partitioned by that job title short. One last note, this order by inside of the window function is completely independent from this order by here at the end. So, this window function will handle its calculation on its own, and then after this order by that happens within the window function happens, then from there this order by is then applied. And we know this based on what we learned in order of execution, where step five has where our step our select statement is run, which that is where our window function is. So, that happens first, and then step six is our order by, where it actually goes then and orders the columns. All right, so now we're going to walk into well, aggregate functions, but we're actually going to go through all three of these showing how or showing the most popular functions from each of these three different types of major window functions. Now, aggregate functions you've done before in earlier lessons, so I feel like you have a good understanding of this already. So, we're just going to look at some simple examples real quick. For these examples, let's go back to that second to last query that we were working with, where we had both partition by and order by, and we were doing a running average of the hour salary. Now, I just want to demonstrate a few of these. I want to demonstrate it simply first, so we're going to I'm going to comment out the order by. And in this case, remember if we did the average of this, and this is filtered for data engineers, this provides us the average of all those hour salaries, and then partitioned by job title short. Granted, we're only filtered to only one job title short, but they get the idea with this. Additionally, we can do things like finding the minimum salary. In this case, when I run it, we get the minimum of eight. I could also do the max. In that case, I get that of 221, which is the highest salary. Or, I can even do something like sum to sum up the all entire column there. And the sum of all the data engineer salaries are 337981. That's hourly. Anyway, I could see this even more useful if you're doing some sort of running minimum, running maximum, or running count or something. So, what we're going to do is go back to this and remove that order by or being order by being commented out. So, in this case, we'll just keep the sum in there, so we can do a running sum of the hour salaries. And we can see that as we go through for data engineers, that hour salary gets accumulated. Similarly, I can change this something like min. And for this, it will track what is the minimum as it goes through here until it gets replaced. So, it was 64. 5, and then it got to 25. 5, and then it sticks with this until it gets to the next lowest of 24. Then it finally gets replaced again by 20. Just like min, we could also do max. And this would keep that max value on what it is. Looks like it hit 70, and then stays with 70 for the remainder of the time there. So, these aggregation functions that we've previously learned about can be even more powerful when you combine them with window functions, and then can partition them or even order them by certain values to get even more specific results. Next up, we're going to look at row and rank functions. Specifically, we're going to be focusing on those first thing uh first three, looking at rank, dense rank, and then the row number. So, for this first example, we're going to scroll on up and we're going to use that third-to-last example that we were demonstrating order by when we were using the rank function with it. So, I'm going to go ahead and copy this and then go ahead and paste it into here. All

### [10:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=39000s) Segment 131 (650:00 - 655:00)

right. Now, previously, we were just using the rank function. I'm going to put back in this order by. Okay, so let's go ahead and run this showing that rank function where we were ranking the salary hour average. Anyway, for this one, right, we weren't doing any partition bys and we were just going down and it was ranking it. Now, if you notice here, we get to this value for data scientist and senior data scientist where you have two values that are same with 200 and it says nine and nine. Actually, let's increase that limit a little bit to see what happens after that. So, I'm going to change this limit 10 to a limit 20 and then rerun this. Oh my gosh, this is not what I wanted. — All right, so you could see, right, we have a bunch of 200s and we have nine ranked all the way up to the 20 values. Let's go a little bit higher with the limit. Change it to 40. And that's not even enough. All right, so I ended up changing this to 140. That took me a second to find it. Anyway, if we see, we get down to this value here and to our values that are what, like 139 and 140. And we can see here that that's the rank, 139, 140, and it jumps, right, from 9 to 139 to 140. So, that's what has happened with the rank function. So, now, I'm going to go ahead and change this instead to change this to the dense rank. And whenever we run this, what we see is that when it gets to that well, 139th value previously, it just goes from nine to 10. And this, in this case, next the rank's rank after nine is not skipped like that last case. Personally, myself, I find myself using rank more than dense rank, but there may be situations where you'd have to use dense rank. So, let's move on to row number, and this is very common I find whenever I want to develop maybe a new ID column for a maybe a data set I'm building. Anyway, for this, what I mean by this? Let's go to Let's just select star from the entire job postings fact set. And for this, I'm going to just limit this to uh 20 results. Oh, and I do want to actually order this by the job posted date. Okay. Let's go ahead and run this. So, these are ordered by that job posted date, and as you can see, even if I order this by the job ID, you'd notice the same thing. It starts at the job ID of 4593. Let's say we wanted to give a new ID and start directly at one. Well, we could use the row number function for this. So, I'll keep all those columns on here, and then I'll add in row number, then we'll use over, and then from there, we're going to order by, and we'll keep that same of job posted date. Now, I'm going to go ahead and run this, and what we get here is that row number based on it ordered by job posted date, and it goes through it just starts at one and then goes through there. One note, uh these are out of order right here, and it probably has to do with that these are probably the same date and time that they were listed. Anyway, doesn't uh doesn't really affect it too much. This gets what it accomplishes what I want to happen of starting at one. I've also seen this row number used as a ranking function, because if you remember, whenever we looked at rank and dense rank, I'll go ahead and just run that query again. Remember, it assigned a specific or at least it had matching values, no matter whether it's rank or dense rank, for those ninth-place values. All of them were rank nine. So, you could use row number in that case to assign unique numbers for the rank. And actually, we can just demonstrate that real quick by changing this to row number. And then running it. And then we can see from the results it goes 1 2 3 4 5 6 7 8 9 and then between that 1 and what was it 139 it started assigning individual numbers. Note that these aren't sorted correctly. You would have to actually put this rank hourly salary into the order by to actually get it in the numbers that you want it in order. And so, I changed that up here in the order by and we get those in the correct order. All right, last up type of functions we're going to look at are navigation functions. So, for this we're just going to focus on, well, one, well, technically two. In that we're going to look at lag and lead. So, let's get into the scenario that we're going to be using lag with. And for this, I'm going to just go ahead and show you what the final table's going to look like cuz it will make more sense what or how actually we're going to use lag. Now, what I want to do is I'm trying to analyze based on a company, how does their yearly salary change with every job posting and specifically ordered by or, yeah, ordered by job posted date. So, for example, this

### [10:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=39300s) Segment 132 (655:00 - 660:00)

right? So, company ID 4493 they have their first posting at 75,000 and then their second posting is at 150,000. What we can do is we can use the lag function to get what was the previous value right here. So, that's what I'm doing. I'm getting that 75,000 here. Now, moving on to the next company of 4594 it has 90,000 for the first value. There's no previous value, so it's null. And then when it goes to the next value, it then can put in say it does have a value above this of 90,000 and it puts it there. So this lag function is in comp it's very powerful and it allow us to compare and maybe perform some time-based comparison on how a company is raising or lowering salaries over time. So let's build this query. So in here let's start adding our columns. In this we want job ID, company ID, job title, job title short, job posted date, and salary year average. We'll add in the windows function a little bit. This is from the job postings fact table. And we want to do this only where the hour or the yearly salary is not null. So we'll add that in. And then for this I'm going to limit the results to well, I know it needs to be 60 to be able to see the different uh kinds of things we want to look at it. So let's now use that lag function to get the previous postings salary. So I'll put in here lag. Then we'll put in over and then open the parentheses. Now how do we want to do this? Well, we want to do right partition by first. And like I mentioned, we want to look at we're looking at companies specifically. So we need to partition by the company ID. And we want to see how their postings are changing over time. So we'll order it by that job posted date. Now I do want it to be finally ordered whenever it shows it to me. So I'll put an order by at the end based on the company ID and that job posted date. Okay. Let's go it Well, let's give an alias real quick for this as and we'll call this previous posting salary. Okay, let's go ahead and run this now. And so let me I got ahead of myself. Lag needs one parameter. It got zero. So in this case, what are we trying to provide as the value that we want to look at previously. What we want to look at salary year average. So, we're going to specify that inside of here. Now, let's go ahead and run this. And we get our results. As we saw previously, right? We don't exceed the previous posting salary for this 4593 portion because well, that's the first posting. And then the second one, we do see 75,000 for the first posting. And then it carries all the way on through for the next company. Then I can even scroll down to inspect further. When we go from company 5494 to 5498, we see for the first [snorts] value, it's null. And then after that, it goes to 85,000 based on that previous salary. Anyway, that's cool. What was actually better than this is we want to see what is the change with each of these postings. So, what I can do here is I'm going to add another column. And I'm going to take salary year average. subtract what was that previous posting salary. So, I'm just going to copy this all right here. Paste it in and then change this alias right here to salary change. Okay, this is now, I feel even more informative of what happened here. And that now we can see that, okay, in that second salary of 4593, it increased by 75,000. Whereas for company 4594, we can see the change that goes on here where it goes up 22,000, down 22,000, stays about the same, and so on. Now, that's lag in that we're looking at values from the previous column. We could also similarly do this to change this to lead. And this would analyze instead of the next posting salary or the previous posting salary, this is going to analyze the next posting salary. And then I change this lag also to lead. And then whenever we run this, it does just the opposite. In that first posting gets what the next salaries value is, so 150,000, and then we can provide that salary change. I find myself, especially for time-based analysis, mostly using lag and not lead, but wanted to just demonstrate both cuz they're both so similar. All right, so that is window functions. You now have an understanding of how to go through and use things like partition by and order by to get even more in-depth analytical measures and provide it to your stakeholders. We have some practice problems for you to go through, and then we'll move into the next lesson after that on understanding how to use functions for nested data. All right

### [11:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=39600s) Segment 133 (660:00 - 665:00)

with that, see you in the next one. Welcome to this lesson on nested data. And nested data is also commonly called something like semi-structured data or even complex data types. Anyway, the gist of it is this allows you to store your multiple different values within a single column, so it's really powerful feature for you to know as a data engineer. In this lesson, we're going to go over the five most common types of nested data, and then from there, we'll dive into two real-world examples with our data sets, looking at arrays and also array of structs. Also, real quick, before we get into it, this is the last lesson before we get into our final project. So, super excited with this last portion we'll be covering on SQL. So, you're probably like, Luke, what the heck is nested data? Well, here's an example of what nested data looks like inside of a table. Take for example here this skills column. For this data engineer, there are multiple skills listed for its associated job. In this case, Python, SQL, and Excel. These are listed in what's known as an array. The other example I have here is on benefits. And this is using a struct data type. And we can notice this because it has those curly brackets around it. But in here we can specify three different attributes about this job, whether it's remote, whether it has health care included, or whether it has PTO. And then these values either true or false. But we'll come to find out that it's not just limited to true false values for this. Anyway, the main point of this is to showcase that we can now go forth with putting in multiple values inside of a single column. So let's quickly break it down onto what these five different nested data types are. And then after we go through these five, we'll then go a little bit further introing them, actually demoing how you can create them yourself in VS Code. First up is an array. It's also commonly known as a list. In DuckDB, you can same thing you can do to arrays you can do lists and vice versa. Well, for the most part. Anyway, these are an ordered collection of the same type values index, meaning in this case I'm demoing it with text values, but it could be number values, dates, or whatever, as long as the same type. You know it's an array because it has square brackets around it. Next up is a struct, also known in other databases or as records or rows, but most commonly known as a struct. This is a composite data type containing multiple name fields. Now here in this demo, this we have it around in this curly brackets. And so we have skill and Python. Skill is the field name, and then Python is the value that we're trying to store within that field. Similarly, level is the field and advanced is what we're trying to store inside of level. In most databases, the data type, specifically of Python and advanced, has to be what was the example here, has to be of the same type. So, these are both probably var chars or text values. And if I wanted to use numbers, both of them would also have to be numbers. If I wanted to use dates, both would have to be dates. Next up is an array of structs. And quite simply, it is basically an array with structs inside of it. Of the data types we're going to cover today, arrays and then also this array of structs are the most common I've found to be used in data engineering in order to store data. And that's specifically for storing nested data. All right, next up is a map. Also commonly called an object and dictionary, but most commonly known as a map. And it's known as a map because it stores key and value pairs. So, the keys are mapped to the values. Unlike the previous one we went over of array of structs, this is an unor- unordered collection. So, they're not maintained in a certain order. Anyway, this data type is, I find, less common used than an array of structs. So, we're only going to spend a little bit of time just for you to understand the basics of this. Last up is JSON. And this is a text-based serialization format for nested data. Now, the thing about JSON is it's not native to SQL in that you'll find JSON out in the real world. If you're working in web development, you'll see it all over the place. And JSON stands for JavaScript Object Notation, which JavaScript, once again, is well, uh, plenty used in web

### [11:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=39900s) Segment 134 (665:00 - 670:00)

development space. Anyway, the reason why I'm including here in SQL is it's common for you to receive data if you're accessing it via an API or something like that in a JSON format and then you have to unpack it or maybe convert it to something like an array of structs, which we'll do. This does have a similar format in structure whereas it does use the curly brackets and square brackets in order to contain data, but doesn't necessarily always follow the same rules. So now what I want to do, I want to jump into VS Code and I want to walk through actually building out each one of these data types. First up is an array. A reminder it's an ordered collection of the same type values and it's indexed being that we can access the value inside this array by a number that identifies where it is in this array. Going to get ahead of myself, let's jump into VS Code. So inside VS Code I'm going to go ahead and attach to our database and Mother Duck. So the first thing I'm going to do, demo those array, I'm going to create a select statement and to create the array we're just going to enter in the square brackets and then you're going to enter in we'll just enter in three values one, two, and three. I'll add a semicolon at the end, go ahead and run this. So it says underneath here main list values, like I said in uh DuckDB arrays are sort of synonymous synon- synonymous with lists. But we can see this data type underneath here and it says int32 and then square brackets. This shows that it's an array with those square brackets next to it. Anyway, let's actually change these to skills instead. So I'll change this to Python, SQL, and R. Whenever we run this we can see that the data type changed to varchar with the square brackets saying it's an array. And then I'll name I'll go also give this an alias of skills array. And now a lot more readable. So let's dive into creating an array but by using a function. And going to the documentation for DuckDB, we can see that all list functions work with the array data type. And that's important because we're going to have to use a list function for this, well, sort of. Anyway, I'm going to navigate over here to aggregate functions. And in here, there's a function with list where you provide args or arguments and it returns a list containing all the values of a column. This also has the alias, conveniently, of array_agg. That's what actually we're going to be using this case. Instead of using list, we're going to use array_agg. So, let's create columns full of data of our skills. So, first, I'm going to add Python and our column name is going to be skill. Then from there, I'm going to do an union all and then I'll use another select to add in SQL. And then finally, one more union all to perform the last select of R. All right, I'm going to go ahead and run this just show what's going on here. And in here, I created a table and with the column name of skill and then it has those skills of Python, SQL, and R. What we want to do now is take that column and convert it into an array. So, what I'm going to do is now convert this all into a CTA. So, I'm going to say with skills as and then open parentheses, indent this all over, remove the semicolon, and then put the closing parentheses. We're then going to query this and first, I'll just query skill from skills just to demonstrate that we did do this correctly so far. All right, we run it. Run just fine. Provides us the same table with Python, SQL, and R. But now, what I want to do is I want to use that function we just learned about of array agg. And then I'm going to wrap that with skills and give this the alias skills array. All right. Now, whenever I run this bad boy, we get all of our skills in an array. We can tell that this is because it has the data type varchar with the square brackets. Now, if you remember, we could also use list. If you haven't seen this before with aliases, I do want to demonstrate this real quick. In that, instead of using array agg, I can just put in here list instead. And whenever I go ahead and run this, it's going to provide exactly same results. Does exactly the same thing. But I'm going to go ahead and change this back. Anyway, I want to get to this last point here where it says that this is an ordered collection of same type values that are indexed. How do we access this via an index? Well, we have an array now. I want to access via an index. So, I'm going to wrap this also in a CTE. We'll call this one Well, I'm We'll call it skills array. I'm going to have to do some changing around here. We'll say as we'll open that parentheses, take away

### [11:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=40200s) Segment 135 (670:00 - 675:00)

the semicolon, add those final parentheses, indent this over, and then I'll change this skills array to not be that as well. I'll just change it to skills. So, now I'm going to select skills, and that's going to be from skills array. Let's go ahead and run this to make sure that I actually built this correctly. All right, so it's working. It's still providing that array of skills where we have our SQL and Python. Now, let's go in and actually access them using the index. So, we can do that by after our skills right here, what I'm going to do is I'm going to put a bracket and then specify what skill I want to get. We'll put one. We'll say, "Hey, this is going to be our first skill. " So, let's go ahead and run this and see what we get. So, with this, our query returns SQL as the first skill. But, hold up real quick. Let's actually run this a few times. Selecting again, pressing shift enter, this time we get Python. Let's try it again, now we get R. See, I'm running this a few times, we can see the skill is constantly changing. And you may be like, "Luke, I thought you said that an array is an ordered collection of same type values. " Keyword here being ordered. Well, if you go into the documentation on aggregation, let's look at this array agg function. As you recall, right? Anyway, it returns a list This function is affected by ordering. When I click this, it specifies that in order to specify an order, we have to use Well, we have an aggregate function, your argument, and then from there you use an order by. So, what's going on here is we're using this array agg function, which is an aggregation function. And yes, it's putting it into an array that is technically ordered, but we didn't specify a consistent order to follow. That's why every time we run it, it runs different values. SQL, R, Python. So, what we can do is what the documentation said is we can add inside of here order by, specifying what we're going to order by. We'll order by Well, there's only one thing to order by, the skill. And in that case, it's going to do it in alphabetical order. Now, whenever we run this, every time we run it, it's going to start with Python. Additionally, let's modify this to add in the second skill and third skill. So, I've added that in. Now, let's go ahead and run this bad boy. And this consistently will stick with Python, R, SQL, which is alphabetical order. Here I am running it over and over again. It's maintaining consistent. It's maintaining ordered. All right, next up is a struct, and this is a composite data type containing multiple named fill fields. So, let's get into creating it. I'll create a select statement, and then from there we're going to do squirrely or a squirrely curly brackets. We'll provide the first field name of skill and then give it inside of it the value of Python. And then from there we'll give the next field name of type and we'll classify Python as well, programming. We'll give this the alias skill struct. Then go ahead and run this. Inside of here we can see that it is of the data type skill struct where the skill name is of varchar and the type is also of varchar. Type is a reserved keyword in SQL so it puts it in double quotes to make sure that we're using it in the proper context. You can also see it's a keyword because it turns blue here when should be white. Anyway, not really best practice for this but just for example purposes we're going to keep it as type. Anyway, I want to do something similar that we did with our raise. I want to use a function in order to build this. For that we're going to navigate over the documentation for struct functions and go to struct pack. This creates a structure containing the argument values. The entry will be the bound variable name. So in this we'll suffice in the parentheses the name of the field and then after a colon and an equal sign you give it the value one for it. So let's do it. I'm going to put in a select statement and from there specify struct pack. Open up a parentheses. I'm going to break this down into lines to make it a little bit easier to read. We'll specify the field name first of skill then that colon and equal sign and then the value we want to put inside of it. And then we'll just go ahead and put the next one in. Specifying type for the field name and programming. We'll give this the alias of just S for right now cuz we're going to expand upon this in a little bit. Anyway, running [snorts] this, we get underneath here that yes, it is of type struct, skill is varchar, type is also varchar, and then it displayed underneath here. Now, similar to what we did with arrays, I want to show now how you can access things like Python or

### [11:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=40500s) Segment 136 (675:00 - 680:00)

programming inside of a struct. So, for this, I'm going to wrap this all in a CTE. So, we'll say with skill struct, and we'll open up parentheses. I'll indent this all over, and then put that closing parentheses, and then from there, we'll select Right now, we'll just do a select star. And that's from skill struct. All right, selecting it all in order to run it. And I got an error, mainly because I forgot to define my CTE uh with that as keyword. All right, as is in there now. I'll go ahead and run this now. Okay, so we're getting back everything that we saw before, but now we want to go in and access values like Python and programming. Well, what we can do is we can do this with dot notation. Specifically, we saw from that table, let me open back up, that the column name is S, and then skill is the field name. So, what we do is we would do S. skill, and then we can also do something like S. type. So, now whenever I go ahead and run this, I get um basically unpacking that struct by having Python and then programming under what it is. Now, based on you seeing it like this, I do want to take this a little bit further in that right now, you could imagine we could have a table with multiple skills and multiple types. So, let's go ahead and create this. So, as a starting point, I'm going to scroll up though, and I'm going to select this portion where we created this table with Python, SQL, and R. Anyway, we're going to go ahead and copy it and then come down here underneath and paste it. And then use shift tab to move it back in. So, right now we has we have Python, SQL, and R characterized under skill. What I can do now is add in something like programming as type. For SQL, I'll specify it's query language and for R, we'll make it programming. All right. Now, going ahead and running this, now adding in a semicolon, we can see now that we have a table. So, now let's get this table into a struct. So, what I'll do first is I'm going to put this all into a CTE. We'll call this with skill table as and then open up parentheses, close it up. From there, we will select and we'll use that struct packed uh formula again. And similarly, we're going to do that same thing where we provided the field name, skill, and then the alias up here of it's well as that's confusing. Uh we're going to change this to skills cuz this is a skills column and we'll call this types column. Uh I don't know if that's necessarily proper English, but we're going to go with it to make sure we can distinguish between the two. Anyway, this up here is skills and we would specify it here as skills. And then we want the field name of type and we want to be the values from types. And this is all from our skill table. All right, let's go ahead and run this. I forgot a semicolon and I have a syntax error at or near type. What I think I did, yep, I did it. I forgot to put a comma after this. All right, let's try to run this again. All right, pretty sweet. We see in here we have the data type of struct with skill varchar, type varchar, and then we have each one of those on their own separate row. All right, next up is an array of structs. And this is an ordered collection of struct records, or better said, this is an array of structs. So, let's start by creating this one. We'll create a select statement and then square brackets. That's going to start our array, and now we need to insert our structs in here. So, for that, we'll insert our squa- curly brackets. So, it's by the field name of skill and then the value of Python. We'll also do the field name of type and the value of programming. We want to add a another struct into here, so I'm put a comma and then go down to the next line. And so, I'll put in SQL as a skill and the type is query language. Just going to do two. I'll give this alias skills array of structs. Let's go ahead and enter this. Putting on a semicolon and running it. I have a typo. I used a period up here for some reason. This should have been a comma. All right, so let's try to run this again. All right, so now we have the value inside of here and we can see from the data type it's a struct and then the skill and then the type varchar in parentheses and then it's right next to this of the square brackets symbolizing it's an array. All of this struct is inside of an array. Anyway, let's create this using formulas. You know, like I like doing that. What we're going to do is we're going to

### [11:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=40800s) Segment 137 (680:00 - 685:00)

build on that second or that last example we were just working on. I'm going to go ahead and just copy this and then paste this down here. If you recall, whenever we run this, this has a multiple rows where each row is its own array. Sorry, I had that backwards. I mean, that it's its own struct. So, we want to make an array of all three of these structs. So, this is actually pretty simple in that we just need to wrap this entire portion right here in our array function. So, I'll go ahead and type out array_agg, open up parentheses. I'm going to just format this by indenting it over, and then put that closing parenthesis for array_agg. Now, whenever I go to run this, we can see that we get all the values within one row, and that it's of struct with the skill and type, and this is all within that array, so it's an array of struct. Now, we're going to take this one step further. How do we access these items in here? How do you access things like R, programming, or query language? Well, we'll put this all into a CTE. We'll call it skills_array_struct, and then we'll give it an as, and then open parentheses and close parentheses at the end of here. Then from there, we're going to select. We're just going to do select star from right now to make sure we built this correctly, and that's from skills_array_struct. All right, and I have a semicolon right here. Caught that. Going to delete it. Now, let's go ahead and run this bad boy. Okay, we're getting the exact same results, right? It's all on one line. Let's now go and access it. Well, this is all First of all, this isn't all an array. Remember to get access inside of an array, we used need to use that number that index notation to access the one, two, and three item. So, I'm realizing I didn't give this an alias up here for this um array of structs that we created, so I'm just going to call this array_struct. And then here, we're going to do array_struct, and we'll just access that first item by doing one. Let's go ahead and run this, and we can see we get that first item. It's the skill of R and type of programming. I can also do it or add to this where we access the second and third item just to show how we can break it out. And here we can see how we've broken it out with that first item, second item, and then also that third item. Okay, how do we access now inside of here something like SQL or query language? Well, remember we're going to use the dot notation for that. After the square bracket notation for inside the array, now we're accessing the inside the struct. We can access something like the skill. We'll access the type for this one. And we'll keep that last one just number three. All right, let's go ahead and run this. All right, sweet. So, we can see that we got the skill back from that first item, which is R, the type back, which is query language for that of SQL, and then this one just maintains as the struct of array. Sorry, just a maintains as the struct. Next up is map. And as I mentioned before, map is also known as things like objects or dictionaries, but it's not as popular as things like arrays or array of structs. So, we're just going to go over this briefly so that you understand what it does. But, unlike the other ones we went over, I don't really expect you to have it memorized on what functions you can use for this. Now, in order to have a create a map, we actually we have to use a special keyword of map in order to create it. You can't just use that square bracket or curly bracket notation. And DuckDB has a few different ones for this on how you can create a map. We're just going to go with the simplest case. So, for this we're going to do select and then specify map, and then from there insert curly brackets. Now, inside of here we're going to specify that key first. Previously we were talking about the field name, but then now we're using something called a key. And it's a little bit different in that we can't just type in skill by itself. We have to actually classify this as a text value by putting single quotes around it. And then from there we can add in the skill of Python. We just have one key value pair in this map to start. I'll go ahead and run this and we can see we have in here that it's of the map data type with varchar and then there's the key is skill and it's mapped to Python. If I were to run this without the single quotes on it and try to get to run, I'm going to get this right here, this binder reference column skill was not found because the front column was missing. Mainly the problem is skill has to be classified as a text value and the key has to. These text values will always be the key. Anyway, I can go ahead and add other things to this map as well such as type of programming.

### [11:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=41100s) Segment 138 (685:00 - 690:00)

Now, we can go ahead and run this bad boy and we can see now that skill equals Python, type equals to programming. Now, it's important to understand in how we built this in this structure, you can't I can't repeat this skill again. For example, if I were to add in I just I'm going to copy and paste it, a skill let's say I wanted to add of something like SQL. Whenever I try to run this, I'm going to get this where map keys must be unique. So, this map that we've just built has to have unique keys of skill and type. Now, how do we access the values inside of here? Well, we'll create this as a CTE calling this skill map then I'll bring up parentheses putting this all inside of here. I'll get rid of the semicolon and give this the alias of skill type. Go ahead and close parentheses and then now inside of here, we're just going to do a select star to make sure this is working run correctly. This is from our skill map. So, running this and adding in a semicolon, we get our values just like we saw before. Now, how do we access something like Python or programming? Well, we're going to specify in here, we'll specify first that skill type. That was the same as select star previously. What we're going to do, very similar if you've see worked with dictionaries in Pythons before, you would use a square bracket and then inside of here you specify the key you want to access. So, we're going to access the skill key. Whenever we go ahead and run this and actually insert in a semicolon, we can see we get the skill of Python. Similarly, I can go in here and add that skill type and add in uh type for this one. All right, let's go ahead and run this. And bam, we get both back, Python and programming, accessing both values within our map. Now, real quick, I want to talk about the differences between map versus array of structs because I feel like they're both very similar in that, you know, map has that key value pairs and array of structs has that field names and then you can nest it down further. Well, both of these have methods to access the data inside of it. We're going to get to using unnest on array of structs here in a little bit, but we demonstrate with map how you can use the key to access inside of a map. Now, other differences include ordering inside of it. That array of structs, there is an order once you set it inside of there. Yes, we whenever we created that Python R and SQL example, it switched up, but it's going to stay like that. Whereas a map, it can change. It doesn't matter. There's no way to maintain an order inside of there. Array of structs are great for things like analysis and also maintaining relationships as you can build out a really solid structure. Maps on the other hand are great at uh storing some sort of metadata or doing lookups with it as it has a key and you can plug that key in and get it back. So, it's really good at those type of values where you want to look up a certain value. Now, inherently with how these are built, array of structs are naturally good at joins and filters, whereas maps are not. And ultimately, why we're going to be using array of structs is it is much stronger for analytical use cases, and so that's why it's become more common in data engineering world to be used because you're build stuff for data analysts. Map, on the other hand, much more limited in that it's not as good for analytical use, but it does have some other positive benefits that we discussed. Last up is JSON, — or JavaScript Object Notation. And this is a text-based serialization format for nested data. It does have a very similar structure to array and also structs in how you can actually build things out. However, you can get pretty complex with what you receive back in JSON. Anyway, the main point of this in understanding you're not going to build pipelines to put data into JSON. Instead, you're likely going to be receiving data in this JSON format, and you'll need to be able to deal with it, to parse it out, and clean it up. All right, so let's go ahead and create our first JSON data. In this, this is text-based, so it's going to be all within single quotes. For this, we're going to maintain simplicity and just create a, if you will, an array inside of a JSON. We'll first with specifying skill, and that will any text value needs going to need to be in quotes here. And for the first skill, we'll specify that is Python. For that skill, Python, we're going to have then have the type, which is programming. Now, we're going to take all this, and

### [11:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=41400s) Segment 139 (690:00 - 695:00)

we're going to cast it to JSON. And we'll give this a alias as skill JSON. All right, let's go ahead and run this. And we got an error because I forgot to put double quotes around this programming right here we have as text. All right, let's try to run this again. Okay. And we can see we have it formatted in the format of JSON. You can also see this as instead of doing this cast formula to cast it as JSON, you could see the function of to JSON and then wrapping this all inside of parentheses. And then running this, we're still getting that JSON. However, I'm not as big as a fan of that, especially in DuckDB, because it well, it has these escape characters in here because it got converted back to string, if you will. So, I'm going to just change this back to casting this as JSON. It's much more readable. So, moving forward with that example with this, I care less about using a function to extract values out of here, if you will. But instead of maybe I want to now transfer this into another data type. So, let's start by making this into a CTE. We'll call this raw skill JSON. Then we'll wrap all of this in parentheses. Removing this uh semicolon at the end. And for this, we're going to select, you know, I like to just put the value in here for the time being to make sure we built this correctly. And that's from raw skill JSON. Sounds a little French. Yep, still doing the same thing that we were doing before. So, now we're going to be putting this into a struct. Now, this is going to get a little complex with the uh formulas we're going to use here. But I don't want to get too bogged down in it. I mainly want to wrap your head around that this is possible. So, first we're going to use that struct pack formula. And I'm going to go ahead and remove skill JSON cuz we're not going to use that. You remember, you list the field name first, so it'll be skill. Now, for this, we need a special formula. So, navigating over to DuckDB for JSON processing functions, we're going to be using this JSON extract string. It extracts varchar strings from JSON at the given path. So, you provide it two arguments. You provide the JSON and then the path. So, let's go ahead and use that function. We'll specify JSON extract string. We need to first by specifying the JSON, so we'll list skill JSON in there. And then the path. What do we mean by that? Well, what we want to do is we want to access the path to skill right here. This is special JSON notation in that we're going to specify this dollar sign or this currency symbol. It means start at the root of the JSON document itself, and then dot notation to access skill. So, we're going to go to the key name skill. All right, then I'm going to close parentheses. Let's just go ahead and run this to show that we got that first skill of Python out of there. And we did. We have it right here, the struct of skill Python. So, we have this first part of this skill Python list. Now, get the type programming as well. So, we're going to insert here a comma, go on down, put in type, and then use that JSON extract uh extract string function for the skill of JSON, and then using that currency symbol to start at the root of the JSON document, specifying the type. All right, let's go ahead and run this now. And bam, we've now pre- or rebuilt the JSON that we had up here, right? We've now rebuilt that into an appropriate format that can now be used in SQL. Now, I do want to show just quick, we're not going to go over in depth, but imagine you had a JSON value that was structured similar to an array of structs. So, you have multiple different structs inside of this similar array, but it is a JSON value. Well, if you remember from our previous from that we were just doing, we did that struct packed unpack that skill and type into a struct. Then we can wrap that into an array. And so then running this, we can see that we get this now in that struct array format that we want. Or better said, sorry, array of structs. I think I went and got that backwards. But point of that is mainly to just showcase there are methods and functions possible for you to get your JSON data into a more usable format in SQL that's much more queryable. This tech This JSON data, which is basically in almost a text format, is not going to be great for an analytical use case and is not what we want to keep it in. All right, so that wraps up the five nested data types. We're now going to go into two

### [11:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=41700s) Segment 140 (695:00 - 700:00)

real-world examples working with our data set, our job postings fact table, along with skills to make them into arrays and then from there even go further and use it and combine it into array of structs. Let's now get into our first of two final examples where we're first going to work with arrays. In this example, we're going to be putting our skills from our skills job dim table, or actually from our skills dim table, into an array. So, here's our problem statement. We want to build a flat skill table for co-workers to access job titles, salary info, and skills all in one table. As a quick refresher, in order to get those skills, we're going to not only have to use the job postings fact table, we'll have to connect to the skills job dim table, and then to the skills dim table finally to get those skills. So, let's get building this query. For this, I'm going to select, and then I'm going to leave these columns blank for the time being as I want to put in what tables we want to pull from first. We want to come from the job postings fact table as JPF. Then, we want to retain all of those jobs in the job postings fact table, so we're going to use a left join to connect that still skills job dim table. And we'll be connecting on the job ID from both of these tables. Now, we need to join in one more table after this. We're going to be doing a left join for this as well. And this is to the final skills dim table, which we're going to alias as SD. For this, we're going to connect on the skill ID from the skills job dim table and the skills dim table. All right. So, now we have our tables and what aliases we're using for this. I'll add in the job ID, job title short, salary year average, and then the skills themselves from the skills dim table. Now, we're going to eventually put the skills into an array, but I want to show you first what this actually looks like with what table we're making right now. So, this table took about 6 seconds to load, a little bit of time, probably put a limit statement on it. Anyway, what we can see is in the case of like job ID 54 or 4594, which repeats multiple times, it's because it has the skills of SQL, Python, and R. But, we don't want for this final table for these skills to be duplicated cuz we just are not duplicated or broken out amongst multiple lines. We just want to make all the skills within one array. So, in order to do this, we're going to use array agg as we've seen before. And we're going to alias this as skills array. Now, if we go ahead and try to run this, well, I'm going to this to show you what error we're going to get. We're going to get this, a binder error in that column job ID must appear in the group by clause or must be part of an aggregate function. Well, the issue is this, right? Array agg, agg means aggregation. It's performing an aggregation as it does this. So, all these columns need to be listed in the group by. So, I group by re-induct DB. So, I'm going to just go ahead and just add all. In other databases, you have to have type out all these different columns. So, now let's go ahead and run this. And I forgot to remove that other semicolon, which was right here. — Running again. This table took actually a little bit less time, only 3. 75 seconds, and we can see that these skills are put within array for all of these different ones. And scrolling down the bottom, it retains all 1. 62 million rows of those job postings. So, what I want to do now is let's put this into a temporary table. And we're going to then shift roles in how you could actually go about analyzing those skills that are in an array. So, first we're going to do that by doing a create and I'm going to do or a replace to make this out of the important. And then we'll put temp table. We'll call this job skills array. And we'll then do as. Okay, going ahead and running this, we built the table in about 3 and 1/2 seconds. All right, so let's shift gears and now we're going to do from the perspective of a data analyst, analyze the median salary per skill using that table, that temporary table we just made. Now, right temporary table, understand it only exists in our environment. We're just doing this for demonstration purposes. You would most likely ship this up to your own data warehouse where you'd want it. I'm just going to do temporary tempo temporary table to make it easy. Well, the first thing I want to do is I want to demonstrate how we can actually extract out the skills and get them row by row so the way I could do an analysis like this to analyze the median salary. In order to do that or demonstrate that, we're going to do a simple example first. I'm going to come up to the top where we were doing arrays initially and

### [11:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=42000s) Segment 141 (700:00 - 705:00)

I'm going to copy this right here. And this is where we were pulling out the different parts of an array and manipulating with it. So, coming down to where we are at the bottom, I'll paste it in. And what I'm going to do is I'm going to delete all this portion up to skills to just show what the output is currently for this query. All right, so this just has SQL, Python, and R in array. How do we unnest these values? Well, for this, that was a little bit too much of a hint, but we have unnesting, which going into the documentation, we can see that we have an example here for unnesting a list or in our case an array. Remember, list and arrays are sort of synonymous in DuckDB, but arrays are generally more acceptable across most databases. And we have unnest for that, and then we also have you can do unnesting on structs. So, super powerful function. So, let's test it out on our example. All we're going to do is previously we had skills in here. I'm going to run the unnest function on this. And then let's go ahead and run it and bring it. Bam! It breaks out those skills on individual rows. So, that's what we want to do. And specifically, do with this job skills array table that we just built. So, we don't need this example anymore. I'm going to go ahead and delete it. For this, we're select the columns that we want, which are job ID, job title short, and salary your average. And then we're going to run the unnest function on that skills array. We'll give it the alias of skill. And this is all from job skills array. I'll do a good practice finally and actually put a limit statement in here. So, we'll run this. And inside of here, we can see how it goes through. And now it Well, we have duplicate job posts as expected because it's now breaking out those skills into individual rows. So, this exactly how we want our data. But, remember we have one more step we want to do. We want to do from the perspective that analyst analyze the median salary per skill. So, what I'm going to do is I'm going to put this all into a CTE. We'll say with flat skills as and then have that CTE within there. We want to select from it that skill column and then also the median of salary or average. We'll give this the alias median salary. This all will be going to be coming from our CTE of flat skills. And we want to order this by median salary in descending order. Also, I just forgot Remember, we're doing aggregation, so we need to do a group by on that skill column. All right, let's go ahead and first add a semicolon. And let's go ahead and run this. And I have an error because I forgot to get rid of one of the semicolons here, specifically that in the CTE. It's like I've done this multiple times wrong within this course. Also, I forgot to put a comma at the end of skill. Let's try this one more time. So, we get this back and you're probably like, "Luke, what the heck's going on? You don't have any median salary values or values. " Um I'm going to look up here and I see what I did. Mainly, whenever I use the CTE, I left that limit 20 in there. And that was limiting what was in our CTE. We want to look at all jobs. So, I'm going to go ahead and remove that, run that again. All right, so now with our query we can then see the median salary based on these skills. Right now we have Fedora, and Debian up at the top spots. All right, so not bad. We just demonstrated how we could put all of the skills within a single table, like we did up here, using that array aggregation function. And then, with it in this manner, the data analyst, or whoever your downstream customer is, has the option to then un-nest these values pretty simply, if I might add, in order to do deeper insights on the skills, if they need to. All right, so let's get into our final example. In this we're going to be building a similar flat table like we did previously, but in this case, we're going to be using an array of structs. So, for this, we're going to be building a flat skill and type table for co-workers to access job title, salary info, skills, and type in one table. If you don't remember, we're going to just query the skills dim table real quick. And in it, we can see that it has not only skills, but what type of skill it is. The goal of this is we're going to use an array of structs to have something like the type as the field, [snorts and clears throat] and then sequel, or the skill, as the value in that field. So, you know, I don't like starting from scratch, so I'm going to come up here to where we created our previous job skills array table, and I'm going to go ahead and copy it. And then we'll paste it right on top of this query, because we don't need this anymore. For the time being, being we're not going to create this, because I'm going

### [11:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=42300s) Segment 142 (705:00 - 710:00)

to be testing building out this query. But this has the majority of what we need, right? Has the job ID, job title short, salary average. Only the skills, we do need to add the type, and we need to make it into array of structs. So, scrolling up to our array of structs example on the intro when we're going through this, I want to reme- recall your memory and how we can create that. What we do is in our case we're going to create or use that array agg function, put it inside of our struct pack function because it's array of struct. And then in here we're going to specify we'll specify the type first and then the skill after this. So let's do that with ours. I'm going to go ahead and remove this alias right here and then delete this out. We're going to first add in that struct pack function. And then inside of here we're going to first specify the skill type and assign it the type from our skills dim table. So we need to give it the appropriate alias before that. We'll also do the skill name. And this is also from the skills dim table. We'll alias it we'll alias this as skills type. All right, let's go ahead and run this to see if we get what we want. And this took about 6 seconds to run and we can see inside of this that it did go through and successfully create that array of structs which where you can do it by inspecting that data type it has here. But also we can just look at it and see that it is an array of structs with a skill type programming, skill name in this case R. All right, so not bad. So now similar to last time, I'm going to go ahead and uncomment this temporary table. I'm also going to rename it to job skills array struct and this has everything I need. I'm going to go ahead and run it and build this temporary table. That took about 6 seconds to build. So now let's get into actually unnesting it and seeing how we can look at this data. We're going to be doing this analyzing it from the perspective of the analyst to analyze the median salary per type of skill. So we're going to be able to use that type field name to actually analyze it because we're going to unnest it. So the first thing I want to do is just to see how to unnest this all. So we're going to list all the different columns we need such as job ID, job title short, salary year average, and then skills type, which is that array of structs. All right, so we know well already we can use a nest on this. So, we're going to go ahead and just put a nest on this. And this is all going to be coming from that Jill Scott job skills array struct temporary table that we just created. Okay, let's take a look at this to see what actually we're getting outputted out so far. All right, so with this what we see is yes, the jobs are getting broken down now out because now I can see that the software engineer job has skills of go, JavaScript, TypeScript, HTML, and so on. So, we've at least unnested that portion. But, how are we going to now access the values inside the struct? Well, if you remember, we're going to use that dot notation. So, we'll do that skill type here. And we'll give this the alias skill type. Similarly, below this we need to actually get the skills out. So, we'll do the dot notation on this unnest specifying the skill name. All right, let's go ahead and run this. All right, sweet. This now unnested this, and we can now see on each of the different rows the different skills types along with the skills name. So, let's now take this further, right? We want to get the median salary per skill type. For this person portion, I'm going to just come up to the top and steal the query that we did previously and paste it in below. This was the query that we used to analyze the median salary per the skills. But, what I'm going to do is take this whole portion that we just analyzed and found was correct here. And I'm going to paste it inside of the CTE. Indent it over and remove that semicolon. Okay, in this case, right? We're trying to do the skill type. So, I'm going to change the skill into skill type. And we'll also need to change what we're grouping by of skill type. All right, cross fingers. Let's hope this works. And it does. We can see from this that um I probably should have ordered this, but it's only 11 values. So, things like libraries, specifically like Python libraries or other languages libraries, has the highest salary at 140,000. But, we can see that things like programming is strong in the middle at 125,000. So, not too bad that we went from this table that we could provide our stakeholders with that had what out of a single row based on job postings and then have that skill and type inside of it that they again then can go forward and still analyze attributes about the skills. All right, it's your turn to give it a try. We have some practice problems for you to go through and now get more familiar with those different type of

### [11:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=42600s) Segment 143 (710:00 - 715:00)

nested data types. In the next lesson, we're going to be jumping into our final project, specifically looking at Git commands. And so, we're going to be diving deep into that and understanding what we're going to be building on the project. All right, with that, I'll see you there. All right, welcome to the second part on Git and GitHub. This one is going to be focused primarily on Git and specifically on branching. Now, branching is not only going to be imperative to us getting through the second project, but also it's just important for us to know cuz Git branching lets you create isolated lines of work so you can build, test, and even fix pipelines without breaking what's in your production line. Data engineers rely on this on a daily basis in order to safely evolve schemas, ETL logic, and infrastructure while multiple changes and teammates move in parallel. Now, for this branching, we're going to cover three main commands. The first one is Git branch, lists your branch, tells you also how to create a new branch, get switch, which allows you to switch between branches. This was previously known as get checkout. And then finally, get merge in order to, well, merge your branches. So, for all this, we're going to walk through some simple examples and then also discuss how we're going to be integrating it into our second project. Before we get into what is branching, I do want to do a quick recap of the Git commands that we learned last time. Previously, we would learn that there could be three areas where you keep your files. Your working directory, where you actually work in, type your code. Your staging area, where you're putting your changes in there before you get to committing it. And then there, your local repo, once you actually commit it. We learned commands like Git add to get your working directory contents into your staging area. And then Git commit to commit what you've done to that local repo. Now, from there, we explored GitHub, where that is our remote repo, and we can use commands like Git push to, well, push those that local repo up to our remote repo. And then say there are changes on the remote repo from a partner or another teammate, we can use Git fetch to get them into our local repo, and then Git merge to get them in our working directory. And conveniently, we learned that instead of having to use Git fetch and then Git merge, you can just use Git pull, which is an all-in-one command to get it from your remote repo into your working directory. Now, for that, previously, we were assuming you were working in all of a single branch, specifically the main branch, or master as it formerly was. We renamed it as main. And this branching in Git is crucial for you as a data engineer in order to collaborate with others. This allows an independent line of development that allows you to work on features, developments, or even bug fixes without affecting your main code base. These are like isolated workspaces that allow you to safely test things out and experiment and make sure your stuff works before you then push it into production. We're going to implement the use of branches in project two as we go through it and this will get us more familiar of how it's actually used. But we'll get on to more into how we're going to be handling this in the next video on the project two intro. So let's walk through a simple example of what you'd see typically with having a use case for different branches. As you saw previously, we worked in that main branch and this is where your main project actually resides. Previously, we'll say we're on version 0. 1 and we're starting now to develop version 1. 0, which has multiple different features. So not only do we create a development branch for what we're developing, but every additional feature for that [snorts] new version, we would start its own branch for as well to keep separate cuz you probably have people working on different features. So let's say just for this case, you have two different features all you're working on for this new version. Now let's say feature two finishes before feature one. Well, we can then merge feature two into the development branch where we're working on version one. Also during this, let's say that we get a bug that is in our main branch and we need to fix it. Well, we create a branch for hot fixes in order to fix this bug and then we would merge this into our main branch and this case will now be at say version 0. 2. But what about that develop branch, right? You we're working in it currently, we need it to be updated with what's in the main branch. Well, we can merge that bug fix also into our development branch. And so our development branch not only has now

### [11:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=42900s) Segment 144 (715:00 - 720:00)

those bug fixes, but also has feature two implemented into it. Now at some point, let's say we get done with feature one and it's done. Well, we can merge it into our development branch. And then from there, we can implement it into our releases branch. Releases branch are just the final area to put it before we're going to actually merging it back into main to do any final touch-ups. So we may find some things that we need to fix and we fix them. Then from there, we merge them into our main branch and now this becomes version 1. 0 that we can use. Now, now that we're done with 1. 0, we can start this cycle all over again and update our development branch to start version 2. 0 for the features we're going to add for that. So in this section, we're going to be going over four new commands of Git branch and then also Git switch. So let's dive into VS Code and start implementing this. All right, so in here, I opened up the version control or source control tab. I haven't committed any to my local repository or remote repository recently. So let's go ahead and do that to clean things up. So I'm going to add all these files by doing Git add and then period where that period is selecting that current directory. And now I can see that these are all staged changes. Then I'll run Git commit, use the M flag, specifying in double quotes that we're adding the problem files. So now we all we need to do is just push this up to GitHub. All right, so our repo's all cleaned up. We can see right here that we're up to add problem files. Once again, as a reminder, we've been working in that main branch. So I'm going to go ahead and clear this. How do we see what branches we have? Well, we can run the command Git branch. In here, it specifies main. Main is our only branch and then it has an asterisk. This symbolizes your head. It's a special pointer in Git that refers to the current commit you're working on. So, not only is on that branch, but also when you make a new commit on a branch, that head automatically moves forward to point to that new commit on that branch. And anyway, with that, we're going to be adding inside of our sequel. engineering folder, we're going to be adding another folder for our second project and implement or build a readme for this. Or at least start building it for this. So, what we're going to do is create a branch that we're going to have to delete, but we're going to create a branch in order to create this readme. And this is just for demo. I wouldn't expect for just a readme you create a branch, but you get the point. So, to do this, we type get branch and then the name of the branch. I like to start mine with either something like feature, but we could also see something like develop if I'm developing like a data warehouse or maybe bug fix if we're doing some sort of bug fix. And that just denotes what this major overall is about and then I'll do a slash and then define what this is. We'll just say this is project two readme. All right, so I'm going to go ahead and press enter. Now, when I run get branch, I can see that we're on main branch where head is at, but we have this new branch of feature project two readme. Now, if we need to get to our head over to that project two readme branch, we could use get switch. And then specify this feature branch. It tells us we switched to branch feature/project two readme and running get branch, I can confirm this. Now, let's say we want to delete this branch because we're done with it. Don't worry, we're going to recreate it after this. But, we can use get branch and then specify this D flag and that means delete and then from there specify the name of project two readme. However, if you try to run this, you're not going to be able to do this right because you can't delete the branch that you're on. Remember I'm going to do run get branch. We're on that branch. So we need to switch to main first. So we'll run get switch main. Then from there, I'll run delete of that branch and it tells me delete branch feature project two readme was basically deleted. Now when I run get branch, I can see that I have only main. Now let's create this branch again, but this time we're going to create it and also switch to it in one command. With this we run get switch instead of get branch. We specify this C flag, which stands for create. So we're going to create and switch to the branch that we're naming right here of feature project two readme. And it tells us we switched to that new branch. Whenever I run get branch, I can see that we switched to that branch and we also created it. Now I do want you aware of this command which we it was previously used and that's get checkout. And this

### [12:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=43200s) Segment 145 (720:00 - 725:00)

is synonymous, if you will, with get switch. I can get checkout main and it switches to that main branch. If I run get branch, I can just verify this. Yep. But get checkout is the traditional command and get switch is so much more powerful because as you saw, we can not only switch to a branch, we can also create at the same time. So for our purposes, I don't really think you need to memorize get switch. Also with this, whenever you type in something like get switch feature, I can just press tab and it automatically fills in the rest of that branch. Pretty cool. You don't have to remember all that. So I'll go ahead and press enter. We're now switched in that. Run get branch and to that I'm on that feature branch. Now, let's get into making changes on a branch. Right now, I can run get branch and confirm that I'm on that feature branch. What we want to do, right, the purpose of this was to create a readme in our project two. So, inside of here, I'm going to create a new folder, and we'll do that two underscore data warehouse or DW mart build. I'll go ahead and press enter, and then inside of here, I want to create a readme. So, I click add new file, and we'll call this readme. md. That's very standard. All right, I'm going to open up also source control now, and we can see that we have changes in our working directory. So, this is looking right. Let's go ahead and add a title to our readme. I'm going to add a with single hashtag for H1 header, and then I go ahead and add this title of data warehouse and mart build. This is our production ETL pipeline. All right, so, I know I have changes, right, because I see that this white dot is over the file. Also, I press command S or control S if you're on Windows. So, I know everything is saved and up-to-date. Now, what we're going to do is we're going to stage and commit these changes into our current branch. To do this, you know, we have do run get add then period, and this adds our readme to our staged changes. And then, we'll run get commit with the M flag, specifying that this is the add the project two readme. All right, so, now we want to push this. Now, we can't just run get push. I'm going to run this. As this says, "Hey, fatal. The current branch feature project two readme has no upstream branch. " In order to push the current branch and set the remote as upstream, use get push and then set upstream, specifying the origin as where the feature branch is. So, we need to run get push, and then you could do this {dash} set upstream, but honestly, the easier and more acceptable one that we saw previously, right, is {dash} u. And that means set up stream. So, go ahead and run this to push it up there. And I mistyped our feature branch. Specifically, it's project two read me. And I realized I got completely confused. You have to specify origin first, and this is specifying the name of your remote repository. Remember that is origin. And then from there, the name of our remote repository. So, I'll do feature and then press tab to add in project two read me. All right. Now, we have it all up to date. All right. So, this went through and actually pushed it up into GitHub. And conveniently, check this out, right? If we were doing this branch as part of a pull request, which is very common in data engineering to submit pull requests for new features, it gives you the link to now go forward and create that pull request using that branch. And clicking that link directs you right into GitHub. You can just go in and create the description of whatever your pull request is if you were submitting a pull request. Anyway, we're not going to do this. This is out of scope of what we're trying to do right now. But, I do want to show you what actually updated on GitHub. So, if you are following along, go into GitHub. What we can see is right here, we don't Well, one, we're on the main branch. We don't have that two {underscore} data warehouse folder. But, you can click this down arrow, and this shows us our different branches. And I can actually, if I want, switch over to our feature branch of the project two read me. And I can see it inside of here, while also navigate into it and see in what we added to it. Anyway, just going back here, you probably did see this up here, right? We could go forward in the UI to compare and perform that pull request and ultimately merge what happened here. But, we're reserving that for the next lesson. We're not going to do pull request. That's just out of scope. Let's now get into merging, right? We've made those additions to our feature branch. We now need to merge it in. Now, there's a few different types of merging, but we're going to focus only on the two most common, specifically a fast-forward merge and then a three-way merge. We're going to start easy with the fast-forward merge and then we'll

### [12:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=43500s) Segment 146 (725:00 - 730:00)

move into the three-way after that. All right, so diving into this. For this type of merge, this is when in your main branch, there are no new commits on that branch. As we can see, we have commit A and B, nothing ahead of it. But, on the feature branch, we have done C and then D. So, what we can do is we can perform this fast-forward merge since we're on this. And this just pushes all those different commits of C and D onto our main branch. This is the scenario we're currently in with our readme that we created. So, let's go ahead and get to work doing this in VS Code. Now, I do want to point out this graph down here at the bottom real quick. In this, this is actually somewhat showing what we saw previously visually what I was showing of how we have our main branch and then our feature branch that we're doing that's ahead of the main branch. Well, in this case, this feature branch is ahead of our main branch. And just visually so you can also see this, right? This is right here, this orange portion, that's what the origin or remote main is at, also with our local main is at as well. And then we're scrolling up here, we can see that the origin also has our feature project two readme branch as well up to this point. And our local one is on feature project two readme. Anyway, the whole point is that we're ahead of main. Now, in order to perform this merge, we need to get back onto that main branch. So, what we're going to do is we're going to switch to main by doing get switch and then main. And then we notice right that we now have some changes that are no longer there anymore because we switched to main and we're back some revisions. Now, it's always good practice to make sure because we're on main that we are up to date with our remote branch. So, I will do a get pull origin main, but I know it's up to date. There's nothing that's happening. It says already up to date. We're good to go. But, this is just best practice. So, now let's perform a merge. We're going to run get merge and then we'll list that feature branch {slash} and then I'm going to press tab to fill it in. And we're going to merge our feature branch into our main branch. So, the first thing it does is it says it's updating these code numbers which correlate to the appropriate commits of the main branch and then of the feature branch. And then it says this. It says fast forward because we're performing a fast forward merge. And below it says it's adding the readme file and there's one file change with a one insertion or one row or one line, which we can see right here. So, now visually let's go and check this out right here where it says, "Hey, we add project to readme. " Remember, that was part of our feature branch, but now that is part of main. Which that main correlates to our local repo's main. However, we are one commit behind our origin main. Which was we did when we did added all those problem files at the beginning of this lesson. Now, you do see up here this outgoing changes and this is just telling us that we're right now we're out of sync of our remote repo. And so, we need to perform that push. So, just to be clear, our local repo is up-to-date at least where the main has performed that fast-forward for local repo. So, we're going to run get push and although we did to specify origin main earlier, actually we didn't have to do this because we had already set the upstream by using that U flag earlier for main that the upstream that we're pushing to always or pulling from for main is origin main. So, we don't necessarily have to specify it here if you don't want to. Anyway, this went through and this updated our two commits, specifically that add project to read me and the add problem files. And from there it added that read me with one file change and one row insertion into it. All right, so with this get push, what it is saying here First, I'm going to start at the bottom. This is saying, "Hey, we went from our old commit on origin main to our new commit. " Then from there, it specified that we pushed it to our remote repository of SQL engineering projects. And then this, which is a fancy way of saying that no objects needed compression, everything we sent just fine. So, now I'm going to clear this all. What branches do we have left right now? So, I run get branch. We have our main and we have that feature branch still. We don't need that feature branch anymore. So, we can actually delete it. We do this by running get branch with a D flag, which stands for delete, and then we list that feature branch. Then when I run this, it deletes the branch. I can check by running get branch and I see that only main is left.

### [12:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=43800s) Segment 147 (730:00 - 735:00)

Now, however, if I go into GitHub, where our remote repository is, right? If I refresh this, this feature branch is still here listed under the branches, and we don't need it up here, either. We're not using it anymore. We only want to just stay on that main branch and don't have any confusion. So, we can delete this by running the command get push and then we're going to tell it up in origin of in GitHub to delete. So, we're going to run with a delete flag and we'll specify that we want to delete that feature branch. All right. So, with that specified, we'll go ahead and run that and it tells us that branch was well, deleted. Let's check it. Refreshing my browser. Now, when I click down on the branches, only man main is in here. So, let's now jump into performing a three-way merge. How are we going to do this? Well, what we're going to do is quite simple. We're going to create another feature branch and then on that we're going to perform a commit. Specifically, we're just going to add our first file for our project. We're just going to add one line to it. It's going to be super simple. Anyway, once we get to get that done, we're going to push that up to our remote repo. So, we'll have that main and then also that feature branch on our remote repo. And this is going to simulate that we're working on this feature branch. Anyway, then what we're going to do is we're going to make changes on the main branch and this will be simulated. We'll be doing this in GitHub because this is very similar to of what would happen in a real-world scenario when you're working with other teammates and they're making changes to main while you're working on a feature. And so, then what we're going to be able to demonstrate by doing these changes on both these branch is then performing a three-way merge, which here in this case is demonstrating how we're getting that commit C from our feature branch and commit E from our main branch into one final merge. So, let's jump into it locally and we're going to first need to create that feature branch that we're going to be using for this. So, we'll run get switch with the C flag for create, and then feature {slash} project two file one. All right, so we switched to that new branch of our feature branch. Now with this, all we're going to do is go inside of our second project here, add in a file. We'll call this 01 create tables. datawarehouse. sql. And then we'll add a line up at the top with a comment block saying step one, DW or data warehouse create star schema tables. Spoiler alert, we're going to be creating a star schema tables in that first uh part of the project, but more on that coming up in the next lesson. Anyway, that's not important. The main thing is this, we just created a new file and we have changes in this. We need to save it, so we're going to go ahead and select command S or control S if you're on Windows. So now we switch over to that source control pane. So we have changes we need to stage them, so I'm going to do get add and then period to do the current directory. They are now staged changes. Then I'm going to run get commit with a message of add file one, keep it real simple. And from here, we have one file created, one insertion cuz we added one line to it. So over here visually, we can see that we're ahead of our main branch on both origin and on main, and we now have this feature branch that is also not in our origin or in GitHub, so we need to push it. So now we'll run get push. We will need to add that U flag because this is a new branch and we don't have the upstream branch specified And then from there, we'll specify origin and then that feature branch itself. Okay, I'll go ahead and run this. So like we saw previously, we have our data up here that says that we basically pushed our changes that we wanted to. If we wanted to create a pull request, we could click this link right here and go into creating pull request. We're not doing that. So bam, it looks like everything is good to go with this. So where are we right now? Well, we did that commit onto our feature branch. Now, we're going to go into GitHub and we're going to add a commit there to our main branch, so that we can now get into after that three-way merge. All right, so I updated or refreshed GitHub. In here, I can see main and I see our feature project two file one. So, I know it has the change that we have. We're going to stay once again on that main branch. We're going to be editing inside of our two data warehouse mart build. We're going to edit this readme. I'm going to go up here to the right-hand corner and go edit this file, and I'll enter two new lines. We're going to enter in a to-do, and we'll say, "Hey, fill this in. " We're going to keep it pretty simple. Okay, I want to commit these changes to this main. And this is once again simulating like a coworker went in and changed this file. It automatically auto-generated this from Copilot for the commit message and the extended description. Looks good to me.

### [12:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=44100s) Segment 148 (735:00 - 740:00)

We do want to commit directly to the main branch. We'll be keeping that. You could select if you were trying to create a pull request. It would actually go through on here, create a new branch for this commit, and start a pull request. We're not doing that. Out of scope. So, we're going to keep that one selected and select commit changes. All right, so now we have those two different changes ahead and they're separate. So, now we're in a situation that we can get into actually performing that three-way merge. Before any merge, we want to switch to the appropriate branch. So, right now, if I run git branch, I'm on that feature branch. I'm going to switch over to main by going to git switch main. Now, notice whenever I do this, well, we actually had some changes that happened. First thing is this. When I switch to main, right, this file that we created is not in main right now. So, that's why it has this line through it. Uh I don't want to confuse anybody, so I'm going to go ahead and just close that. But, it doesn't exist in main now. It's still in the feature branch. Also, during that, I don't know if you noticed this, but this add to-do section is what we did in read me. And this got auto fetched or got fetched for us automatically. I had it set up in VS Code to perform Git fetches from time to make sure that I'm always up with origin. Remember, normally we're performing Git pull because that's a Git fetch and a Git merge. Anyway, the Git fetch is what is the portion that moves what we have up in the remote repo to our local repo, but not necessarily in our working directory. That's for Git merge. Anyway, if you don't have that auto Git fetch sets up, it's not a problem. You can just run Git fetch. And then run it, and it'll update. In my case, nothing happened. So now, before we perform this merge, we need to get both of these main branches in sync. Mainly, we have our origin main, which is one ahead of our local main. So, we're going to actually do this by running Git pull. And once again, we don't have to use that U specifier because we have already done it for our main branch already locally. All right. So, in this case, it said did updating. It did in that case also It performed a, if you will, a fast-forward merge by updating that main branch to the fast uh to the origin main branch. And it specifies what files were changed, specifically that readme with basically the couple of lines we entered into it. Anyway, we want to We're going to get to We want to perform that merge. I do want to show visually, though, what it looks like before. If you click auto right here, you can then go to change it from the current history item reference to view all history items reference. I'm going to click okay. And in this, we can see specifically that feature two project two file one addition, which also they have it in the origin or the remote repo as well. Okay, so we can see visually in this case that okay, this is perfect. This is setting us up for our merge. So enough already, let's go ahead and run get merge and then we'll do that on our feature branch for project two file one. Now over on the left-hand side, I could go through the UI to do this. We're not going to do it. Anyway, this does show us right now that we have those stage changes for our SQL file that we added locally. What this did here in the terminal is it opened up an editor. Remember, we set nano as our editor. So if your editor or if you didn't do that, you're going to have vim and operating with that. Anyway, in our case, what we can do is we can just enter a message into here. And I just said, "Hey, perform merge adding file one for project two. " And then down at the bottom, I can see first we need to write it out. So we'll press control O. And it says file name to write uh basically to our merge message. That's what I want to do. So I'm going to press enter. And then now that we have it saved, I'm going to press uh I want to exit out of it. So we're going to do control X. If you're in the vim editor, in order to exit out of it, you're going to just press uh colon and then WQ, which means write and then quit. So now we can see we have outgoing changes and that we're performing this merge locally. I'm actually going to even change this back. We're on all. I'm going to go back to auto and select okay. Not much change in the view. But anyway, locally I can see we're one ahead of origin main. So I need to now push this up to GitHub. So I'm going to run get push. Don't need to specify that U flag since we know what it is already for origin. And bam, it ran through and updated all the objects as it needed. And it updated our specific uh Git repository. Everything's looking good with this. And we can see visually now in our graph that we this three-way merge based on what happened here with our yellow little thing that goes out for adding file one. Pretty cool. Anyway, like last time, we

### [12:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=44400s) Segment 149 (740:00 - 745:00)

need to clean this up. Right now, I'm going to run get branch. We can see we have that feature branch. I no longer need it. So, I'm going to run get branch and then they run the D flag and then specifying the feature branch to delete. And that is only locally that it was deleted. So, when we're to delete the remote branch, we'll run get push and then we're specifying that origin and inside of it delete and then that feature branch of project two file one. All right, and it says that it was deleted appropriately as well. Going back to GitHub and making sure that it's refreshed, checking inside of here, we only have main inside of here. So, with that, we successfully demonstrated a fast-forward merge along with a three-way merge. Both of these are two of the most common type of merges you're going to encounter when performing merges. Now that we have merging down, we're going to go over conceptually what a pull request is because a pull request involves merging and specifically involves using branches. So, a pull request or PR is a way to propose changes to a repository. What it's doing or what we're trying to do is requesting to pull your changes into another branch, typically from something like develop or bug fix into that main branch. Now, we've already discussed the benefits of branches and how they allow you to perform test and develop on a separate branch without affecting your main pipeline or whatever it may be. But with this PR, we can do additional things in that we're provided an area to one, perform code reviews where team members can review what changes you're actually trying to do with this PR. Then from there, they can comment on specific lines and suggest improvements based on that PR. You can run automated tests before performing the merge, and then you can also do some sort of documentation if you want to explain what changes are happening and why. Anyway, this is all bundled, if you will, within a PR. So, let's go over a typical process for a pull request. First, this starts by creating a new branch for what we want to do. And this is branched off of main. From there, let's say we add feature one, and we merge that stuff into that develop. And then we develop feature two, which we also merge into there. Well, now at this point, we'll say, "Hey, all of our features are complete that we want with this develop branch. We want to now get this implemented into main. " So, we'll at this point submit this pull request or PR. And typically, this would go out and notify the appropriate stakeholder or owner, so they can start through their review process to approve it. And so, what you can do inside of something like GitHub is you can chat about it. And so, usually, your PR is probably not going to get accepted on the first round, so you may have some proposed changes, such as maybe implement change number one. And then there may be more chats involved, and uh change number two, maybe. All right, so now it's finally ready to go and get merged back into main. At that point, the person that's in charge of managing main would close the PR and merge your develop branch now into main to implement those different features that we've developed. So, you have all the requisite knowledge for performing pull requests now that we've gone through and understanding branches and how to perform different merges. So, we're not going to walk through another example with implementing a PR, but we do have the practice problems implement you walking through a PR. And so, you'll get practice with not only branches, but also pull request during that point. Last thing I want to cover in this lesson is around. gitignore. Now, this. gitignore file is something you should always be including in all the different repositories that you build on GitHub and specifically use Git to track. Anyway, this file tells Git which files and directories to ignore and not to track. And there's a lot of different use cases on why you may not want to track different files. Specifically, you could have files with sensitive credentials. This thing I have things like API keys or passwords within the files, those type of things, definitely don't want to go public on GitHub or tracked in Git. Additionally, large data files such as CSV files or even database files, GitHub actually has a limit on the file size that you can upload into it. I think it's up to like 100 GB. Well, just Google that. Actually, it's 100 MB, so way smaller. Um anyway, I commonly find that my data files are much larger than this and so, you'll get into errors if you try to even push these files that you have that are that large. Additionally, you may have other files like environment specific files, IDE settings, or even

### [12:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=44700s) Segment 150 (745:00 - 750:00)

log files that you just don't want other people to have and that may be specific to you and your testing and tracking. So, let's demonstrate this. gitignore file. What we're going to do is we're going to add it in the root of our directory. I'm going to add new file and I'm going to call it. gitignore cuz it ignores. Now, let's say we went through and created a database file that was like super large and we're going to create a DuckDB database. Anyway, I'm going to go ahead and create that file right here and call it large file. duckdb and that's what the database file's named. Anyway, right now, whenever I added it, you can see it went green. It's untracked. Now, I can even scroll over to source control. It's right there. We're calling it large file and it's untracked. Now, I don't want Git to track this. So, what I'm going to do is I'm going to go back to my Git ignore file and inside of here, we're going to use special syntax to be able to track files. In that in our case, I'm going to just for the first portion, I'm just going to write large file. duck. db and I'm just writing explicitly what that file name is. Now, whenever I go to save this, watch what happens to large file. duck. db. It updates and it no longer is tracked in here. Even I can go into source control and it's no longer tracked. Now, you can imagine though, say I have another file or another database file that I create such as even larger file. duck. db. Well, that one become untracked now too as well. And it would be a pain to have to go to this. gitignore and just have to add in every single one. So, we can use the special syntax for this. Specifically, I'm going to remove this large file and I'm going to put this asterisk here. And then I go ahead and save it and now this does some special matching in order to match through and find just any file within anywhere in here to not track duck. db. So, this. duck. db file here, I'll just go ahead and drag it into our data warehouse mart build right here. Even inside of here, it's getting picked up not to track. You can see it's a little lighter gray. And even on source control, you could see the only the. gitignore is there. Now, I'm going to go ahead and drag this out of here and then let's say that we actually we were keeping our stuff maybe in a folder called data. Pretty common for it to be called data. And you know, I put these folders inside of here or these items inside of here. And then for some reason, maybe I add another file. We'll call this super large CSV. And with this, I don't want to charge it. Now, I can go in and you probably should add CSVs to this, but why can't we can Why don't we just go through and actually mark out that we don't want to keep this data folder? Well, for this, I'm going to just go into here and type data and then do a slash command after this to tell this that it's a directory. Now, whenever I save this, this entire data folder goes gray and even that CSV no longer is being tracked. Now, it's important to understand though one little thing about the fact that you need to add this dot get or dot get ignore file early on in your project. Because let's say, oh, now I don't want to include this one {underscore} EDA project folder right here. And so, I could come into here and I could do one {underscore} EDA and then do the slash and then save it by pressing command S. Well, if you noticed, these things are still this lighter color gray and they're not getting removed from get because we already committed them in the past. I can even go to source control and see that, oh, scrap, it's not in there for actually removing it from get. It's still up there. In that case, we'd actually have to go to the terminal and I'm not going to run this command cuz I don't want to remove it, but we'd have to run get RM or get remove and then {dash} [snorts] R to specify we want to do recursively for this whole directory and then from there specify the directory of one {underscore} EDA. Then you could uh, remove it from get and then make sure it's saved in dot get ignore and it won't be there. Not going to do that, but I just want you to know there are options for this. And then after you did this get remove, you'd have to then push these changes or well, commit these changes and then push this up to GitHub to get it into your remote repo as well. So, what files should you add to your Git ignore? Well, conveniently, GitHub, their GitHub account, has this Git ignore repo, which has a bunch of different {dot} Git ignore files depending on what programming language or what you're using. So, if you're using something like Python, you'd go to their python. gitignore, and then you could just copy and paste all the contents from this into your {dot} Git ignore. And this would be my standard approach for any programming language. Unfortunately, there's nothing

### [12:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=45000s) Segment 151 (750:00 - 755:00)

in here for SQL files or SQL projects in general. But, I have a generic one that I've put together based on what I saw in the GitHub one, and you can get to it via our GitHub project for this course. What I'm going to do is now just navigate into the {dot} Git ignore file. And this has everything we want into it. All we're going to do is just copy it all, command C or control C if you're on Windows, and then inside of here, I'm just going to paste it, command V. And then, once it's in here, I'm going to command S to save it. Also, I'm going to do a little bit of cleanup real quick. I'm going to remove this data folder cuz we don't need a need it anymore. Anyway, what's going on here? Well, first of all, this is broken down into sections for different files you may need. This hashtag is similar to like a comment in SQL, but it's a comment in a {dot} Git ignore file. So, anything after a hashtag line is completely ignored. Anyway, I have a section for local database files. This {dot} wall is something that happens with DuckDB whenever it's running and building a database, so I like to make sure that isn't tracked if I accidentally try to commit during that. I also have raw data files in here, any secrets lists that you use. {dot} env files are very popular for this. Also have things like system files, log files, tooling, and then I included also my own files down there. You could probably just go ahead and delete this. One thing that you may encounter while going through this course is this. vscode file. This happens to appear and it has the settings that you may have depending on if where you were did in VS Code and set during the project. Here I am in my other editor for SQL data engineering courses. Anyway, I want to just show this real quick. I have this. vscode file with a settings. json in it. This is specific settings specific to setting up DuckDB for VS Code that we may have implemented. Anyway, I don't need this to go on up to GitHub or other people seeing it, and so that's why. vscode is tracked in there. All right, so that was quite a bit of stuff, but I feel that wraps up what you need to know as a data engineer when it comes to Git and GitHub. We've covered branching now, how to do PR or pull requests, and then also this. gitignore file. We have some practice problems for you to go through, and that's our last set of practice problems before we get into our final project, which we're covering the details of that in the next video. All right, see you there. All right, in this video we're going to go over a brief synopsis of our second and final project in building out a data pipeline. And I'm not going to lie, I'm quite proud of this project. I've spent a lot of time developing it because I wanted to implement everything we've learned up to this point and utilize it in building out a real-world scenario case where we're building out this type of data pipeline. For this, we're not only going to be using those skills of SQL and tools like DuckDB and MotherDuck, but also those other skills we learn with Git, GitHub, and terminal to make this all happen. And this is going to be turning CSV files that are in the cloud into data warehouses and data marts. But real quick, let's have a breakdown of what data warehouse are and data marts are cuz it's really important we understand the differences. So first, data warehouses. Remember, Bill Inmon defined this and it has to have these four things. One, that it's subject oriented. Two, that is non-volatile. Three, that it's integrated. And four, that it's a time-variant collection. And combining all of these four things, we put data in a manner that supports management management's decisions. And this is a data warehouse. Now, for this project, we're not only going to be creating our data warehouse that we've been using from this for scratch, we're also going to be creating three and possibly four different types of data marts. These data marts are going to be extracted from our data warehouse. I like to make them smaller visually cuz imagine it like a smaller subset of the data. But smaller's not necessarily the correct way to describe the difference. Anyway, I put together this table that better describes the differences between a data warehouse and data mart. Data warehouse, the scope of it is more enterprise-wide. All business data would rely in it. Whereas a data mart would have specific departments or even specific use cases. For the schema design, typically you're going to find a data warehouse in what we've seen four of a star schema. Whereas data marts can have those star schemas, but also they could have denormalized or dimensional tables as well. So we're going to be integrating ours flat tables. Now, the

### [12:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=45300s) Segment 152 (755:00 - 760:00)

purpose, based on that scope and schema, is going to be a little bit different. Data warehouses are centralized level levels of knowledge. So you have these reusable data storage areas for this. Whereas data marts are more, well, specific to departments, so they're optimized for specific analytical questions that departments or even individual people need answered. Users for this would be, well, the enterprise-wide for a data warehouse, so anybody in the company would want to access this or multiple teams or even data engineers. Data marts on the other hand are going to be specific teams or specific people that have a use case for it. Data warehouses should have the complete historical data that's necessary for them, whereas data marts are usually aggregated or even filtered subsets of that data. So, moving into the project goals. For this, we're going to be building a production-ready data warehouse using star schema design patterns. For this, we're going to be getting our data from CSV files that have everything that we need for this, and we're going to be building out our job posting warehouse. This will be using the same star schema that we're used to, but we're going to be building it from scratch, so you'll get familiarization with that. And that's going to help us with our second goal of implementing proper data modeling techniques with fact tables, dimension tables, and bridge tables. From there, we're going to also implement this proper data modeling in building out our different data marts. We have three specific ones that we'll go through and break down even further in a bit. And this will satisfy our third goal of creating specialized data marts optimized for different analytical use cases. Now, putting this all together, this allows us to get that data from data storage into data warehouses, and then from there pushed into data marts and in a manner that is now available to be served to others that need this data, whether through Excel, Power BI, Tableau, or Python. And I want to be able to do this controlled in an automated manner. And so our fourth goal is that we're going to develop this using automated build scripts that can rebuild your entire pipeline with only one command. Going to be pretty cool. Just as a sneak peek, this is the file right here, and it goes through and reads these seven different files or six different files that we're going to be putting together for the scripts for building out each one of those pieces. And so by running this file, we can build our data warehouse and data marts from scratch. So let's now get into the project scope breaking into how we're going to build this data pipeline up over the next four lessons. If you want, you can navigate to project two in GitHub, and you can see what we're going to be building out there. In total, we'll be building a total of six, possibly seven files if you decide to. And each one of these control all the different aspects of the warehouse and data mart build. Additionally, we'll be building out with this a read me, which is located below that goes over the different problems we're encountering and what skills we're demonstrating and building out with this. Oh, and also like I showed you earlier, that script that we actually used to right maintain or run this entire pipeline with only one line of code. So has everything we need in it. So in the first lesson, what we're going to be doing is we're going to be taking our CSVs, which are located in Google Cloud, and from there, we're going to be creating our tables needed in our data warehouse, and then importing that data into it. And this data warehouse is going to be Well, we're going to do it in Motherduck or I'm going to actually develop it locally as we go along to simulate more of what we'd do in a real world scenario. So for this first lesson, we're going to be building out these first two strip first two scripts. This first one is where we go through and actually create all the different tables we need. And then from there, the second script is to load our schema into the data warehouse. It has a few different insert into commands in order to do this along with some data checking towards the bottom. In lesson two, we're going to start building out our data mart. The first one we're going to be building out is a flat table mart. This one's pretty simple in that we're going to be transforming our data warehouse into a flat table. And this is not only going to integrate all that company data, but also using what we learned on nested data to also incorporate our skills and type. We'll do this all within a single script in order to create the table and then insert it in from that data warehouse. In the third lesson, we're going to get into building out our skills mart. Now, our skills mart is designed in a star

### [12:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=45600s) Segment 153 (760:00 - 765:00)

schema pattern, but it aggregates our data from our data warehouse to better showcase aggregations of special context we want to get around skills. So, based on a skill, we'll be able to tell how many postings it has and other key statistics that it may have with it. This will also be done in its own single script where we build out those three different tables and then insert into it the required data. The fourth and final lesson for our project build will be around building out our priority mart. And we've done a lot of work with that already in previous lessons. If you recall, we built out that priority jobs snapshot table and that was built using our data warehouse combined with priority roles which could get updated daily. And we built SQL in a manner that supported incremental loading and updating of that priority job snapshot, which is super important as a data engineer to know. And so, in start of our priority mart, we're going to build out our priority roles along with that priority job snapshot. For this, we're going to break this down into two different scripts. The first script being the initial load, which is showcased here where we create and insert into it all the different thing that we need from it. And then in our next script, we're going to demonstrate how we can update our priority roles table and then go through and incrementally update it as well. Specifically, how we use that merge into for the priority jobs snapshot table. Now, you'll also have the option to go through and if you want to build the company mart. And that's just an extra mart that you'll go and go through and build and get practice building out star schema designed patterns. This mart is a lot more complex than the skills mart, so it's really going to test your skills interested in going through this. The SQL code for this company mart is included in GitHub and it has everything you need in order to investigate how to go through this. Now, for those that purchase the course practice problems, in your notes, you have even more details on how to build out this company mart and it breaks it down step-by-step on what you need to do to go through and build this out. So, great option for those that maybe want to beef up this project. One last thing to include for this, we are going to be implementing the use of branching when building this project out. Specifically, we're going to get a develop branch for our entire project two to work along with individual feature branches for each one of the major aspects. So, basically, every lesson's going to have its own feature. I told you, we're going to be implementing everything we learn throughout this entire course and so, all that work is going to pay off. All right, so with that, there's no more practice problems for this lesson or for the remainder of the course. We're now going to be jumping straight into the project starting with the next lesson on building out our data warehouse. All right, see you there. All right, in this video, we're going to be building out our data warehouse. We're going to break it into bite-size steps to do this though. First, I'm going to go over what our Git workflow is, then from there we'll get into implementing it using a script to create our tables, and then another script to insert finally we'll perform some data validation to make sure that it got loaded properly into the data warehouse. As a quick reminder, this is the data warehouse right there we're going to be doing. We have plenty of familiarity with this. But, the main reason why I'm showing this is because it's important to understand we have four different tables that we're going to be loading into it, and we need to make sure that we get all these tables created and inserted in properly. So, for this, I've developed a Git workflow that we're going to use to build out this entire project. For this, it consists of well, it consists of five other branches besides main. We'll have our develop branch, and that will be the main branch for developing all of project two. And then we'll have individual feature branches as we build out each one of these things. Each lesson of these each of these four core lessons will have their own feature branch. And this one will have the feature branch of data warehouse. And in that, we're going to be doing three different commits along the way. That's going to be the commit for our first script for creating our tables, our second one for loading the data, and then our third and final one for building our automated script that's actually going to be controlling both of those steps to run it with one single line of code. So, let's get into creating these different branches and before we do that, we need to make sure that our Git changes are all caught up into main. And in the last lecture, we did Git ignore. So, we need to go ahead

### [12:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=45900s) Segment 154 (765:00 - 770:00)

and get that updated in main before we even jump into this. So, I'm going to do Git add and then period to do everything in the directory. And then from there, we're going to commit it with the message of add Git ignore. Now that everything is committed, we'll go ahead and push those changes up to our remote GitHub main branch. And it looks like everything was updated successfully, no issues. All right, so now let's actually get into creating those branches. We're going to create our develop branch first. For this, we're going to run Git switch with the C flag and we'll create a develop branch/project 2. From there, I can run Git branch and see that we are on the develop project 2 branch. So, now that we have this develop branch, we can now create our feature branch from this. In here, I have an initial connect commit. We don't have necessarily anything to commit right now, so we're not going to do any commits. We're just going to create the feature branch. So, once again, I'm going to run Git switch with that C flag and then we're going to name this feature data warehouse. I'll then run Git branch to make sure that we are on that branch. Yep, we're on the feature branch and our develop branch is there as well. All right, so for this, it's going to be pretty simple in that we're going to be using create table statements. Uh by the way, this is the file that will be green that's on GitHub. Anyway, we're going to be using create table statements to create each one of those four tables in the data warehouse. So, that's what we're doing in this section. Now, we do have to do this in a correct order because if you remember from our create statements that we've used, we go ahead when we define these primaries and specifically these foreign keys, they have to reference for the foreign keys the other table that we're in. So, we have to do it in a certain order. So, for example, the skills job dim has both requires foreign keys for the job ID of the job postings fact and the skills dim table. So, this table we're going to create last. And then the job posting facts table has a foreign key of company ID. So, therefore, we're going to create this one second to last. And then as far as what we create first, could be either company dim or skills dim. And I guess I should be more specific. The company dim table just has to be created before the job postings fact table. And then the skills dim just has to be created for the skills job dim table. All right, so we've already created our script that we're going to be using for this. And that's that 01 create tables data warehouse sequel. For the time being, I'm going to close out this terminal. So, let's start creating tables and we'll start with that company dim table. Inside parentheses, we'll specify the two columns of company ID and also name. Now, with all these, we also need to specify the data type for any ID, it's going to be integer. For the name, it's going to be varchar. In addition to data types, we need to also specify if it is a key, specifically if it's a primary key. So, we'll include that as well. And then between each of the columns, right every line, you need to include a comma to break up all of this along with a semicolon because that's our full command for this. All right, next up, we can do job postings fact, but I'm going to do the skills dim cuz it's easy. In here, we have our skill ID column, which is an integer and it's our primary key. We have the skill itself, which is a varchar. And then the type, which is also a varchar. If you're wondering how I'm getting this all to line up, I'm just using tab to do the spaces. I like this because it keeps everything aligned and keeps it formatted nice and easy to read. Anyway, at the end of this, I'm going to do a semicolon. Next up is our job postings fact table. This thing's a beast. It's got quite a few values in it. The job ID of integer, it's a primary key. The company ID, which is also an integer and it's a foreign key, but we'll make note to that at the bottom. Job title short of varchar, job title of varchar, job location of varchar, job view of varchar, job schedule type of varchar, job work from home of boolean, search location of varchar, job posted date of timestamp, job no degree mention of boolean, job health insurance of boolean. Make sure you spell it right. Job country of varchar, [snorts] salary rate of varchar, salary year average of double, salary hour average of double. So now we need to get into, well we saved the best for last of the company ID, right? The company ID from this company dim table, which is a primary key, we need to be able to reference that. So we'll start by defining the foreign key and then we're going to put it into parentheses of company ID. And the reason is you could do a comma and then add other IDs in here. We'll have that, we'll do that in another, actually in our next table. And then from there we need to go and identify what table this actually references. So we're going to use the references keyword and specify we're talking about the company dim table and specifically the column of company ID. All right, so this is good for this

### [12:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=46200s) Segment 155 (770:00 - 775:00)

create table statement. We only have one more create table statement to do. And this is for skills job dim. Now in this one we have two columns, right? We have our skill ID and integer type and then also our job ID of the integer type. Now both of these are not only the primary keys of this, right? Because you have, they need to be unique for this, but they're also both the foreign keys to reference the other tables. So we're going to define this a little bit differently. Don't forget about your commas at the end of this. We're going to call out the primary key, and then in those parentheses, we'll call out the skill ID, and then also the job ID. And then we'll do the same thing for foreign key, specifying that skill ID and job ID. Sorry, actually I had that wrong. Uh we just specify the skill ID because we need to reference where it goes to into that's to that skills dim table that we created above with the skill ID column. Similarly, we run also foreign key again for job ID, and this references the job postings fact table job ID. Okay? And then with this, also make sure we have a semicolon at the end. So, now after creating these tables, right? We did all four of those create tables, I want to verify that they actually created those tables. In DuckDB and a few other databases, you can just run show tables. However, for a more database-agnostic approach, you do something like this of select table name from information schema, which you learned about before, specifically the tables, and where the table scheme schema is main cuz we're adding these all into the main schema. All right, so, I already did way too much, and we should probably be testing along this way. Let's actually go ahead and test this. So, I'm going to save it by pressing command S, open up terminal. All right, so, I want to run this file from the command line. Honestly, the easiest thing right now we're on the terminal, we're at sequel. engineering. projects. What would be easier just to navigate into our two data warehouse smart build folder. So, we can do that by CD-ing into it or changing directory into it. Okay, so, we're inside of there now. I can do an LS to see what files we have access to. Okay, we have the create tables. All right, now we're going to run a DuckDB command to run this. Now, normally, right, we just run DuckDB and I can uh and connect to MotherDuck and connect to that data jobs uh data warehouse in the cloud. I don't want to do that. We're developing this all locally. I want to test it locally. So, because of that, we're going to create our own data warehouse locally. We're going to call this data warehouse mart. duckdb. This is going to create this duckdb file inside of our data warehouse mart build. All right, but I do want to run this script. Well, if you remember, we can use the C flag in order to do this. And this executes commands for us. Specifically, we have to put those commands in double quotes. And we're going to use a dot command of dot read cuz we're going to read this do dot read. And then 01_ create_tables_data_warehouse. sql. I think I spelled it correctly. All right, let's go ahead and run this. All right, couple few errors in here. First one is this, a syntax error at or near salary_average. I can see that. I forgot a comma right here after this one. And then I also have foreign key skill_id syntax error and that is due to also not having a comma as well. Sorry, not syntax error, parse error. All right. So, I made these changes, going to do command S, and then from there, I'm going to press up and run this again. And well, first of all, this showed us all the different tables that we created, which is great. But, previously when we ran this command, we got this company_dim and skills_dim table in here. Then when we tried to rerun it again, we got this catalog error. Table names with company_dim and skills_dim already exist. Now, we want these scripts to be idempotent, be able to run multiple times again and again. And this is really important as data engineers to be able to do jobs we can do it multiple times while expecting the same results every single time. So in order to fix this, we're going to scroll up to the top up here and we're going to drop each one of these tables if they exist. Starting with that first table of company dim. And skills dim, job postings fact, and skills job dim. All right, I'm going to go ahead and run this. Now whenever I come up here, press up and run this file again. All right, so I got some errors with this, but I don't expect to have these errors. You may not have got this. What I'm going to

### [12:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=46500s) Segment 156 (775:00 - 780:00)

do is I'm going to go ahead and delete both of these duck DB files. This dot wall file should not still be here. This is just a temporary file while you're building out a database. Anyway, I'm going to go ahead and delete this other duck DB file. Let's try running this again. All right, it works fine that time. Now let's run it again to make sure we don't get any errors. And silly me, I probably should actually read this message better. It says could not drop the table because this table is main key table of the table job postings fact. And going to our drop table statement, this is due to having company dim first. Basically, it this company dim table is relying another table. So what we need to do is drop tables that don't rely on another table first. Specifically, the skills job dim. We're going to drop that first. Oh, sorry, I said that wrong. Skills job dim does have references to other tables, but we're going to drop it first so that way other children of it, such as the skills dim, can then be dropped and aren't doing some sort of check to make sure they're not there. Anyway, this will make sense once we run it. So I'm going to take also job postings fact because that's a main one that references foreign key in it. And then underneath this, we're going to list that company dim and also the skills down. Okay, I'm going to go ahead and save this. I'm going to delete both these files again for this DuckDB. So, we're going to run the file first, okay, those tables are created. Now, whenever I run it again, no errors, thank goodness. — All right, so we got this script built. Because of that, we need to go ahead and now commit it. So, we're going to run get add period. This stages all our changes. Also, it's probably good practice you would do run get branch before that to make sure you are on the right branch. I knew I was on the right Anyway, we'll now run get commit with the message of create star schema tables for our data warehouse. All right, sweet. Um we can go ahead if you want to push it, you can. I'm not going to. I'm just going to keep on going through as it is. So, now that we've built those tables in our data warehouse, we need to insert that data in. As I mentioned, we're going to be using CSV files and each of these CSV files correlate to a table. And they are on Google Cloud and I've made these publicly available to you so that way you can go and access them. Now, real quick, this use of Google Cloud Storage is very popular in data engineering. This is using something called like object storage. So, in Amazon, you have what is known as S3. In Azure, you have blob storage. And then in something like Cloudflare, you have R2. Anyway, these have cloud storage so you can keep your files in a cloud location and then as necessary take them and maybe move them into a database. Similar to what we're doing. Anyway, just actually show you, here I am inside of my Google Cloud account. And here are the different CSV files. I actually have an extra one here of just the job postings flat, but we're not going to be using that. But anyway, these are all the different CSV files. And with them, I can do something like for company dim, I can click this and go to copy the public URL. And then in the web browser, put in this URL of storage. googleapis. com/sql_d-data engineering company dev. csv. And this goes through and well, you can't see it right here, but it's actually downloading to my desktop right now. Anyway, the reason why I gave that backstory for this is because now with DuckDB, we can actually not only can we import data in with CSV, we can actually query it directly. And DuckDB supports reading and querying a bunch of different data sources. So as I mentioned before about S3 S3, Cloudflare, Azure blob. And then for us, we're just going to use this direct CSV we're able to read. They also support tons of other different things. Anyway, clicking on CSV. And so with this, we can see that for CSV import, that inside of a from statement, you can write directly not only a table, but you could write a CSV file or the path to a CSV file that you have. So they're all just using select star from flights. csv. Anyway, I took that company dim table that I just downloaded and put it in VS code. You don't have to do this if you don't want. But because this is just demonstration purposes only. But I put this company dim CSV file inside of here. And as you can see, it has the column names of company ID, name, link, link Google and thumbnail. These are the column names. If you're not familiar with CSV, these are the column names and then everything below it is the actual values. And everything it's a CSV or comma separated values and thus it has those comma separated. Anyway, point I'm trying to demonstrate this is I can just query this file directly. So inside of our main project where this is company dev. csv, I'm just going to create a test. sql

### [13:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=46800s) Segment 157 (780:00 - 785:00)

file. And then in here, we're going to run select star from in single quotes company dim. csv. Now, we're going to open up a terminal. I have a terminal open, but it's open to the two data warehouse mart build folder. I want to be in this parent directory, so I'm going to actually just go up one and now I'm in it. Then from there, I'm going to just start up DuckDB locally by running DuckDB. And now with this, I can go ahead and run this. Actually, I don't want to select all Well, we're just going to run all. Bam! And we actually they we queried a CSV with DuckDB and that was really fast. We can see it returns actually in columns the company ID, name, middle columns link Google and thumbnail. Anyway, the point of that is to demonstrate that we can use DuckDB to query a CSV directly and then we can use that to insert into our tables. Now, for this, we're going to be using the read CSV function because it allows us to not only specify the source, but also provide additional parameters. Specifically, scrolling down to the parameters, we're going to use this one of auto detect. We're going to set it to true and this auto detect CSV parameters. Actually clicking on this says that it detects the dialect of the CSV file such as the delimiters, detects the types of the columns and detects whether or not the file has a header row. So, this should handle everything we need for it. So, this test file was just for demonstration purposes only. I'm going to go ahead and close this out. No, I don't want to save it and I'm going to delete this along with this company dim. csv. I don't want to add this to the project. Additionally, we have our create tables data warehouse. I'm going to save this. I want to now create our second file that we're going to be building this out and that's going to be that O2 loads Well, that schema is spelled wrong. Schema data warehouse sequel. So, first let's just query that uh company dim table together using that URL to get to the CSV. So, I'll do select star and then from read CSV. From there, you need to put this in and type it exactly as written. So, this is HTTP https://storage. googleapis. com/sql_ de/companydim. csv. The only thing that's going to change for the rest of these links is this slash portion here referencing what CSV we actually need for this. Now, we did say we're going to specify another parameter, right? And that is of auto detect. And we're going to set that equal to true. Also, I notice I have the spelling mistake up here of from. Also, we're doing a test of this, I'm going to just put in a limit of 10. So, I'm going to open up this terminal. I'm going to actually if you didn't follow along, I'm just going to start over again. In that, you need to run DuckDB. And this is just using a in-memory transient database. And I'm going to go ahead, I'm going to try to run this. Now, I'm sort of surprised by this, but this actually worked for me. Running It may not work for some of you, but it we did query that company dim CSV file. And I was very confused until well, I read the documentation, which I'll recommend you do, too. Anyway, this system to where DuckDB can read these remote files in the cloud, it is possible to do that because it auto loads this extension HTTPFS. Anyway, it says that this extension is an auto loadable extension implementing a file system that allows reading remote files. Anyway, the point here is it auto loads into here. So, we don't need to Previously, I thought we had to write this code, we don't need to do that in order to install and load this extension because it's auto done. Anyway, let's now get into actually inserting into these tables. We're going to first insert into that first table of company dim. Now, in parentheses, we need to specify the columns we want to actually import into. If you recall from our query, there's actually more actual columns in this table. So, if you wanted to, you could, but going back to our create table statement, we've only listed company ID and name. That's all I feel we really need for this. We don't need these other links, so I'm not going to, but you can if you want. So, inside of here, I'm going to list that company ID and name. Also, I'm going to go ahead and just copy this and put that into the select statement as well. Remove this limit 10 from it. And then make sure we have a semicolon on the end. Now, the next table that we're going to insert into, it's really important we get this order correct, is the skills dim table. Now, why is it important that we get this order correct? Well, as you remember from our fiasco with the drop tables that we have to drop tables that

### [13:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=47100s) Segment 158 (785:00 - 790:00)

don't depend on other tables, we need to be able to insert into this that don't depend on other tables. Specifically, something for like the company dim table, okay, it has a primary key in it. It doesn't require other like other tables to check for foreign key constraints. Specifically, right, if I get to job postings fact, what's going to happen when I go to actually upload or insert the data into job posting facts? It's going to do foreign key constraint checks on the company ID column. And what it's going to do is make sure all the values that it has in job postings fact company ID match up to what exist in the company dim table. And if you haven't inserted into the company dim table yet, that's going to be a problem because it's not going to be able to make sure anything's matched up, and it's going to throw an error then. So, that's why we have to do this in order. Anyway, skills dim doesn't require or it doesn't depend on any other tables as well. It doesn't have any foreign keys. So, I'll list the columns of skill ID, skill, and also type. From there, we're going to select into it. We'll select these three columns. And then for the from statement, I'm going to just go ahead and copy this from the company dim table and change company dim to skills dim. All right, so next step, we need to insert into the job postings fact table. We can't do the skills job dim table. Inspecting that skills job dim table right where we created, it does a referential foreign ID check on the job ID column of the job posting fact. So, if the job posting postings fact table isn't populated with all the job IDs, it's going to throw an error whenever we try to do this. So, we have to do the skills job dim table last. Inside parentheses, I'm going to list all the columns. And then from there, I'm going to get into writing the select statement for this. I'm going to once again copy all these columns and paste them into here. Once again, copy that from statement from up here, paste it in here, and then change this name right here to job postings fact. csv. All right, the last table we need to insert into is the skills job dim table. This has that skill ID and job ID. For this, we're going to select the skill ID and the job ID. And then I'm going to copy this from statement, paste it in here, and change the table itself to skills job dim. All right, so we have all of our insert into statements. I'm going to go ahead and just put a comment up at the top that this is uh step two. And this is the load data from CSV files into our tables. All right, I'm going to go ahead and press command S. Now, let's go ahead and run this file in order to make sure that it's going to work properly. I think mine's going to work properly, but the thing is this, if you get an error during this, we're going to need to do some troubleshooting and also fix it. So, we need to develop a path forward to be able to One, you can't run the script over again cuz once you insert into a table, you try to insert again, it's going to be a mess. It's going to just keep doubling the data, not good, but not what we want. Instead, what we want to do is every time we run this, we want to create our automated script to actually run step one where we drop all these tables, then create the tables, and then from there insert into it. So, that way if we have errors in this as we're building out, we can easily fix it and rerun it to where we start from scratch and run it again. So, I'm going to go ahead inside of this data warehouse spark mart build, I'm going to create another folder, and I'm going to call this build dw marts. sql. It's a SQL file, and for some reason I made this into a folder. So, I'm going to delete this and try again. All right, so that worked this time. I'm going to move it all the way over to the left-hand side. Now, if you remember previously in SQL, I'm going to go ahead and exit out of this duckdb right now. I'm going to scroll up here and go to the command. We would run this command duckdb data warehouse marts duckdb-c read 01_create_tables_data_warehouse. sql. Well, what we're going to do is we're going actually going to take this command right here, this dot command that can be run inside of SQL files, and I'm going to just paste it right here. And then from there, put a comment above it that this is step one. Additionally, we need to fill in step two. So, I added in a comment and then that dot read command of the SQL file we will run. Now, this is pretty cool, right? This is a SQL script. We can then call this SQL script within which then calls our first SQL script and then calls our second SQL script, making this an automatable process. So, I'm going to go ahead and save this, and now we're going to be basically testing this second file. So, first thing though, we need to make sure our terminal is pointed to this two data warehouse marts build folder. So, I'm going to CD into it by selecting it. All right, now we're in it. Then from there, I'm going to run duck DB, then specify that duck DB file of data warehouse marts, then the C

### [13:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=47400s) Segment 159 (790:00 - 795:00)

flag, and we're going to run a dot read command of our build marts script. Actually, meant this to be build data warehouse marts. sql. So, we'll update this then. Okay, we got our command. Let's test it out. This is going to be testing. We already know the first script works, but does the second script work? All right, and we got an error. So, this is actually good for demonstration purposes. In it, we say binder error reference column skill not found in from clause candidate bindings skills ID. So, this is actually a typo on my part. In here, I said when we create our skills dim table, I gave it the name skill, and then also in the load schema, I also gave it the name skill. But, if I actually take this URL, and then go to navigate it to it in a browser, you can do this as well. Previously, I was in another browser. I'm now in Safari. You can it won't necessarily doesn't have to download it. It can potentially just show it to you. So, I just found a workaround. Anyway, the point of me doing that is because I can see and I can access those column values. And I can confirm that column name is incorrect. So, inside of here, we're going to change not only our create table statement where we reference skills right here. I'm going to command S that one. But then, also, we're going to change our skills dim table where we reference skills as well. And command S it. We also have one other binder error down here. And it says table job posting fact does not have a column with name job health insurance. Anyway, I went to that first table of create tables, scroll down job posting fact, I misspelled insurance. Anyway, retyped it in correctly, command S it. And then this last error, this constraint error was due to these other columns not loading properly, so we're just going to ignore that for now. All right, so [snorts] let's run this build warehouse mart sequel script again. I'm going to press up and run. All right, not bad. Looks like it loaded all of those different files. However, that did take me that was quite a few minutes that it took to load. So, when I'm working with about 16 gigabytes of RAM. Now, I was curious during the process like, "Hey, what table's running now? " I would like some updates as it's doing this. So, what I can do is in our second file right before we insert into like the company dim table, which is our first table, I can run a select statement and then in single quotes put loading company dim table and we'll do this as info. Basically, she's going to perform this query, let us know that it's loading it and then it's going to perform it. I'll also do this for the skills dim table as well, job postings fact table, and then skills job dim. All right, I'll go ahead and save this. I'm going to just run this just to show what it looks like. And it starts out by telling me, "Hey, first it's loading that company dim table. " So, it goes through and it was getting the company dim table, so that took 8 seconds, took a little bit longer than that. Skills dim, got the job postings fact table, and then unfortunately, I got an error because I had an F right here after my semicolon. Anyway, you get the point. There's now markers along the way that tell us what table it's downloading. Note the skills dim table, it loads because it loads so fast, it's not showing a progress bar, but we're going to be validating the data next. Make sure we actually got it loaded. All right, so we're going to get into data validation. I do want to Well, I re-ran that script also and just showed that everything loaded correctly and it has those tables for it. Anyway, we've made a lot of progress with all of these different sequel code that we've generated. I want to and now at this point, I want to commit these changes. So, I'll do a get add period to add all the files. So, I'm going to run get commit with the message of load schemas and create master build script. All right, so those changes committed, let's move into the data validation. Now, for this, this is nice that we have this master build script, but I don't want to I mean that took a few minutes every single time I'm running this data warehouse. And this data validation script, we're going to just be putting it in at the bottom. So, anyway, I'm not going to run this uh data validation or run this master build script until the very end. Instead, what we're going to be doing is I'm going to enter in the data validation queries in here, but what I'm going to do is I'm going to run it directly on our database. So, as you know, we can run duck DB and then specify the database of data warehouse marts of the dot duck DB file. So, I can do things like select count from company dim. All right, let's see how many files in there. 215,000. Looking good. Next, I'm going to select the count from the skills dim. Running this, we have 262 skills. Okay, I actually want to put these We have to do two more tables, but I want to put this actually in just all one table in where I have a row for what table it is and then a row with the count of the rows for that table. What

### [13:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=47700s) Segment 160 (795:00 - 800:00)

do I mean by that? Let's Let me just look at this query that we're going to build out. So, I'm modify this top query right here and say company dim as table name. And that will be the first column. Next will be count, and that will be as record count from our company dim table. Then, I'm going to remove this semicolon, and I'm going to do a union all to our skills dim table right here. But in this case, I'm going to add those two columns. We'll do skills dim and then add a comma and then we want to count. I don't need to again give it aliases because during the union all it will automatically inherit the alias of your topmost table. Anyway, let me just show you what this looks like whenever we run it. We get this table where it says, "Hey, the table name and record count has company dim, skills dim. " And it has the information we want in it in a single table. Union all's pretty sweet. Next we'll get the job postings fact table, its count from table. I'm going to remove the semicolon up here so it doesn't throw me off. And then finally throwing the last union all, looking at that skills job dim table and getting the count for that as well. All right, I'm going to go ahead and then run this all. Have a typo, should be skills. All right, we'll go ahead and run it again. And I can tell [clears throat] from the results of this, what I know about the number of rows in this, all of these have the appropriate number of values in it. Specifically, look, we're going to look at job postings facts and remember we saw before it had 1. 6 million rows and that's what it is here. Now also, I do like to look at the first few rows of the data whenever I'm importing it in to make sure that it did import in correctly. So normally I would do something like a select star and then from company dim and then limit this to just five results. Also right above this, I'll actually just specify what this table is of the com- company dimension table sample. All right, I'm going to go ahead and run this. All right, and so then we get this returned whenever we run this, it will say the company dimension sample and then it will give us just those first five rows from it. I went and also added in sampling the first five rows of skills dim, job postings fact and skills job dim. Going to run this to make sure this runs just fine. And scrolling through this, we have everything from our company dim table looks good, our skills dim table looks good also, job postings fact is looking good, and our skills job uh bridge or skills job dim table. All right, so that is everything for the data validation that I want to do. What I'm going to do now is go ahead and save this file, and then just for good measure, I'm going to quit out of here. And then I want to run this now script from start to finish to make sure that I import it correctly, and I do all that data validation correctly before I then go ahead and submit it. Now, I pressed up and got this command right here. I'm going to go ahead inside of my build data warehouse mart script. Don't insert it into here, but well, you can it has to be behind a comment, otherwise it's just going to loop itself and your computer will probably blow up. Anyway, I like to have a reference in here, so if I need it, I can just copy it at a future time. Anyway, we have it in CLI, let's go ahead and run it. All right, so it's complete. We loaded all the table names in, loaded all the different tables, and then did our basically those five query checks with the limit five checks on all the different tables. Everything's checking out good. So, our build scripts are looking good. All right, so let's get into committing all these changes. I'm going to go through and make sure all of our files are saved, open up the source control pane, click get add or type in get add period, and then we stage all our changes. From there, we're going to get commit the M flag of data validation check set. So now, we've just gone through and did those three commits. We did that for our tables, creating the tables, loading the tables, and then previously I was said we're going to do it for a master build script. We actually did it for the data check or the data validation. Now, what we need to do is actually merge our feature branch for the data warehouse into our develop project two branch. So, inside of terminal, I'm going to run get branch just to check where we're at. Right, we're on the feature branch. I want to switch to our develop branch because we're about to merge to it. So, I'll use get switch and then call out the develop project two branch. Run get branch for good measure. All right, it's there. Now, we need to do the get merge where we're merging that feature branch for our data warehouse. All right, from this we can see that it is a fast-forward merge just like we want and then it tells me that these three file additions have been added to it. We know those three files, we can see them up above. All right, so not bad and then we can see over in the graphical user interface for the graph that we were on that develop project branch. Our commits are up to date with this where we're we are

### [13:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=48000s) Segment 161 (800:00 - 805:00)

ahead of our main and also origin main which is on GitHub. So, now we no longer need that feature branch anymore. So, I'm going to go ahead and it was only locally so only needed to delete it here. So, I use get branch with the D flag and specify feature, press tab, data warehouse. We don't need it anymore. Running get branch, just check what we have. Everything's looking good. All cleaned up with this portion. All right, so you should be proud of yourself because we just built a simple data pipeline that implemented a schema design, loaded our data in from a cloud storage device and then once we got it in performed data validation to ensure we loaded it properly. We did a lot of stuff and that's a lot of things we've implemented in this course so far. So, you should be super proud of it. All right, in the next lesson we're going to be getting into building out our first data mart. With that, I'll see you in the next one. All right, in this lesson we're going to be building our flat mart or our flat table mart. This is going to be denormalizing our star schema of a data warehouse into a single table. So, for this video we're going to go over why we're even building this flat table mart, and then we'll actually get into building it along with what we need to do validate it, and then from that we're getting it updated into our master build script and implementing it into the appropriate branches. All right, with that let's jump into it. All right, so this is the first lesson that we're actually going in and building our data mart. And this data mart is going to be called a flat mart. And in this is going to include all the relevant columns from our web data warehouse. So, everything for we see from here is going to be integrated into it including our skills. Specifically for this one right because most job posting well, all job postings only have one company per job posting. But job posting is going to have multiple different skills. So, we're going to be doing or using that array of structs in order to put all of the skills into our flat mart. Anyway, just as a refresher right that data warehouse is made for like enterprise-wide where all business data lies in it. Data mart is used for a specific department or use case. In our case we have a specific UK use case. We want this a flat denormalized table in order to be able to query quickly without having to join other tables. So, why even build this flat table mart? Well, in our case query speed's going to be increased because there's no joints required. It's also a lot simpler especially for maybe something like a business analyst that isn't as familiar with performing joins and aggregations. Additionally, we're not going to do it in this portion, but you could optimize this and maybe tailor it to a specific need such as you could tailor down to a business analyst or maybe senior data analyst or you can even roll it up to where you perform certain aggregations within it all within a flat table. Now with this flat table, there are some trade-offs with it. You could see some data duplication depending how you do it. In our case, we're not going to have well actually we will have data duplication especially when it comes to the companies and then something like the skills and specifically the skill types. And because of this data duplication, this is also going to cause storage to go up slightly. Overall for our table though with the size of it of 1 and 1/2 million rows, I'm not too worried about it. We also have issues around updates and flexibility, right? So now we're having an additional mart on top of our warehouse. So that's going to add extra time anytime we need to rebuild this or maybe investigate it due to issues in it. All right, so let's get into building this bad boy. And for this right, we're going to be extracting this data out of the job postings warehouse and putting it into its own data mart of a flat table mart. Technically, we're going to just use a schema for this but we'll explain when we get there. Now as far as our branch workflow is going to go for this, we're going to create a new feature branch for our flat mart and we'll perform a couple commits during this process. Once we have the full flat table built, we'll then merge it into our develop branch for project two. All right, so let's jump in. We'll start by running get branch to see what branches are available. We have develop and main. So I want to run get switch with the C flag to create a new feature branch for the flat mart. Running get branch just check I switched. Yep, on it. Inside of our data warehouse mart build folder, we're going to add a new file for that third script called create flat mart. Now for this right, we're not going to create a separate database for this although you can if you want. Instead

### [13:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=48300s) Segment 162 (805:00 - 810:00)

what I prefer to do is we're going to keep the data warehouse on the main schema and then all the marts in their own separate schemas. So, because of that, I'm going to use create schema and then call this flat mart and that's where we're going to be putting all the or well, the only table for this inside of here. Oh, and before I get forget, we're going to put a comment up here on what this script is actually doing. Now, I do want to clarify something real quick. Say you didn't want to create a schema, but you did want the flat mart to be in its own separate database. Well, in that case, we wouldn't create our own schema, we would just use the main schema of a database and whenever we run this file, we specifically call out to use DuckDB and then maybe we'd name it something like flat mart. duckdb for the database we want to put into here. And then similar to before, we use that C flag and then we use dot read specifying this file. Now, that is a completely valid option to go forward with this. I'm just trying to keep it simple by putting the different marts into different schemas and also keep it to where we don't have a bunch of DuckDB database files at the end of this. Anyway, uncomment that, let's continue on. Let's start by querying and actually well, performing a query to generate this flat table. If you're remembering from our data modeling, so the goal from this is just combine all of our dimension tables into our fact table to make this flat table. So, we need to select all the different columns. I'm a little lazy, so I'm actually going to come into the load schema data warehouse and we queried the job postings fact table in that. So, I'm going to go ahead and select that and then paste that into here. Now, we're going to have multiple tables in here, so I'm going to go ahead and start putting in the tables so we can start giving them aliases for what we need to know what the aliases for these columns. So, first we'll be coming from job postings fact as JPF. I need to add JPF to the front of all this. Next up, we need the company dimension fields. So, for this I'm going to join the table first. We're going to do a left join with company dim as CD. And we're going to join it on the company ID of each of those. Now, one thing I didn't include that company ID from the job postings fact table. So, we are going to include it from the company dim table. Specifically, I added in well, company ID and then we took name and I gave it the alias of company name. All right, so the last column we need I have in here as is as skills and type. If you remember from earlier, I said we're going to get this and collect this as an array of structs. And so, in order to get this in, we need to do left joins to get not only the skill job dim table, but also that skills dim table. So, let's start by doing the each of those left joins. First, let's start with skills job dim and we do this on the job ID from both of those tables. Next up, we do the skills dim table and we do it on the skill ID from each of those tables. So, now we need to bring in from that skills dim table the skills and the type. We're going to do this in an array of structs. So, we're going to run the array agg function and then inside of that we're going to need to run the struct packed function to make this into structs. And then we'll have two things in here. We're going to have type and this is from the skills dim type column along with name. And this is from the SD skills column. I'm only just naming this name because technically this entire portion well, not that technically I'm going to show you. It the alias as skills and types. So, it's intuitive that the name is referring to the skill. All right. Now, remember right, this is an aggregation function. So, because of that we need to do a group by underneath this. Now, you need to list all of these columns in here depending on what database you're using. Except if you're using DuckDB, conveniently, we can write group by all. We're going to do that cuz we're using DuckDB. So now, let's go ahead and we're going to save this first. Let's go ahead and actually test this bad boy. With the terminal, make sure that you've navigated to the data warehouse smart build. You would do that by doing CD into that appropriate directory. We're already there, so we're good to go. And then from there, we're going to run DuckDB and then list our data warehouse smarts. db file that we want to connect to perform this query on. All right, so I'm going to go ahead and copy this all. And then I know this is going to query all the rows, but I want to make sure that it works. So, that's what we're going to just do it all and I'll press shift enter. And it comes back with 1. 62 million rows, which is is expected. And looking at the query, there were no errors and it looked like it ran just fine and everything's in there. So, this is really good. So now, this brings us to the question of what

### [13:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=48600s) Segment 163 (810:00 - 815:00)

are we going to put this query into? And what do I mean by that? Well, previously, right? We used when we were creating our data warehouse, we used these create table statements to create the tables, list the columns, define the different data types along with their primary key and foreign key. And then we had to go after that and perform insert into's with that appropriate query that we generated to query our CSV. So, do we want to do that? Or, if you remember from our DDL and DML lessons, we had three different types of tables, if you will, of CTAs, views, and temp tables. Well, what we could do instead is use the CTAs or create table as in order to build this table and it takes a lot less code and it's a lot easier. Well, I would argue that this case of building a flat mart is a perfect use case of a CTAs. It's a very clean implementation as we can just list the query and then use the create tables as super simple. It's also a lot faster than creating this whole table and then inserting into. And for this use case of the mart, it's fits its use case specifically. Additionally, just drilling down slightly further with this, this flat mart doesn't have, if you will, doesn't need to assign any primary keys or foreign keys or specify data types cuz it's going to be inheriting it from the job postings fact dims and bridge tables. So, because of that, CTAs is perfect use case for this. Now, that doesn't mean we're going to be using CTAs for all of our different data marts. There are use cases for using create and insert when you want production level quality for your table behavior. Specifically, as I mentioned before, if you need to enforce constraints or schema control, or like we'll be doing in our third one, performing incremental loads where we need to have and ensure that our data quality is on par. This is important that we actually go through the extra effort of creating the create and insert. So, for all the marts after this, we're going to be using this create table and insert method. All right, so let's go ahead and do this, right? We create our schema of flat mart. From there, we want to create table. We'll specify the schema of flat mart and then we'll give it the name job postings. Then we need to give that [snorts] as cuz it's a CTAs. All right, so now I'm going to save the file. Let's go ahead and actually run this time. And this is actually going to be building this table in our DuckDB database. And so to do this, we're already in here, so I'm just going to stay within this command line right here inside the DuckDB database, and I'm going to just run the read command specifying we're going to be doing this from the 03 create flat mart SQL file. It's got to be saved. Right, let's go ahead and run it. And looks like it was run with no issues. I'm going to just check it real quick by running a select count star from flat mart. job_postings. And we see that we have 1. 62 million rows. All right, remember we want this script to be idempotent to be able to run again whenever we rebuild it. If I were to try to go ahead and I'm going to click up and navigate to this and try to run this file again, I'm going to get two different errors. catalog error where the schema with name flat mart already exist, and then also the table with name job postings already exist. So in order to fix this, I'll add above the create schema drop schema if exists and specify that flat mart. And then for the create table, we'll do create or replace table. All right, going to go ahead and save this. Let's try to run this again. I'll press up. Now, we got a few different errors with this and somehow the table also loaded with this. Anyway, it says, "Hey, cannot drop entry flat mart, the schema, because there are entries that depend on the table job postings depend on schema flat mart. " Basically, what it's saying is, "Hey, there's a table inside of flat mart. I can't drop it because there it's not empty. " So what we're going to do is we can use drop and then dot dot cascade to drop all dependents. So we'll put cascade on here, and then I'm going to go ahead and save this. Now, let's try to rerun this. And when we do that, we get no issues. Also, I'm going to come up here and where we have the select count from this table, I'm going to go ahead and paste this now at the bottom cuz that's a good data validation script. All right. And that actually brings us into our next portion of performing some

### [13:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=48900s) Segment 164 (815:00 - 820:00)

data validation to make sure that we have everything in this table that we want. We already did this the count, right? We want this automated process whenever we load this flat table to happen automatically. And because this I like to have right now you can see this table doesn't really tell us like, "Hey, what is going on here? What table is it? " Cuz you can imagine we're going to be populating a lot of tables. We want to know what it is. So, I'm going to modify this script up here. I'm going to put a string of flat mart job postings and we'll give it the alias of that's the table name. And then we'll do count and we'll do this as the record count. Now, running this, we can see that we have the flat mart job postings and it has 1. 61 million rows in it. Additionally with this, I want to do a sample of maybe the first five rows of the table. So, I'll first start with this string of hey, this is a flat mart sample as info and then get into the query itself where we're going to just select all the different columns from flat mart. job postings and then just limit this to 10 values. All right, let's go ahead and just run all this and see how this looks together. So, with this we get the total job postings 1. 6 million. Then from there it says, "Hey, we're getting into the flat mart sample. " And it shows us our 10 results for this. Now, I do want to add one other thing and that's up at the top to tell us that we're loading this flat mart cuz it did take a few seconds to load. And I'm going to put that right above where we're creating or replacing this flat mart table. And I'll say, "Hey, we're loading the flat mart. " And then we'll put this as info. All right, so let's test this whole script out by pressing up. And we're going to run this dot read on this file. Say I'm loading my flat mart. It then displayed the progress bar. It got into the different postings. How many is 1. 6 million along with that 10 row count for this. Now there's only one other thing left to do, right? That's update our master build script. So I'm going to go ahead and put a third entry in here. And that's for the step three of creating a flat mart where we do use the dot read command on that number three file. Okay, go ahead and save this. Now in terminal, I'm going to go ahead and get out of here. And although I've tested this three or this third script, it should work just fine, but I always like to just go through and test all of these again using this master build script to make sure everything gets built perfectly. Now if you it took like, you know, 10 20 minutes to load this, you feel free to skip this. It should continue to work. But as best practice, you should test as much as possible. So I'm going to come up here and copy this command that we put in here earlier for reading this script. And then let's get to testing. All right, so I just got done. Today for some reason, I did restart my computer. It ran a lot quicker. It only ran in like a couple minutes. But I loaded all the tables, did the initial data analysis for the quality checks of the data warehouse, and then it got into that of our flat table down below, and everything checked out. So everything's working great. All right, so you know what we need to do now? We are Well, let's check our branch. We need to update and commit, and then also merge. So right now we're on that feature branch as we wanted. We need to commit to that feature branch. So, I'm going to run a git add dot to get all the files in a directory. And then from there I'll run git commit with the M flag saying we're going to we built and validated that flat mart. All right, sweet. Everything's looking like good with this. Now, let's go ahead and merge this. Specifically, we're going to be merging into our develop branch. So, we need to switch to that. So, we'll run git switch and then that's to our develop branch. And then I'll just double-check that I am on the right branch. We are on the develop. So, we'll then run git merge from that feature branch pressing tab to add that flat mart. Go ahead and do that. And so, we can see with this we added that uh three crate flat mart file along with adding additions to our build data warehouse mart uh master build script. All right. And we do still have running git branch properly this time. We still have that feature branch. We don't need it anymore. So, we need to clean it up. So, we're going to run grit git branch with a D flag, specify that feature branch, and delete it. And then running git branch just to check. Yep, all cleaned up. Just got main and develop. All right. So, great work on this. You did an excellent job of denormalizing our star schema table into a single flat table and now having it available in this new data mart. In the next lesson, we're going to be kicking things up a little bit with the data mart that we're going to be building and specifically more of a star schema framed data mart. And I'm excited about that one. All right. With that, I'll see you in there.

### [13:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=49200s) Segment 165 (820:00 - 825:00)

— All right. In this video, we're going to be building further on our data mart. Specifically, we're going to be building one around skills demand. And this will be Well, last time was where I denormalized as a flat table. This is going to be a dimensional mart. So, it's going to have a fact table and two dimension tables. So, for this we're going over not only why we're building this skills mart, but then we'll also get into how to actually build it along with making sure that we're updating that master build script and sticking with our plan for maintaining branches. With that, let's jump into it. So, first off before we dive into the why, let's look at what we're actually building. So, we have a singular fact table and two dimensional tables. The fact table is centered around understanding analytics for demand of the skill. Now, this fact table is an aggregation of our bigger data warehouse in that we're aggregating it not only on a monthly basis by skill, but also by that job title short column. And then from there we get some different counts based on those aggregation we want to do it by. So, we'll get things like the postings count, remote postings count, the health insurance posting count, and the no degree posting count. Now, because we have this these values we need to have dimensional tables for it. For the skills, we're going to have a very similar table to our skills dim table in our data warehouse where we have skills and type. And then for the monthly start date, we're going to have a dim date table which is actually pretty common in SQL. And we're going to be able to look at based on a month start date things like the year, month, quarter, quarter name, and year quarter. So, why the heck are we building this skills mart? I mean, we already have this data warehouse already which has all this information in it. Well, a few different purposes. As you can imagine, our data warehouse is a little smaller I would consider it small with 1. 6 million rows. You may have data warehouses that are extreme like millions or billions of rows. In our case, we're aggregating skills. So, this is going to be a lot faster be able to perform time series analysis and really any analysis on it whenever we're trying to dive down to the skill level that we've done. And because we've aggregated it down at the skill level, this makes our data mart in our case much smaller than that row level data in our data warehouse. And then the way we're going to build this is done in a way that we're doing additive measure so we can they're simple they're safe to actually re-aggregate cuz we're just going to be adding to each other. Specifically, we're going to be doing counts. If we started adding averages or medians to our dimensional mart, it's going to cause all sorts of problems. We're not getting into that. All right. Anyway, I just want to demonstrate real quick the power of this for specifically for the end user. So, I went through and I've already built this skills mart because well, I'm an instructor and have all the notes and so I can do that. Anyway, let's say we have this query. We're not going to break it down completely. I'm just going to show you the break down the gist of it. But let's say we have a query where we want to find out say for a senior data engineer, what are the top what is the top skill per quarter? Well, we can do this pretty simply with this data mart. Anyway, let's go ahead and run this on this skills data mart. And so, based on the aggregation we performed, we can see on a quarterly basis what is the different or what are the what is the top skill for each of those quarters along with whatever associated count we want to go with this. Anyway, a mart like this would be really powerful to somebody that maybe would have the job of ensuring they stay on top of skills and are analyzing it on a quarterly basis. All right. So, let's walk through the Git workflow for this. We're going to create a new feature branch for the skills mart. Then from there we'll perform either one or two commits for the different feature. After we've tested it and updated that master build script, we're then going to merge it into our develop branch. So let's get into VS Code and start doing this. Inside of here, I'm going to run Git branch first and see that okay, we only have the develop branch that we're on and then main. This is good cuz where we want to be. We're going to now run Git switch with a C flag. So then we run and create this feature one with skills mart. We'll run Git branch for good measure. We're on the feature branch. All right, good. Let's now create our SQL file inside of our data warehouse part mart build project two folder. Well, this will be the fourth file and we'll name it create skills mart SQL.

### [13:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=49500s) Segment 166 (825:00 - 830:00)

And I like doing this number convention because well, it keeps it in order. And I put that zero in the front because it could build up higher than 10. In that case, I still want it to be sorted. Similar to last time, we're going to create a schema for this data mart. And this one will just be called skills mart. Also, if you remember go if we go back to that last remember, we also did drop schema if exists along with that cascade. So we're going to go ahead and just add that in. So I put that at the top and then just change this to skills mart. Also, put a comment at the top for step four, mart create skills demand. Mart, not mart. All right, so what's the plan for building this mart? Well, remember when we were building our data warehouse, you have to build it in a certain order. Specifically, that fact table has those foreign keys that depend on those dimensional tables checking them in order to then build out the fact table. So because of that, we have to build our dimensional tables first. Now, also we are going to be enforcing primary keys and foreign keys. And so, we're going to be doing this referential integrity check with this as we go through. Because that, unlike last time, where remember we could use something like a CTAs in order to build our table, we're not going to be able to do that in this case. We're going to actually need to use create table and insert in order to build these. So, for all three of these tables, we're going to like I said, we're going to do the create table and insert. But, I find it easiest if we just build the query first, like for dim skills, build the key query that collects the skill ID, skills, and type, and then from there we'll do the create table and insert into. All right, so starting with that query, we want to get uh three different columns, specifically our skill ID, skills, and type. And that's all from the skills dim table. Now, let's go ahead and test this query out. So, to do that, first thing right where right now we're in the SQL data engineering projects folder, I need to navigate to into our second project folder, which is the data warehouse mart build folder. So, now I'm inside of there. Now, I can run DuckDB, specifically running it on our data warehouse mart for DuckDB database. So, now let's query this, and the query runs successfully. We have all the different skills that are appearing, and it's 262, as expected. So, now let's create that create table statement, and that's for our skills mart schema, and we're naming this You can name it either skills dim, or it's also common to convention to name something with dim first, and so dim skills. We're going to switch it up and do that for this one. For the values here, I'm going to take these three right here and paste it in. And then we're going to go through and just specify the data type, along if it's a primary key. Skills is a varchar, type is also a varchar. I'll put a semicolon at the end and now we'll get into the insert into. This is going to be for the skills mart schema in that dim skills table. This is going to be using these three columns right here. So what I'll do I'll just enter down and paste them right here. Make sure they're formatted. And then it's using this select statement. All right, I'm going to go ahead and try and run this and so I may I got this schema with name skills mart does not exist. I never did that first portion right here. Anyway, what I think is I think the safest option is just will you do a dot read on this right here. So I'm going to save this and we'll run dot read and then use that create skills mart SQL file that we're trying to reference. And it looks like I have a typo table with the name dim the kills and I can see I do have an error right here. I have the kills and it wants to be actually skills. I'll go ahead and save that. Let's try this again. And ramming the wishes. Now that we have our skills dim table, we're now going to get our date month dim table. For this we're going to be basing it off of our job posted date column in our job postings fact table. And then we'll get these well if you will aggregations from it things like year, month, quarter based on the job posted date. So for this we're going to start this query with select. We want to get the month start date. So we're going to use the function that we learned about last of date trunk and in it we're going to specify that month along with the column we want of job posted date. We're going to do this as month start date. Next we want the year, month, and quarter so we can use extract for that. And if we specify extract, we specify year from job posted date and we got to make sure we spell that correctly. And we're doing this as year. Then I added in month for job posted date and finally quarter. When I say finally quarter, I want to test it now. We do have a few others

### [13:50:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=49800s) Segment 167 (830:00 - 835:00)

that we want to run still. What I'm going to do is from job postings fact. All right, so let's go ahead and run this and we're going to shift enter. All right, so not bad. It looks like we're getting back what we want from this with the month start date, year, month, and quarter. If you do note the month start date, it's repeating, right? Because you have all those different rows. So, this is like a job posting somewhere in January, another January. We want distinct values to just limit it down. So, I'm going to change that to distinct and now we're going to run that again. And now we're coming back with only 30 rows. And scrolling up, okay, this is looking a lot better. I do want to order by though that month start date. So, I'll throw an order by in here. And then put month start date. Running this. Boom. We can see that we have it starts in 2023 and then goes up from here to 2025. All right, there's a few more columns I want to add to this. Specifically, I want to get the quarter name. So, right now our quarter table just has the one. I want it to where it says Q-1. And so, I'm put single quotes Q- then we're going to do the concatenate operator. And then from there we want to put in this quarter up here. By the way, I forgot a comma up here. We want to put this quarter here, but right now if I look at this type, this is an integer. So, I'll need to put this in here and then we'll need to cast this as a varchar. And then from there we'll give this the name or alias quarter name. All right, running this to test it. It's looking good. That has the quarter now name now of Q1, Q2, Q3. All right, last one we're going to do is a year and quarter combination. So, the first thing we're going to list is the year. So, I'm going to go ahead and copy this right here. We're going to have to concatenate this all, so I'm going to automatically cast it as a varchar and then add in that concatenate operator. We'll then do a dash cuz we're going to do the quarter. We'll do another concatenate operator, and then we need the quarter, so I'll actually just copy it from up here in the quarter name, and this one cast to varchar. So, we'll give this all the alias year quarter. All right, make sure we have all of our commas as appropriate. Let's go ahead and run this bad boy, and this comes back with what I want. It has the year at the front and then quarter next after that. Now, I wanted that year first because I want to make sure that whenever I go to if I were to sort it by this quarter uh year thing, that it would sort properly, and so that's why I put year first. All right, so similar to above, we need to do the create table and also the insert into for this. So, I'll start with creating the table with our skillsmart schema for dim date month. I'm going to name again, once again, dim at the front of this. Then we're going to go in and start filling the columns. First is that month start date, and that is of the date type, and it's also our primary key. Next is year, month, quarter, and all those are integer. Quarter name of varchar, year quarter of varchar. Now getting into the insert into. This is going to be for the skillsmart dim date month table, and then I'm going to go in and list all the different columns for this to make sure that we put it into the correct order when we insert into it. I'm going to go ahead and save this file. Let's now test this. We're going to run that read create skillsmart SQL script, and it looks like I forgot a semicolon after this create table. We'll go ahead and add it on, save it, try it one more time, and it ran just fine, no errors. All right, last up is our fact table, and this is going to be the most complicated of all the queries we have to build in order to get this. This is also going to take a CTE because we're going to have to get our counts for whether they're whether or remote job or they have a health insurance or no degree mentioned cuz right now they're boolean values. We need to get the count for that. And so because of that we have to convert it to either a one or zero and then we'll do that in the first CT and then in the outer one we can then add those ones if it's true and zero falses in order to get those final counts for those. All right, so let's build that query. We'll start with our select statement. For this we're not going to be using the skill name, right? We're that skill ID. And now I could do this from the job postings fact table. I don't want to have the duplicates and have to narrow it down. So I think what's going to be better instead is first I'll add in that job postings fact table, but then from there I'll do that left join to make sure we can serve all those job postings, right? And we'll join in the skills job dim with the alias of SJD. Okay, so we're going to take it from there. We'll say SJD. skillid. Next we need that month start date. So we're going to use that date trunk function again. Actually, I'm just going to come up here and copy this and then paste it down here. The only thing we

### [13:55:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=50100s) Segment 168 (835:00 - 840:00)

need to pay attention to is job posted date is from JP F alias. Next we need the job title short. We add that in and now we need to convert all of our boolean flags from true or false to one or zero. So for this we're going to use a case when statement. So when job work from home is something like true, then we want it to be one. Else if it's false, zero. And then we'll add this as is remote as for the alias. Also did this case when for job health insurance if true and we mark this or give this alias of has health insurance. And then finally for the job no degree mention of true, the they said that to one and I said it with the alias no degree required. All right, so this is looking pretty good. Let's go ahead and run this to test it. And then I need to add a semicolon. I forgot to add a comma after skill ID. Whoopsies. So, running it again, I got a parser error after the as. I realized, remember when you do a case when you need to have an end at the end of it. And I didn't have an end right there. Oh my goodness, few mistakes there. Anyway, trying to run it again, I got quite a few mistakes in this one. First of all, I got a parser error syntax error near this uh this left join. So, first of all, it's skills job dim on as or as SJD and I never did an on where job ID is equal to the job posting's fact job ID. All right. For real this time, let's see if it's going to work. All right, and it did finally work. We have our table back. Aggregated at a monthly basis for each of these. And then it goes through and it performs the counts for this. This is uh yeah, these numbers are low for marking if it's remote, health insurance, or whatever. It's not the total count, so a lot of these are false values. That's why it's returning zero. So, I think we're fine to go now to build that outer one to actually start getting counts for all this. So, we're going to put this into a CTE called job postings prep as and then put it into parentheses. And then from there, we're going to use this CTE. So, we're going to select the skill ID, month start date, job title short. Next up, our first aggregation, we want to actually get a count of all the job postings. So, we're going to do a count function. And this is going to be doing a count star. And I think I I've actually going back to what I said earlier about this, the zero or one right here. Now, it's important to understand with this, right? This is at the bottom when we look at this, this is 7. 48 million. Our fact table of job postings fact had around 1. 62. But with this 7. 48 million, I'm actually realizing that this is too high or higher than what I would expect. Mainly, I went back and I inspected the query up here. And right now, we're doing a left join on the job postings fact table, and that's preserving, like I said, all the job postings. Well, the problem with that is we only want to look at skills or jobs or you know, jobs with skills, and there's some jobs without skills. So, technically, we need to perform in this case an inner join. So, whenever I run this, we go from that 7. 48 million rows down to 7. 19 million rows, and that makes sense because about 300,000 jobs didn't have any associated skills from what I recall. So, that's more in line with what I want. Anyway, I guess we're getting on a tangent. But the main point I was getting to, this count star is going to work out appropriately based on how we're going to aggregate this because, yes, we have 7. 5 or 7. 2 million jobs in there or 2 million skills in there that we did an aggregation for, but we're going to be aggregating it down, like I said, on these three things of skill ID, Monster day, and job title short. So, we'll check the numbers when we get there. Anyway, from there, let's move on to we want to do a sum of is remote to get the counts of this. So, we'll call this remote postings count. Sum of has health insurance, and this is as health insurance postings count. And then sum of the no degree required, and this is as no degree postings count. Also, one little minor thing, I'm pretty particular about this. The column name is actually job no degree mentioned. Like, it doesn't mention degree. So, I'm going to change this from required to mentioned. And then, change this one also update this column to no degree mention postings count. Just want to make sure that whoever is downstream of this is sure of what data they're actually analyzing. Anyway, we're going to be doing this all from our CTE of job

### [14:00:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=50400s) Segment 169 (840:00 - 845:00)

postings prep. And then, we did aggregation. So, we need to do group by. We can list the columns, but we're in DuckDB. So, we're going to go group all. All right, let's go ahead and run this bad boy. All right, and looking good. As we can see up here at the top, we need to actually organize this by month start date. But, we can see something like senior data analyst had 312 job postings. And then, out of those, 32 were remote and 81 were health. And based on percentages that I know just off the top of my head, those numbers make sense. So, I know that count star is correct. Which says count star right here, I didn't give an alias to this name. I think I got on a tangent. So, I need to add that of as postings count. And then, also, I want to add that order by. We'll do skill ID, month start date, and then job title short. All right, let's go ahead and run this. See what this looks like. All right, this looks a lot more readable and presentable on what's going on here. It's ordered correctly and then has all the appropriate values in here. All right, looking good. Now, we need to do the create and insert into for this. So, above this query, we'll start with our create table. This will be the skills mark schema and our fact skill demand monthly. We'll do this as And getting ahead of myself, we're not doing a CTAs. We're just doing a create table statement. So, inside of the parentheses after this, we need to now list the columns. So, I went through and put all these in of skill ID as integer, month start date as date, job title short as varchar, posting count, remote posting count, health insurance postings count, and no degree mentioned postings count all as integer. Now, we need to do the insert into our skillsmart fact skills demand monthly table. Once again, we want to specify the columns as good practice, and it's all the columns above except with their data types. Okay, I did forget something for this create table. Remember, we wanted to set up primary keys and foreign keys appropriately. And I don't think Yeah, I don't have them added to any of our tables. We [clears throat] actually need to Oh, sorry, I do have them up here. I have them up here for the primary keys for those two tables. So, for this table, our primary keys the skill ID is repeating, month start date is repeating, and job title short can repeating. Because of that, the unique primary key we can make is a combination of all three of those. So, I'm going to go ahead and just copy these right here and then paste them in. All right, next up is we want to do the foreign key. We'll start with our skill ID first. And this references our skillsmart dim skills table and that skill ID. Then our next one up is our foreign key for our month start date where we reference our dim date month table with month start date as the column. All right, so this is looking good. Let's go ahead and save this and now try to run it. We're going to do in that dot read command on create skillsmart. sql. I already have my first error in that I can see where our create table statement we never put a semicolon at the end of this. I'm going to add that in, and then I'm checking my insert into. Yeah, we did have it at the end of this. So, we're going to go ahead and save this. Let's try to run this again. And ran just fine. All right, so now that we have all of our tables created and inserted into, we're now going getting my favorite part of performing data validation. Since we have three tables, I'm going to take a similar approach that I did whenever we load our schema. Remember we did this portion right here at the bottom where we use union all and basically got the count for each of those tables. And then after this, we do a basically a five row sample of each of those tables. So first, I start by selecting the skill dimension in quotes as table name, the count as record count, and then of our dim skills table. I do a union all and next we get to our dim table or a date table. Look at the count for this and we're doing this from dim date month. And then finally, our skill demands fact table's last doing the count of this. And I have a typo in this. This says table with name dim skill does not exist. That's because I misspelled it. I forgot to add a S on the end. All right, let's go ahead and try again. All right, we can see these values. Yep, this makes sense. All of these values, 262 in the skills dimension table, 30 in the date month, and about 52,000 in the skill demand fact table. Next up, I query the first five rows out of the skills dimensional table. I do the same type of query for the date month dimension table. And then finally, from the fact skills demand monthly. Okay, let's go ahead and query these three and they all came back good. We saw the skills down, date dims looking good, and also our fact table. All good. Now similar to what we did for those other tables, I'm going to just put these other messages when we're loading each one of the tables. So in this case, we're loading the skills dim table. This one added for the date dim

### [14:05:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=50700s) Segment 170 (845:00 - 850:00)

table. And then this one for the skill fact table. All right, all buttoned up and clean. It's part of good measure, we're going to run read on this file. And it goes through, tells me it's loading all the different tables, and then provides the key statistics about the table along with those final queries we're looking at. Everything looking good. All right, so as always, we're getting towards the end now. We need to update our master build script. We're going to add in that fourth line or that fourth step. In here, I put the step four for creating the mart and then added the file in using that dot read command. All right, go ahead and save this. Once again, I'm pretty OCD and I like to test every single time all the way, so I'm going to test it one more time for this build script. So, I'm going to quit out of DuckDB right here. And then come up to the top and grab this line that we've been saving. Paste it into our terminal, get to building our data warehouse and data marts. All right, so that took about a minute to run. Scrolling through this all, we're not going to walk through everything again. Uh scrolling through it all, I didn't get any error messages, so I know that it built correctly. All right, looking good. Also, not just looking for error messages, also looking to make sure that anytime you have rows that you're querying that they actually return rows along with their associated counts. Anyway, first thing we need to do is now uh get into committing these changes to our feature branch. So, we're going to do get add to add all these uh these two files and their additions. Then we'll run get commit with a message of build skills demand mart. Okay, that's updated good. We'll run get branch and we are on that feature branch, so we're going to now get switch to our develop branch. We'll run get branch for good measure before we do that. So, okay, we are on develop. So, now let's do our get merge of that feature branch into our develop branch. Not bad, it looks like everything was added. These two file changes were added into it. All right, so you know what we got to do, we got to clean this up. We no longer need that feature branch, so I'm going to run get branch and then a D flag to delete our feature branch. Running get branch again, yep, it's deleted. We have main and only develop. All right, great job on building this skill mart. This is a great exercising in demonstrating your abilities to perform dimensional modeling. This hopefully gets your head around how you may have to build different types of data marts or even data warehouses for your stakeholders. All right, in the next video we're getting into our third and final data mart. With that, I'll see you there. — All right, welcome to the final mart to build and this is going to be similar to or based off of the priority mart that we built back in that DDL and DML lesson. This mart is going to be an excellent demonstration of your skills in performing things like batch loading or incremental loading as I've also called it. Anyway, for this lesson we're going to be going over why we're building this mart, then get into actually creating the mart, and then we'll create another script that actually performs those incremental updates to the mart. Then finally from there we'll update our master build script, commit, and then merge all our branches. So as a quick refresher, back in that lesson on DDL and DML, we built out a priority roles table and also a priority jobs snapshot table which is based on merging, well, initially combining job posting specs, company data, and priority roles into our snapshot table and then from there performing incremental updates with merges. So for this lesson we're going to be building out that priority mart and it's going to contain those two tables, priority roles and that priority jobs snapshot. With snap which the prior job snapshot is obviously based off of from our data warehouse and that priority roles table. After we've created those two tables, we'll create another script that will go through and simulate updating priority roles and then performing that incremental update to priority job snapshot using merge. Now, these incremental updates are what's also known as batch loading is super popular in the industry as you normally have something like batch loading or you have streaming, which instantly puts the data in. Anyway, data analytic pipelines need to be updated at most daily for stakeholders or business leaders to make decisions. They don't need streaming data. So, batch uploads are super frequent in data engineering. So, that's why I think it's super worthwhile for you to include this in the project. All right, so what's going to be the Git workflow for this? Well, first we're going to create a new feature branch for this priority mart. Then from there

### [14:10:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=51000s) Segment 171 (850:00 - 855:00)

we'll go through the process of building out the scripts for it. And when we're done everything, including the data validation, we're then ready to merge it back into our develop branch. All right, so let's start by creating that feature branch. We use Git switch with the C flag and then name that new branch feature/priority mart. We'll run Git branch double check. And yep, we're on that feature priority mart branch. Now, for this one, we're going to add two files. Both the files we're going to need for this. First, we'll be adding the create script and I'll start with 05 and then say create priority mart. sql. So, that's going to be our create script. We also need for our incremental updates. So, that will be 06 and that'll be update priority mart. sql. All right, so let's go ahead and get started. We're starting in that number five file for creating the priority mart. We'll start with a comment that this is step five for mart create priority roles mart. Like all of these, we're going to be using a schema for the mart. So, we need the create schema for priority mart. And then we want it to be adding a potent so we'll also use that drop schema if exists priority mart cascade that cascade is to drop the schema if there's tables inside of it which there will be. All right, so the first table to create is our priority roles table and you we can go in and dig into our notes for this one but this one's honestly so simple and we also did a lot of manipulation with it that I'm just going to build this from scratch. So we're going to create table first and this is in the priority mart schema called priority roles. We'll have our role ID it's an integer and it's a primary key. Role name is next is a varchar. Priority level is last and it's an integer. From there we're going to just insert into it directly the values we want to put for this and this is for the priority mart priority roles table and as best practice we'll go ahead and list out the columns of role ID, role name and priority level. Then we'll list out the values we want to put into this. First will be data engineer and we'll give it a priority level of two. Second will be a senior data engineer with a priority level of one and last will be that software engineer with a priority level of three. Up ahead here I'm going to go ahead and create that query that displays that we're loading this table of loading roles for priority mart. Also I'm going to just go ahead and do the data validation for this of a select star from priority mart priority roles. All right, this is looking good. Let's at least test our script there's enough lines in here. So in terminal we'll start up duck DB um also make sure that we're navigated into that project two folder to start with for this. But we're going to just go with the command duck DB and then specify our data warehouse uh duck DB file. And from there I'm going to run a dot read command of our 05 create priority mart. sql script. And I have well two errors. First off is the insert into well mainly I forgot to put a semicolon after our create statement and then I think that caused our second error of table name with priority roles doesn't exist because it never got executed. All right, make sure we save that file. I'm going to press up, rerun this. All right, looking good. Loading roles for Priority Mart and then all of them appear right here. Good. Next, we need to build out the priority jobs snapshot. For this, we're going to go into our notes that we previously had and I'm just going to copy and paste and adapt as necessary. For this, I'm going into I named it priority job snapshot. initial under my 1. 2. 4 and this has the create statement. It also has the insert into and conveniently has the little data validation at the end. So, I'm going to just copy this all. Inside of here, I'm going to go ahead and paste it and now we need to go in and what's called as refactor it because right now the schema is wrong on here. It's pointing to the main schema and that's not correct. We want to actually call that the Priority Mart schema. So, I'm going to go ahead and copy this. We're going to place the main in our create or replace table statement. We'll also do it in the insert into and then down here, we job postings fact is in the main schema, so we don't need to specify it. So, I'm going to remove this data jobs. Also, it for the company dim and our priority roles is no longer in a staging schema. We're going to update that to also be in the Priority Mart. The only thing last thing is down here for our data validation. We're going to go in and update this to say, "Hey, from Priority Mart. " Okay, one other thing to do. I want to get a table that says we're loading this Priority Job Snapshot. So, I'm going to paste this here and say, "Hey, loading snapshots for Priority Mart. " Okay, I think we're good. We're going to test it out. Inside of terminal, I'm going to press up. We're going to read this file after we saved it and it looks like it went through. We got data engineer, senior data engineer, and software engineer. And for some reason in our table, we only got dead engineer and software engineer. So, did some investigating, found out what the issue is. Issue is

### [14:15:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=51300s) Segment 172 (855:00 - 860:00)

here. We're using So, the inner name in here, I misspelled senior dead engineer. It's supposed to be well, I have a good year. Uh I'm going to add the end. Now, this is a problem with the database design. I'll be completely transparent with you. In that we're using this priority roles table, right? And so, we're relying on whoever manages this, they spell the names exactly right. Cuz then in our priority jobs snapshot, in order to get the correct roles, we're matching on a name. We're literally doing job title short from job posting facts to the role name. I would say this is not necessarily best practice. Normally, what you would expect is some sort of, like I said, ID here. Some sort of integer numerical value that there'd be less likely of an issue of somebody uh fat-fingering it. Unfortunately for us, our job title short column does not We didn't do a dimensional table for it like we did for the companies. So, if we did, that was it That was how we would probably That's actually is how we would link the role names. But since we don't, relying on this method, this is not necessarily the best, but definitely something you should be aware of. Anyway, thanks for coming to my TED Talk. Let's actually, now that I saved that, I'm going to read this file. And uh whenever we rerun this now, it has all the appropriate roles in it. So, we have our create table script built. We're now going to move into our incremental updates. For this, we're going to first start by in our script simulating that the priority roles table gets updated. As we talked about in that lesson, you know, it could get updated as much as daily. So, we'll do a couple of different changes in there to that priority roles table. And then from there we're going to modify or refactor our merge code in order to incrementally update our priority jobs snapshot table. So, let's get started in that six update priority mart SQL file. We'll first start with adding a comment. This is step six for our mart update priority roles, and then we're going to be doing two different uh things here. In that we're going to update the data engineer to priority one in a priority roles table, and then we're going to add data scientist as level two. No, we'll make it level three. So, for this we're going to run an update, and this is going to be in our priority mart schema for priority roles. We want to set priority level equal to one, and that's where role name equals to data engineer. That'll be good enough for that one. From there we're going to insert into to add that data scientist to our priority roles table, specifying those columns of role ID, role name, and priority level. And then we need to specify the values, and that'll be four, data scientist, and of the level three. After this I want to do some data validation to make sure that I have the correct priority roles table. So, I'm going to come into that five script, and I'm going to go ahead and just take this uh select star from priority roles. Also, up above here, just as a statement, I'm going to modify some this statement up here in that it's saying updating roles for priority mart as info. And then after that I perform the query. Now, important thing to understand with this, right? This script, it will be, if you will, idempotent. In that whenever we're building our master build script, as long as we run five before six, you know, we can perform this update and insert into. Now, if we try to run six alone over and over again, it's not necessarily going to be true because we're going to be doing that update and insert into, and so those will have already been performed. But, that's not the point of our idempotency. We're trying to make it to where we can run it with the master build script, so that's just something you need to be aware of right now while we're going through this design phase of it. Anyway, let's test this. So, in here, I'm going to go ahead. So, we're going to run a dot read, and we're going to do this on 06 update priority mart. sql. Go ahead and run this, and we have our first error, and that I forgot to put a semicolon after our insert into. Okay, so now if we try to run this again, it works. And then if I try to run it one more time, I'm going to get an error right roll ID 4 violates key constraints because that's a primary key, and we can't add data scientists again. Anyway, not a big deal, right? We just need to remember to run 05 create priority mart, and then run 06 update priority mart every time we do this to maintain that item potency. All right, so now we want to add in that script we used to incrementally update priority jobs snapshot performing the merge. We also used a temporary table in addition to that merge. Let's go find that. So, for you that should be in under 1. 2. 4 priority jobs snapshot file file, and I'm going to get this temp table. We're not going to use that

### [14:20:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=51600s) Segment 173 (860:00 - 865:00)

update insert or delete statement that we generate, but instead we're going to get this merge into, and then also this final check query. We discussed about the use of CTAs, but we're not using CTAs for this. We're using that merge into. So, I'm going to go ahead and just copy this all, paste this in, and then remove this commented out portion cuz that's just going to confuse people on what was happening here. All right, so now we need to go in and refactor this. We'll start with our temp table first. Job postings fact is in that main schema, so we're going to remove it here, and also next to that company dim. The last place to update is right. We don't want staging. So I'm actually going to just copy priority mart up here cuz we're going to do this a few times. And replace staging priority roles with priority mart priority roles. Okay, temp table is updated. Moving into the merge into, we're not merging into the main schema. We're merging into the priority mart schema. And scrolling through, I don't think there's anything else to actually to update except getting down here to our final check query or data validation. We need to be the priority mart priority jobs snapshot. Okay, not bad. I do want to dress this up a little bit. I had this line here [snorts] of above the temp table for creating temp source table for priority mart. And then above our merge into, I put a statement of batch updating priority job snapshot for priority mart. Then from here underneath, we have this final check query. All right, let's go ahead and save this and let's test it out. First thing we need to do is run that number five script of creating our priority mart. And then we're going to run our six script of updating the priority mart. And so this looks like it performed just fine. We said updating roles priority mart, it added data scientist, updated data engineer. It then created a temp table and then it performed the batch update. And then we can see that the software engineer and the senior data engineer were performed at different times than the data engineer and the data scientist, which were after the fact because we had to update the level and update table at the data scientist. So this checks out. All right, so both these files are looking good. Make sure they are saved. There's only one last thing we need to do now is update our master build script. In here, we're going to update step five for our create priority mart and then our step six for updating our priority mart. Go ahead and save this and then just for good measure for me, I like to just run this entire script. So in terminal I'm going to quit out of here and then I'm going to run the command that's in the top of our script right here because I saved it up here. Paste it in, maximize this. Now let's run this. All right, so not bad. This took only a couple minutes for me to load through scrolling through the logs. Everything looks like it's adding up so I don't have any issues with this. Pretty impressive what we built so far. Actually say what we built. All right, so you know what we got to do now? We got to commit these changes and then merge the branch. So I'm going to do get add dot to add the current directory. There are stages the changes are all staged now. So we're going to commit them with the message build priority mart and update mart. All right, so this adds all our files. I'll just double check if I should have done this before what branch we're on. We're on the feature branch still. We need to merge this into the develop branch so we're going to switch to that branch. So I'll switch over to the develop. Then from there we'll run get merge with the feature branch. All right, and we can check our little graph over here and see that we're build priority mart and update mart is in the develop branch and that is in line with that feature branch. We don't need the feature branch anymore so we're going to run get branch with the D flag for delete and then list that appropriate feature branch. Go ahead and delete it. It was deleted. Run get branch for good measure. Now we're actually not really done because so we did go through a good job so far. We built this data warehouse from our CSVs and then we built our three data marts. But if you notice from our diagram, right? We said we're going to get our job posting warehouse and our marts onto mother duck so that way we can serve this to other customers that can then now access this in mother duck via Excel, Power BI, Tableau or Python. Anyway, how the heck do we do this? Well, actually, it's pretty simple. So, previously we run this DuckDB command and we've been specifying our data warehouse marts. duckdb that we want to run it locally. All we need to do now is change this to Motherduck on where we want to build this. So, for this I'm going to run duckdb and then do motherduck is where we want to run it. And I'm going to call this data warehouse marts. Then once again, we'll run that C flag and do that dot read command on build data warehouse marts. sql.

### [14:25:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=51900s) Segment 174 (865:00 - 870:00)

Okay, let's go ahead and run it up here. And silly me, I got this error unable to open database data warehouse marts. Basically, the gist of this is that the database is not created. We haven't created it yet. Anyway, if I were to go to duckdb and just do md: and then run dot databases, we have md information, sample data, data jobs, jobs mart which we created at one point in my db. We haven't created this location of this data warehouse mart that we want to name here. So, because of that, we just need to run a command of create database and then what we want to name it. In our case, data warehouse marts. Now when I run dot databases, we can see that we have at the bottom of this data warehouse marts underneath it. All right, so I'm going to quit out of this and now pressing up to go back into that command, we're now going to run this. And right now it's going through the loading information. I'm going to go ahead and maximize this as it's going through and keeping track. All right, so this went through and just skimming through all these different commands, everything worked exactly as it was supposed to. It looks like there's no errors. So, completely good to go with this. And so, if we navigate on to Motherduck, we can go in inside of your login here. I can go to attach databases right here and I can see the data warehouse marts. In here I see our flat mart, our main mart, which is our actual data warehouse, our priority mart, and our skills mart. If I want to, I can create a new notebook and inside of here we can query our data warehouse marts table, querying that maybe even that job postings fact table, and everything appears just fine. Anyway, the main point of showing you inside of here is this. We can go here to the data warehouse mart, click these three dots right here, and we can go in and actually share this. First, you need to specify who has access. You can do anybody in your organization, if you're working in an in your organization. Anyone with the share link, that's how I shared the initial database at the very beginning of this course. Or you can even do specified users with the share link. From there, as in this case I selected anyone with the share link, I have it to where it's auto updating. So if you were to have new data populate, and then you were to refresh this data inside of here, your end users will get that update. Then from there, you can change the share name as you want. I'm just going to keep it as data warehouse marts. And then from there, create share. Now all you'd have to do to get it out to your stakeholders is give them this attach command, similar to what you had whenever I started at the beginning of this course. They give this attach command, they run it inside of Mother Duck, wherever they're at, and they'll have that database on their attached databases. You can also navigate to any of your shares later down here at the bottom by going to shares I've created. And right down here I have data warehouse marts. Click on this, and then you can go to copy share link, and recopy that link whenever you need it. So we have actually finally satisfied this entire data pipeline, building it from CSVs in our data warehouse, and then our three data marts. Congratulations on this. Now, if you're feeling spicy and you want to get more experience building another dimensional mart, I've put together a company mart that's a little bit even uh well, a little bit more complicated than our skills mart. You can access the final script to see what to base it off of and how to actually build it out inside of my project two folder in GitHub. Inside of here in the files, you just navigate to this number seven one of create company mart, and this has all the different code needed in order to generate this bad boy. Uh this thing is a behemoth. Now, for those that purchased the course practice problems and notes, I have a specific lesson page on this. In this, it breaks down what the heck this company prospecting mart is, explains all the different tables of it, how we're going to build it out, and then walks you through step-by-step on what you need to do, what commands you need to run in order to build this out, and why we're actually doing that. All right, so awesome job on this pipeline. Now that we've had this built, we need to get it out and get it documented it properly in a read me, and then from there share it. So, the next step we're going to be jumping into is building out the read me for this project. With that, I'll see you there. All right, in this video, we're going to be putting together our read me for our project two. We're also going to need to update our read me that's on our entire repo of that sequel. engineering project. Or I mean, projects. Anyway, remember we have two projects in there, and mainly the purpose of this entire repo

### [14:30:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=52200s) Segment 175 (870:00 - 875:00)

is so you can add additional projects as you build them to this repo. Anyway, for this, I'm not going to handhold you as much as I did in the first project is you should have the basics of how to work with markdown and write in and build out a readme. I will be walking you through the sections that I do recommend though. And I'll briefly cover some core topics or core things that you should cover in this. But I can't stress this enough, you need to build this readme and adapt it to your own personal need. If you're say looking for a specific job like an analytics engineer role, you need to frame it more towards analytics engineering. I'm going to be just going at a general approach, something as a data engineer. So like I said, we're not going to be going through and actually going through any of the syntax again for how you should be updating or writing in markdown. I did include this link for this markdown cheat sheet, which I find is a great resource in how to build this bad boy out. Additionally for this readme build, I'm going to be using specifically in this one a lot of different pictures to not only show the pipeline, but also for the data warehouse, to the flat mart, and all these other different pictures. Well, you can steal I mean use those pictures if you want. If you navigate to the main repo inside of here under the resources/images folder, I have all the different images that you need within here. Right here is the project two data pipeline, and all you got to do is come up here and download the raw file. And then from there just take it and put it into your image folder on your repo. I ended up just moving all those different images into here. All right, for this also I do want to call out we are on the develop branch, which I'm fine with now, right? We've done all of our features. Now we're just updating the old documentation. So perfectly acceptable to do this on the develop branch. We're going to merge it into main at the end. So let's get into building this out. I'm going to navigate into our project two folder and update or open our readme. Conveniently in here we left this to do of fill this in. All right, so there's a few core sections that I want to do for this and it's going to be very similar to what we did in the last project. Actually, I'm just going to take that last project and I'm going to I know it started here at this executive summary. Take all of it, the contents, paste them in, and then delete all the different sections within it keeping the headers. All right, so this gives us an idea of the sections we're going to need for this. I'm just walk through briefly what I've included in each. First thing I did was a one-line sentence basically explaining what we did with this pipeline and then included an image of the pipeline so it really captures their attention of what's going on here. Next I got into the executive summary focusing on key things that I found are most important for data engineering such as pipeline scope, data modeling, ETL development, and then the mark mart architecture. How we used like a flat mart, a dimensional mart, and also a automatic updating mart, or better said incremental updating mart. Next I got into the problems and context. I started with the challenge first explaining why we needed the data warehouse and then why we need each one of those marts. And then from there I talked about the solution of building this end-to-end pipeline that extracts the data from our CSVs into those locations. Next is the tech stack and this is very similar to what we did last cuz our tech stack hasn't really changed since project one. But I did add in here for the language that we have we're using DDL for schema design and DML for data loading and transformation. Oh, and then also these other two of the automation of our master SQL script for pipeline orchestration and that we use Google Cloud Storage for source CSV files. Now instead of analysis overview we're going to change this up a bit cuz we're not doing an analysis here. Instead we're going to go in-depth with the pipeline architecture that we did. I start this section with the overall architecture itself of the image we used above and basically explain it in a couple sentences of what we did. Next, I go into each individual database or if you will data warehouse or mart that we built. Starting first with the data warehouse and with this one I link to the appropriate SQL file that we did to in this case create the tables and load the schema and then from there explain the purpose and the granularity that this schema goes to. Next up is our flat mart which demonstrated our uh ability to denormalize dimensions and I included attachment to the file we used for this along with purpose and grain. I did the same with that skill mart which for that time series skill analysis with added to measures defining the file purpose and grain and then finally that priority mart which demonstrates our ability to perform incremental updates or batch

### [14:35:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=52500s) Segment 176 (875:00 - 880:00)

processing. For this one remember this one had two files so we needed to link those appropriate ones and also went into purpose and grain. Now, if you went ahead and did that company mart, go ahead and also list that here along with the appropriate SQL file and purpose and grain. All right, last up is getting into the SQL skills demonstrated for this and I'm actually going to re- uh format or reframe the title of this section to data engineering skills demonstrated. We're going to actually change these uh subsections as well because we did more than just SQL in this. We did a lot of other things as well. For me, I broke it down into three main sections. The first was around ETL pipeline development talking about how we did ETL along with that demonstration of this incremental updates and how we used our master build script to orchestrate it all. I then focused on our ability to perform dimensional modeling cuz we not only did that with our data warehouse, but also with our skills mart that we built. And then from there it moved into advanced SQL techniques that we did. We covered a lot of SQL concepts in that little second half of this course, specifically around DDL and DML. So we need to make sure that we capture that here. And sorry, we did cover a lot of stuff, so we actually I made a fourth section around data quality and production practices. And that how we actually designed all of our scripts to be idempotent, to make sure that we perform our referential integrity checks, we're ensuring that our data's meeting a specific type. Want to capture that all here. All right, and that's it for this read me here. So I'm going to go ahead and save this. This is now good to go. And we just have that one more script to update or one more readme file. Remember we have our project one and our project two. Then we have our main readme. And this readme directs them to the appropriate project depending on what skills you're trying to showcase. Remember that first project was around EDA, exploratory data analysis. They know here to go to that one. We needed to do the same thing for our second project. So for this I left it the same structure and where it's a clickable link up at the top to direct them to the appropriate folder. Make sure I click this. Oh, it's not even it's not working. And it's not working because I spelled the file path wrong for this. Should be data warehouse mart build. Let's go ahead and try this again. I'm going to click it. All right, and it's navigating me to the explore pane's opening up that. So it's going to work fine in GitHub. Make sure all your links are working appropriate. Nothing worse than some future employer come to take a look and your links are broken. It shows that you don't have attention to detail. Anyway, besides that include a picture to try to entice them to go and read into this more and I just include a simple description underneath it along with the skills demonstrated. So, I'm going to keep it simple with this one. You can feel free to dress this section up more if you want to include more appropriate skills or whatever on this, you can. I really just leave it up to you. All right, so most exciting part. We're going to now get into actually committing all these changes and merging this into main and getting this onto GitHub. Make sure that you've gone ahead and saved all your files, specifically those read me's that we recently just did. For this, I'm going to go ahead and list get branch to show what branch we're on. We're on the develop branch. We need to get this into the main branch after committing it. So, we run get commit and then M. Oops, getting ahead of myself. We're going to run get add period for at all files in this directory. They're all staged. Oh, that's why. I'm not in uh so, I added only that in that this directory. And it only added the read me from our project two folder because in the terminal, I'm in the project two folder. I need to actually go up to the main directory and now run get add period for the current directory. Everything gets added in. Now, we can get into committing our changes with the message of add read me with project documentation. All right, seven files changed, added all these in. This is looking good. We need to run get switch to main. All right, anytime we're about to merge in uh new changes, we need to make sure that we have everything appropriately pulled in from our origin up in GitHub. So, we run get pull and it says already up to date. Remember, we could run get pull origin main, but we've already specified at the beginning and it was is not necessary to specify again. Anyway, we're going to run get merge and this is from our develop branch for project two. Exciting times. All right, so this added in all of our different SQL files for building out the smart along and our data warehouse along with our readme files. And then those images if you stole I mean took them to use in here. Now because of that, you know, what I like

### [14:40:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=52800s) Segment 177 (880:00 - 885:00)

to do for cleanup, we need to get rid of that develop branch. So we're going to run get branch with the D flag specifying to delete the develop project two branch. All right, and looking good running get branch just check. Yep, it's all up to date. Now we're ready to push so we can just go get push and bam, GitHub now should be updated. But you should always go check it and make sure it is. Key things I like to check with this, here I'm verifying okay everything looks like it's up to date. I do go through and still double check clicking the different links. Does it navigate to where I want to? Are all of the images also appearing? And it's looking like everything is good. Nice. All right, so we have it updated on GitHub. Now we only have one step left and that's to get your project shared properly on LinkedIn. That's what we'll be doing in the next lesson. All right, with that I'll see you there. All right, congratulations. This has been nothing short of your hard work put in to building out this entire data engineering pipeline. And so we need to showcase it. In this lesson we're going to be going over well this is our final lesson actually. In this lesson we'll be going over how to add your certificate if you purchased the course practice problems you'll be getting a certificate. And then also updating your project session and a suggestion of making a post to showcase that you have both these things. All right, first thing probably most curious about is that certificate of completion. After you complete the end of course survey for those that purchased the course practice problem there's a survey at the very end. After you complete that, I'll email you your certificate with your name on it. So, you'll get it that way. This will have a link associated with it along with you can download it. For this, you're going to navigate to your profile on LinkedIn and then go to your licenses and certificate section to go ahead and add this in. Note, if you don't have this section, you're just going to come to the top, go to add profile section, add licenses and certificates. Then from there, you're going to go ahead and fill this out for it. For this, you have the name of SQL for Data Engineering, list me as the issuing organization, when the issue date is, there's no expiration date. Then from there, paste in that credential ID with the certificate along with the credential URL that links to the certificate. Then under skills, I've listed five of the most important I found of SQL, data engineering, Git, terminal, or GitHub. Then from there, you can also do add media if you want. You can add a picture of the certificate just as a nice to have. Anyway, this looks good to me. I'm going to go ahead and click save. Next up, we're going to add the project itself. So, we're under project section on your profile, we're going to go ahead and click add. For this, we need to fill out all these sections. Don't reinvent the wheel here. Use anything that you've used from your readme, just repurpose it for this. In this, I include the project name, the description or the one-line sentence that we use at the front of it along with those five core skills. Next up, it has add any media. For this, I'm going to navigate into our original projects folder. And then I specifically want to link to our second project, so I'll navigate to that. Oh, I want this one. I'll copy the link here. And then in LinkedIn, I'll go to add media, add a link, paste in our GitHub here, click add. And it looked like in my case, it couldn't do it, so I just went to the root of this SQL Data Engineering project, and then in the description said, "Hey, navigate to project two in the in this repository, click apply. " And then from there, include your start date and finish date. I started this almost 6 months ago building this, so I hope you found it worthwhile. Hopefully, it didn't take you 6 months to actually complete, though. And then if you worked with anybody along this, feel free to add contributors. Okay, everything looks good. Now, the last thing to do is to create a post, and this is to broadcast out to the community that you've completed this, and also start getting eyes on your project. Anyway, Carol did a great job with this in that Carol recently completed it and not only posted their certificate here on LinkedIn, but also walked through the different parts of that Power BI course and what they did. And so, you can do the same. Calling out those key skills, and then from there at the bottom included a link to the project. Now, also don't forget to tag me in it as I love to be able to get notified when you're doing this and be able to comment on it and check out your projects. All right, congratulations again for building out this behemoth of a project that has been nothing short of your hard work. So, congratulations. I'm planning on building out a lot more data engineering courses. This is the first one in my series. I'm hoping to go next into things like DBT, Python, and then more narrow down further to such as Airflow and Spark once I have those. But, this all depends on if people are interested in it and if they're taking the course. So, make sure that you're

### [14:45:00](https://www.youtube.com/watch?v=UjhFbq4uU2Y&t=53100s) Segment 178 (885:00 - 885:00)

sharing this in order to get the word out and actually help with funding future courses. All right, and with that, I'll see you in the next course.

---
*Источник: https://ekstraktznaniy.ru/video/23795*