the most underrated data job in 2021

i've been noticing this trend myself but i didn't have the numbers to back it up until interview query released their annual report growth in data science interviews actually platformed in 2020 and grew by only 10 after previously growing by 80 year-over-year for the past few years yet data engineering specific interviews increased by 40 in the past year that's right the most underrated data job is data engineering with rising numbers of job availability and an average salary of 151k which by the way is higher than the average salary of a data scientist which is 139k this is a job that is somehow way under

the radar for people looking into data related jobs so without further ado in this video i'm going to walk you guys through what a data engineer does and why it actually makes perfect sense that companies are aggressively hiring them and of course point you towards how

to become one data engineers often describe themselves as plumbers of the data world they are responsible for ensuring that raw data is transformed into data that is clean accessible scalable and consistent to allow users to consume the data in an efficient and useful way or in a more concise term etl extract transformation on a less abstract level if you've ever had to go and get your own data and deal with missing values and wrangle it you will know that it is no small feat but now imagine that at a huge scale as well and that is where data engineers come in data engineers first need to bring in data from a variety of data sources which are often in a bunch of different and unintuitive formats they need to clean it up nicely and write data pipelines to streamline and automate the process since lots of data is time indexed and you certainly don't want to have to manually do the process of retrieving new data every single day so you have to write these scripts which are commonly known as data pipelines that do the process automatically every day then data engineers build tables from this raw data into new tables that are easier for end users to use the end user often being data scientists such as myself and data analysts or business functions this is necessary because oftentimes the raw data comes in a form that is not very intuitive or nice to query for example if you have say like some data about usage of an app the information might be spread across a bunch of different tables so such that if you want to get user id time of access maybe what they did on the app and how long they used that app in each session you would need to go and join together a bunch of tables not only is this very annoying to do it's also very expensive doing all these drawings every single time well luckily in sub's a data engineer and that puts it all into a single table that has clear column names and that makes your end users such as myself very happy if you think the data engineer can then run off leaving everybody very happy forever and sadly you are mistaken data engineers can't just be like fine after making nice tables for everyone they also need to continuously monitor their data pipelines to make sure that nothing messes up in the ideal situation the data engineer would have thought of all the edge cases but that is very difficult and sometimes things just pop up and you just haven't taken an account for it sometimes it's not even a data engineer's fault because wherever they are sourcing their data from those people might have just decided to go change the way that the data is organized which would then cause the data engineers pipelines to mess up and the tables to mess up you see detail process is really a living process and it also adapts over time finally in some companies data engineers are also tasked with building dashboards and metrics for the teams that they work with i won't go into too much detail about this because the process is just so different depending on the industry the company and the team but i do want to bring this up because i want to further emphasize the fact that data engineers don't work in isolation not only do they need to continuously interact with their end users to make sure that the nice tables that they're building from the raw data are actually the ones that end users care about they're also interacting with many people in the business and making sure that decisions are being driven with data efficiently and effectively so yep sorry if you're a data engineer you can't just in the corner and code buyers all right so this is by no means a comprehensive and in-depth description of what data engineers do but i do have provided you an overview that gives you a taste for the job and perhaps pique your interest like your interest in the role so now let's briefly dive into why data

engineers are so in demand and then go through an overview of the skills you need to become a data engineer alright so it actually makes perfect sense why companies are aggressively hiring data engineers with no stopping anytime soon pldr there's just so much data everywhere really that is available from third-party data vendors and if you work in a product company from your third-party apps and devices this along with increasing belief that everything we should do should be driven by data leads to huge investments and getting as much data as possible from as many sources as possible while this is absolutely awesome because being data driven has shown proven results the issue is that the data is essentially useless unless it's processed so you know you have your data scientists that are supposed to go find gems in the data and reveal insights where build models etc but data scientists can't do that the data is not already clean and structured so we need data engineers to help with the etl process first in smaller companies where places that don't use so much data the data scientist often takes over the job of data engineers but as more and more data pours in doing etl is absolutely a full-time job and it's a job that is highly valuable because here's the thing if you have a nicely clean data set you can already extract a lot of low-hanging fruit insights even without doing like fancy models or fancy analysis but on the other hand if your data is not clean it doesn't matter how fancy your models are or how fancy your analyses can be because well you just can't do them and that's why arguably data engineers are more essential than data scientists in the business all right i assume that you if you've watched until now you're at least a little bit curious about becoming a data engineer so now let's talk about the

technical skill set and how to become one data engineers are primarily expected to know sql and python in terms of languages they are also expect to have a grasp of algorithms and understand how databases work and have some sort of understanding of big data platforms like spark or hadoop please take what i say here with a green assault because just like how data science jobs are extremely variable so are data engineer jobs so these are just the core skill sets that are common to most data engineering jobs and you would have to do your own research into specific data engineering roles at specific companies how do you get a data engineer job just like with data science sadly you can't just learn the relevant skills and expect companies to just give you interviews from my own observation working with data engineers data engineering is more similar to software engineering than data sciences and you get lots of people with software engineering backgrounds going into data engineering and unlike data science you don't really get so many business or analyst type backgrounds in data engineering with that being said i'll also link you guys to this awesome video by carolina that i came across when doing research for this video and it goes into far more depth about how to become a data engineer plus she's actually a data engineer so here you have it a brief overview of the most underrated data role i hope this overview was useful and i'll link below all the resources that i came across when researching for this video alright see you guys in the next video