Python Django Tutorial: Full-Featured Web App Part 13 - Using AWS S3 for File Uploads

hey there how's it going everybody in this Django tutorial we're going to be learning how to use Amazon s3 for our file storage instead of our local file system so using AWS and s3 is a nice cost-effective way to store our files and also it's going to be very scalable secure and have some good performance now I'm calling this video part 13 of the series if you've been following along then you might notice that the last numbered video was part 12 where we added the password reset functionality I had a few videos after that about deploying to a linux server but I didn't number those because I'm going to show multiple ways to deploy our application and all of those deployment methods are separate from actually adding functionality to our application so the videos where we are adding new features will be numbered and the different deployment videos won't be so I just wanted to clear that up so in this video we're going to be moving our file storage to AWS s3 buckets now I actually started making this video because I was wanting to make a video where we deploy our application to Heroku and Heroku doesn't have a standard file system for us to use so they require that we use s3 to store our files and s3 is a common file storage solution for a lot of websites so I figured that we could see how to make that switch so let's go ahead and get started so first things

first if you've been following along with this Jango series and you didn't watch my video about deploying to a linux server then you may not have seen a correction that I made to our Jango code in that video the correction that needs to be made is very small and it's in our users models dot py file so I'm going to open up my application here and that is in our users app here models dot py and the mistake is down here in our save method so when we're calling the save method on our parent class this Super dot save part right here we need to accept any other arguments that our parent class might be accept expecting so in this case we need to pass in args which are positional arguments and kwargs which are keyword arguments so we need to pass those in to the save method so to do this I will just say star args and star Cork's and we need to be expecting these in the save method itself as well so I'm going to copy those and paste those in there as well and again what that does is it allows us to accept any arbitrary number of positional or keyword arguments to our method and we're actually going to be getting rid of this save method later so we probably didn't need that change but if you've been getting an error with that save method I just wanted to show you the fix for that before we get started so with that correction in place let's move on and see how we can now use s3 buckets for our file uploads so first of all you're

going to need an account with Amazon Web Services so this is just like signing up on any other site if you go to aws. amazon. com that i have open here then you'll be able to create an AWS account so just walk through their instructions to get set up with an account on their site now I already have an account that I'm signed into so whenever I click on create an account it takes me here to this management console so once you have an account you're going to want to open up your management console here so that we can create an s3 bucket so this management console here might be a little intimidating if you haven't used AWS because there are so many different services that are listed here so the service that we want to open is called s3 now you could scroll through here and find this but instead of searching through all of these down here we can just search for it here so I'm going to type in s3 and we can see that this pops up here scalable storage in the cloud so what s3 stands for is simple storage service so let me see if I can make this a little larger here so that we can see a little better ok so now what we want to do is create a new bucket so this bucket is going to be what holds our files now bucket names have to be universally unique so they can't just be unique for your account so there can't be any other bucket in the world with the same name so sometimes choosing a name can take some trial and error so I'm going to go to click on create bucket here and I will create a bucket I'll call this Jango - Blagh - files and that should be a unique name now I like to keep my bucket name simple and since you're going to be using these and Bram's I also try to only make them lowercase letters here and only add in dashes if the bucket name is too complicated than sometimes that can interfere with certain programming languages okay so you can choose whichever region you'd like I'm just going to keep this selected as US West Oregon here and now I'm just gonna click through and accept most of these defaults so you can change these if you like and enable versioning I'm not going I'm just going to accept all the defaults here and click through when we get to permissions I'll click Next and finally I'll click on create bucket ok so once we have our bucket created we're going to need to change some permissions with this now I've got these permissions from the Heroku documentation for setting up an s3 bucket so I'll just paste those in once we get this pulled up so to do this I'm going to click on the bucket here and now for our top menu here I'm going to click on permissions and now I'm going to go to this course configuration if I hover over this we can see that this is cross-origin resource sharing it defines a way for client web applications to interact with resources in a different domain so I'm going to click on that and we can see that it gives a sample policy there I'm going to paste the one that I got from Heroku and I will also put this on my github as well so I have this open here so I'm going to copy the one that I have and again I'll have this on my github and I'll leave a link to that in a description section below if you would like to copy and paste this as well so I'm going to paste that in there and then just click Save so this is basically just telling our bucket that we're going to be performing some actions from some domains here now in the Heroku documentation they say to have this allowed origin set to your domain name but I'm setting this to an ask asterisk for now which will allow the origin to be from anywhere now we can tighten this up later if we'd like once we get this application deployed but for now I'm just going to leave that as an asterisk ok so once we've got that bucket created and those permissions set up then we're going to want to create a new user so I'm going to go back to my management console here and we want a

user that has more limited striction x' than our admin user that way it adds an extra layer of security now to do that we're going to open a service called a or IAM which stands for identity access management and again I'm just going to search for this so I'll type in I am and click on that and now we want to create a new user so I'm going to click on users and we can see that right now only have this AWS admin account here so now I'm going to click on users and now for the user name I will just call this Django underscore user and now we have to select an AWS access type here now I'm just going to click on program programmatic access so programmatic access means that we'll use this user to access AWS with an Access ID and a secret access key but we don't need access of the AWS console with this user so I'll just leave that unchecked so now I will click Next and now we've got to give this user some permissions and I want permission to access our s3 buckets so I'm going to go to attach existing policies directly and click on that over here and now for the policies that we want to search I'm going to search for an s3 policy and if we search that we can see what we get a few results here but there's one called Amazon s3 full access so I'm gonna check that one there now Amazon s3 full access might give more permissions for our application than we actually need but we're still protected with our secret access key and things like that now if you wanted to be extra safe then you can learn more about AWS policies and tighten up that policy even further if you'd like but for this series I'm just going to do the Amazon s3 full access so now we'll click Next and I don't need any tags so I'll click Next and finally I'll click create user ok so now that we've created our user it's going to give us an access key ID which we can see here and a secret access key that we can use in our program so that AWS knows that we have the correct credentials for this user now you don't want to put these credentials directly in our code because that means anyone who has access to our code will be able to see our secret access key so anytime we want to access sensitive information like this in our code we just use environment variables or something like that and you can also put those in configuration files that you don't commit to your repository but in this series we've been using environment variables so far so for example if you've been following along with this entire series then you should already have some experience setting environment variables since that's where we put our username and password for our email service but if you don't know how to set environment variables then I do have a separate video on how to do that for your operating system so I'll leave a link to that video in the description section below if you need to give that a watch now since I'm

on a Mac I can put these environment variables in my dot Bash underscore profile file but like I said if you're on a different operating system and don't know how to do this then you can watch that other video that I just mentioned on how to set those so I'm going to do this and my dot Bash underscore profile so let me open this up I'm going to open up my terminal here and let me clear this screen okay so I'm going to open up my bash profile which is in my home folder in order to set those environment variables so that's bash underscore profile now I have that sublime command to automatically open this up and sublime but if you don't have this command setup then you can just open that in sublime manually okay so I'm going to put these here underneath this comment where I say that I'm going to set my Django blog variables so to do this on Mac we can use the export command so I'll say export and first we're going to set the AWS access underscore key underscore ID and we're going to set that equal to it's going to go back to AWS here I want to copy the access key ID that they gave me when I created that user so I will paste that in there now I'm going to copy this and paste this under here but now instead of the AWS access key ID I'm going to also add the AWS the is secret whoops secret underscore access underscore key and this is going to be equal to if I go back here I'm going to show my secret access key and copy that and I'm going to remove this user before I put this video out live so this access key ID and access key won't be active when I release this so you won't be able to use these so don't worry I'm not accidentally exposing a secret keys that will be active on my account by the time this video goes alive okay so I'm going to save that and actually while we're at it let's also add our s3 bucket as an environment variable as well that way it can be updated in one place for our application if the name of the bucket ever changes so let me go to view here I'm going to turn this word wrap off so that that's not going on the next line so I will say export and I'm gonna call this AWS underscore storage underscore bucket underscore name I will set that equal to the name of our bucket is Jengo - blog - files and I'll save that so I'm not just choosing arbitrary names here for these environment variable names these actually have to be named this way to interact with a module that we're going to use here in a bit that we'll see once we actually start using this in our Django application so be sure that you're using at these same environment variable names as well okay so now that we have all of that set up on our AWS and have the environment variables in place let's now change our Django code to use s3 instead of the local file system so first we'll need to install some packages and these will be boto 3 which is the AWS Python package and Django storages which is an easier way for Django and s3 to interact and Django storages is also what's going to use these environment variables here so with that said I'm going to pull my terminal back up here let me clear this out and now I will install those packages so that is pip install that's boto 3 so I will run that okay and once that's installed I will clear the screen here and now I'm also going to do a pip install of Django - storages it's not just storage but storages so I will run that okay and once that's installed I'll clear my screen okay and once those are installed we're going to need to make some changes to our settings not py file so I'm going to open up my project here let me close down these other files that we've been messing with and now I'm going to open up my settings dot P Y so that is within Django project settings dot py and again I'm going to turn off my word wrap here so this isn't wrapping on two different lines

lines okay so first the Django storages package that we installed is a Django app and needs to be included in our installed apps so I'm gonna scroll down here just a little bit until we see our installed apps and I'm going to add this and I'm just going to add it here to the bottom and we don't put Django storages here this is actually just going to be storages so I will save that okay and once we have that added to our installed apps now we're going to need to set some more variables for our settings so let's scroll down to the very bottom and add some of these so I'm gonna scroll down to the very bottom of our settings dot py file here and I will add these to the bottom and the variables that we're gonna add here are actually the same ones that we just added to our environment variables so actually let me open that back up so that I can copy these so I'm going to copy all three of these here that we put in our environment variables and I'm going to paste these in to our settings here and now we want to set each of these to the value that is in our environment variable so to do that this is what we also did with our email user and email password so I'm just going to copy that line here or we're doing Oh s dot environ get that's how you get an environment variable and the environment variable that we want to get we call this AWS access key ID and I will just copy this line here actually let me copy the equal sign too I will just copy this here for these other two and change this one to AWS secret X key and copy this one here and make this AWS Storage bucket name so technically whenever I said that these environment variables had to be named these for our Gengo storages module really they just have to be named that here in the settings not py file but I think it makes it easier to remember if you just have those same names in the environment variables as

well okay so that will provide AWS with all of the credentials that it needs to access these buckets through this user but we also want to set some additional settings here so another one is going to be AWS underscore s3 underscore file up all caps file overwrite and I'm gonna set that equal to false because if we have if a user uploads a file that is the same name as a file that someone else uploaded we don't want those files to be overwritten because then you know if somebody uploaded file that was you know user dot JPEG or something like that we wouldn't want that to change everybody's file that was called user dot JPEG we would just want it to rename it to something with some characters appended to the end okay and another setting that we want to set is going to be one called AWS underscore default underscore ACL and we're gonna set that equal to none so it's recommended to set this to none since the current default of this value can cause some issues and this will be set to none by default in future versions of Jango storage so we shouldn't have to set it but for now we do have to set this to none okay and finally I'm gonna set a last setting here and this is going to be default underscore file underscore storage and I'm gonna set this equal to storages which will use that storage storages app dot back ends dot s3 moto3 dot s3 boto three storage and this is actually all gonna be in a string so I forgot to put my quotes there so I'll add those quotes in there now I didn't just memorize

this stuff here I actually got this from the documentation so if you ever want to look at this then I'm you can just type into Google Django storages documentation and it should be the top result here so if we go to their storages file and then say that we're saving our stuff on Amazon s3 then we can look through here and see all of these settings that need to be set where some of these are optional but the AWS access key the secret access key the storage bucket name here's that AWS a default where it gives its recommendation here so if you want to see more information about everything that we set in our settings file then you can come here to their documentation and look through here and there's a lot of different things and examples that they have so let me open back up our project here okay so

we're just about done here but there's one more thing that I have to do so remember with our profile pictures we set that up earlier in the series so that the pillo library resizes those images by open opening them up on the file system and then resizing and receiving them now this worked well with our images when they lived on our local file system but now they're in an s3 bucket and using pillow is just going to give us some issues so I think it's best just to remove that image resizing for now and if anyone is interested then I can show you in the future video how to do this for files and s3 but for now we're just going to remove that functionality so that functionality is actually in our users app and models dot py and this is actually what we changed at the beginning of the video the one that I said had an error that when we first created it in the series now that was the save method down here and we're just going to you can remove or comment out the save method here I'm just going to comment it out so that we can at least see how we resized those images if we ever want to see that again or you can completely remove it but I'm just going to comment that out for now so now images aren't going to be automatically resized like they were on the file system but for anyone who is curious if we wanted to add that functionality back to our application we could simply create an AWS lambda function that resizes our images for us automatically when they're uploaded to s3 but I don't want to get too far into the AWS weeds in this video so maybe we'll do that in a future video okay so now all of this should be working now we don't currently have our current profile pictures on AWS so if we start up the site now then those aren't going to show up so if we want to get the existing pictures to show up then we'll also have to transfer those to our s3 bucket so I'm gonna do that simply by opening our s3 bucket in the browser and dragging and dropping

our existing profile pictures into that bucket so I'm going to go back to our browser here and s3 bucket so I'll go to our management console and search for s3 go to s3 and we only have one bucket here which are our Django blog files and currently this is empty so I have my application open here in the finder so this is our Django project and our profile pictures are currently saved in our media folder so I'm going to open up that media folder and we have our default and then we also have our profile pictures so I'm just going to highlight both of those and you can add those to s3 just by dragging them over here and dropping them and we can see that it's asking us if we want to upload those and I will just say next and we could set some permissions here if we wanted to but I'll just click next and finally upload ok so it looks like those pictures were uploaded to our bucket so if I was to click on the default JPEG and open that then we can see that is that opens on our bucket so that is good now remember we're not currently resizing these images that get added to our bucket so be careful that you're not dumping you know extremely large files in there s3 is fairly cheap I think it's like two cents per gigabyte or something like that but still you might as well keep it small if possible okay so now that we

have that in place we should be able to run our application with our media files being served from AWS s3 instead of our local file system so let's give that a try so I currently have a server running here so I'm going to stop that okay and I will open up a new terminal here so we get those environment variable changes and now let me navigate to our project so that we can run this now I am working within a virtual environment you might not be using a virtual environment but since I opened up a new terminal here I need to activate that really quick so that is in Jango env bin activate if you're on Windows then you'll activate a virtual environment a little bit different than that but now we can see that I have my Jango environment activated here and that's where I installed all of those new modules so now I can run my Django project so we can do that just by saying python managed py run server okay and that says that it's running on our localhost here so I will go back and reload that okay so our website looks the same let me make this a little larger here it looks the same but if I open an image for our site if we look at the URL for this image then this says Django blog files dot you know s3 uswest Amazon AWS so this is actually coming from our s3 bucket and not from our local file system so that's good that means that it is serving up those s3 files now let me also try to upload a new profile picture to see if that works so I'm going to log in as the core ES user that I created earlier in this series and now let me go to my profile and upload a new image so I'll go to choose file these are going to be located on my desktop so I'll just upload this avatar image here and update that okay so we can see that it updated our image here so if I open this image up in a new tab then this should be coming from AWS now we can see that this didn't get resized so that was expect but if we look at the image URL here this is coming from our django blog files s3 bucket so that is actually uploading those files as well and if we wanted to be completely sure of that then we could actually go back to our s3 bucket here and reload this and if I click on profile pics here then we can see that avatar square was uploaded so if I open that in here then we can see that the last modified time is the current time so that was just now uploaded okay so that seems to be working well now like I said if anyone is interested then I can show you how to create an AWS lambda function that automatically resizes images in an s3 bucket but I didn't want to get into that here since this video is a bit long already but we can see that with those changes that we can now have all of these files being served from AWS which is definitely nice in terms of allowing our application to scale and also we'll be more efficient as well and also if you're deploying to a service like Heroku then you actually have to have these files being served from a service like s3 because they don't have a typical file system that allows user uploads and we'll be seeing how to deploy this application to Heroku in the next video so that we can see what that looks like okay so I think that is going to do it for this video hopefully now you have a good understanding of how we can use AWS to serve files for our website but if anyone has any questions about what we covered in this video then feel free to ask in the comment section below and I'll do my best to answer those and if you enjoy these tutorials and would like to support them then there are several ways you can do that the easiest ways to simply like the video and give it a thumbs up and also it's a huge help to share these videos with anyone who you think would find them useful and if you have the means you can contribute through patreon and there's a link to that page in the description section below be sure to subscribe for future videos and thank you all for watching you