Python FastAPI Tutorial (Part 16): AWS S3 and Boto3 - Moving File Uploads to the Cloud
47:37

Python FastAPI Tutorial (Part 16): AWS S3 and Boto3 - Moving File Uploads to the Cloud

Corey Schafer 16.04.2026 3 428 просмотров 170 лайков

Machine-readable: Markdown · JSON API · Site index

Поделиться Telegram VK Бот
Транскрипт Скачать .md
Анализ с AI
Описание видео
In this video, we'll be learning how to make our file storage production-ready by moving uploaded images from local disk into AWS S3. We'll walk through creating an S3 bucket, configuring the bucket policy and IAM permissions, and integrating boto3 into our FastAPI application. We'll also refactor our image processing to separate it from the storage layer and handle boto3's blocking calls properly in our async app. By the end of the video, you'll have a production-ready file storage setup that can scale beyond a single server. Let's get started... The code from this video can be found here: https://github.com/CoreyMSchafer/FastAPI-16-AWS-S3-Image-Uploads Full FastAPI Course: https://www.youtube.com/playlist?list=PL-osiE80TeTsak-c-QsVeg0YYG_0TeyXI ✅ Support My Channel Through Patreon: https://www.patreon.com/coreyms ✅ Become a Channel Member: https://www.youtube.com/channel/UCCezIgC97PvUuR4_gbFUs5g/join ✅ One-Time Contribution Through PayPal: https://goo.gl/649HFY ✅ Cryptocurrency Donations: Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3 Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33 Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot ✅ Corey's Public Amazon Wishlist http://a.co/inIyro1 ✅ Equipment I Use and Books I Recommend: https://www.amazon.com/shop/coreyschafer ▶️ You Can Find Me On: My Website - http://coreyms.com/ My Second Channel - https://www.youtube.com/c/coreymschafer Facebook - https://www.facebook.com/CoreyMSchafer Twitter - https://twitter.com/CoreyMSchafer Instagram - https://www.instagram.com/coreymschafer/ #Python #FastAPI

Оглавление (10 сегментов)

Segment 1 (00:00 - 05:00)

Hey there, how's it going everybody? In this video we're going to be learning how to make our file storage production ready by moving our uploaded images from local disk storage into AWS S3. So if you've been following along in this series, then at this point we have a full blog application. So we've got authentication, we've got password resets, we've got a Postgres database with Olympic migrations, and earlier in the series we also added file uploads where users can upload a profile picture, we process it with Pillow, and we save it to a local media directory. And that local storage is totally fine for learning and for local development, but it's not how you typically want to do this in production. So let's talk about the problem with local file storage real quick. So if you're deploying to most modern platforms, especially container blade based platforms, the file system isn't permanent. So if your app restarts where you deploy a new version, those containers can get wiped and rebuilt, and any local files are not guaranteed to stick around. And the other problem is that your files are tied to that one machine. So if you ever need to switch servers or run your app on multiple machines, it's easy to move the code, but now you have to figure out what to do with all the uploaded media. So the usual solution is you store your uploaded files in object storage. So for AWS, that's S3. This is probably the most popular solution out there, and there are also a lot of S3 compatible providers that you can run locally. So object storage is basically a separate service that's built to store files. The files live independently from your application, it's durable, it's scalable, and it's the industry standard way to do this. So in this tutorial we're going to use AWS S3 because it's the most common, it's the one you're going to see everywhere, and once you understand S3, you can apply the same concepts to basically any object storage provider. All right, so before we start changing code, let's take a quick look at how our application is currently handling images so that we know exactly what we need to change. So if we look at our project structure here, we have this media directory here, and inside this media directory we have a profile pics folder, and this is where all of our uploaded images currently live. And if we look at our users router in our routers users. py file here, if I scroll down to the upload profile picture endpoint here, then let's see exactly what we are doing here. So we can see that we are taking a file, we are validating the size. So right here is where we are validating that size, and then we are passing it to this process profile image function. And currently that function does two things. It processes the image using Pillow, and it saves it to the disk. And if we look at image_utils. py, so let me open this up, image_utils. py, then here we can see that process profile image function. So right in here it opens the image, it processes it, and then it saves it directly to that media folder right here with image. save. And our user model has an image path property that returns that location. So our goal is to separate those concerns. So we want process profile image to just process the image and give us back the bytes, and then we'll write a new function to take those bytes and upload them to S3. So the big thing to understand is we're not changing the image processing part, Pillow is still doing the work, we still want it to resize, normalize the JPEG, and other processing. The only thing we're changing is the storage layer. So let's go ahead and get started with this. So first things first, we need to install boto3. So let me pull up my terminal here. So boto3 is the official AWS SDK for Python. It's maintained by Amazon and is the standard library for interacting with AWS services. This is absolutely a skill worth learning since so many applications interact with AWS services these days. Now if you're using pip, then this would just be a pip install of boto3, but for this series I've been using uv, so I'm going to say uv add boto3, and we can see that installs correctly. All right, so now let's set up AWS S3. So you'll need an AWS account for this. There is a free tier, and even beyond that for profile pictures like we're doing in this app, this costs basically nothing. So I'm going to open up my AWS S3 console here. So once you create an account and you have an AWS console, then basically you

Segment 2 (05:00 - 10:00)

can search for anything that you need here. So if you search for S3, that's what I've already done here, and I click on this. I currently don't have any buckets, so it's asking me to create a bucket. If you have buckets already, then you'll see a dashboard, and we'll see what that dashboard looks like in a second. So I'm going to click on create bucket here, and then we're going to have a general configuration here. So I'm just going to say general purpose, global namespace, and here we have the bucket name. Now bucket naming is important. So S3 bucket names have to be globally unique, and they have to be lowercase, and you can't use underscores. So pick a name that you're unlikely to collide with. So let me see if fast-api-blog-uploads is taken, and I think that should be good. Now going back up here really quick, you'll notice the bucket namespace options. They have global namespace, that means that your bucket name has to be unique across all of AWS. Account regional, this is a newer option where the name only has to be unique within your account and region, but it appends your account ID and region to the name, so it ends up being a little bit longer. It looks like they have tagged this as the recommended approach going forward, but I'm going to stick with global for now. But no matter what you choose, the rest of this video should remain the same for either of those options. So now we have to choose a region here. We are going to leave a lot of these options set to the defaults, but we are going to change a couple of them here. Now stuff on AWS changes frequently, so it looks like it already has my region set here. This is US East 1, so I don't actually have to select that manually. You may have to select that. So now let's go down to object ownership. So new buckets, they default to this something like bucket owner enforced and ACLs disabled, and that's exactly what we want. Older tutorials sometimes used the access control list ACLs, but those are legacy. For modern setups, we manage access with bucket policies and IAM policies. So I'm going to keep the defaults here. Now the big thing for this tutorial is the block public access section here. By default, AWS blocks all public access, but anytime you're serving content like images to users, you need public read access, and that's a really common use case. So for our profile pictures, we need to allow that because the browser needs to be able to load these. So I'm going to scroll down here, and first I'm going to uncheck the block all public access there. And now I'm going to keep these first two options here, and we will uncheck this block public access to buckets and objects granted through new public bucket or access point policies, and this block public and cross account access to buckets and objects through any public bucket or access point policies. We will leave those unchecked. I will acknowledge this part here. This is just an acknowledgement that we know what we are doing. Of course we know what we're doing, come on now. Okay, so and with this I should also mention, we're doing this intentionally because we're going to write a bucket policy that only allows reads for a specific prefix and not the entire bucket. All right, so everything else we can leave here as a default. So scrolling all the way down here, I'm just going to go to create bucket, and let's create that, and it looks like that was successful, so that's good. And now we have our dashboard here, and we are inside of our new bucket. Okay, so now we have a bucket, but nothing can access it yet. So we need two things. We need a bucket policy that allows the public to read images from a specific path, and an IAM policy and an IAM user that allows our application to upload and delete images. So let's do the bucket policy first. So I'm going to click into this bucket here. Now if it didn't take you to this page, then you may have be on this page here where it lists all of your buckets. So let's just click on that bucket there. Once you are there, let's click on permissions, okay? And now we want to scroll down to bucket policy here. Now this is going to be written in JSON. Now I have the bucket policy JSON saved in this project as AWS bucket policy JSON, so let me pull this up, AWS bucket policy JSON here. So let me copy this, and we will paste this in and then we'll go over it here. So, I'm going to edit our bucket policy

Segment 3 (10:00 - 15:00)

here. I will paste that in and now let's take a look at this. Now, if you see this 2012 version date and think that looks super outdated, don't worry, that's not when the policy was written. It's the version of the policy language itself. AWS hasn't needed to update the format since then, so every policy that you write today still uses that same version. It's basically just a syntax version number that happens to be a date. But, you can see here in this effect, we have this effect of allow and the action is S3 get object. And the key part here is the resource. So, we're not making the entire bucket public. We're only allowing reads for objects inside of this profile pics prefix. So, the browser can load profile pictures, but you're not accidentally exposing other things that you might put in this bucket later. So, make sure to update the bucket name to match yours. I called mine fast API blog uploads with that profile pics uh prefix there and then we can save that. So, I will save the changes there and we can see that it successfully edit edited the bucket policy. Now, if you get an error at that point, the most common causes are that your bucket name doesn't match what's in the policy exactly or AWS could possibly still be propagating the bucket creation and you might just need to wait 10 or 15 seconds and try again. Okay, so now public reads are allowed for profile pics, but we still need to give our application permissions to upload and delete images. So, now we're going to create an IAM policy. So, here up in search, we can just search for IAM and go here and now let me close down all those pop-ups there. So, I personally pronounce this as IAM. It's for identity and access management. It is IAM. So, we want to create a policy here. So, let's go to policies and now we want to click on create policy here. And now we can use a visual editor or a JSON editor. Again, I'm going to click on the JSON tab here. And now I also have the IAM policy JSON saved in my project. So, let me open this up. So, this is AWS IAM policy. So, let me grab that and we will talk about this as well. So, let me go back here and paste this in and we can see here again that we have that version and we are saying that we have an effect of allow, action is put object and delete object and here we are saying that we have our bucket name, that profile pics prefix and then an asterisk there for everything inside there. So, notice what we're doing here. So, we're following the principle of least privilege. So, we're only granting put object so that we can upload and delete object so we can delete old pictures. We don't even need get object here because our app doesn't need to download the files. It just generates the URL for the browser to fetch them. And again, we are scoping this to our specific bucket and specific prefix. So, if that all looks good, make sure that the bucket name matches yours and not what I have on my screen here. We'll click on next and now I'll give this a name. So, I can just call this fast API blog S3 policy. Now, everything else is optional here, I believe. So, now I'm just going to go to create policy here. Okay, so now we have a policy, but now we need an IAM user for our app. Now, if you're deploying within the AWS ecosystem, then you can use IAM roles instead of access keys and boto3 will pick up credentials automatically, but access keys uh work great for local development or if you're hosting on a VPS or anywhere outside of S3. So, we're going to create a user with access keys. So, I'm going to go to users here and now let's click over here to create a new user and now I need a username. I will just have this user be fast API uh {dash} blog {dash} S3. So, now let's click on next and now we have to set permissions. So, in permissions, where it gives us permission options here, I'm going to click on attach policies directly and I am going to search for the policy that we just created. So, that was fast API and we can see just by typing fast it pulls it up here. So, fast API blog S3 policy. So, now I will

Segment 4 (15:00 - 20:00)

click next and everything here should look good. So, now I'm going to click on create user and user was successfully created. Okay, and now we need access keys. So, I'm going to click on this user here. Let me close down the sidebar there. And now for this user, I'm going to go to the security credentials tab here and now let's scroll down. We want to go down here to access keys. So, down here with access keys, I'm going to click on create access key and AWS is going to ask what this key is for. So, uh we can see that we have CLI, local code, stuff like that. I'm going to click on applications running outside of AWS. So, with that I can click next and now we have an optional description here. I'll just put in fast API blog for the description there and now let's create the access key. Okay, so now we've created that key. Now, this is important. So, AWS will only show you the secret access key one time. So, you need to copy both values right now. Uh the access key ID and the secret ask access key and store them somewhere safe. So, I don't know if there is a way for me to copy both of these at once, but let me go ahead and grab this. I'm going to go to my {dot} env file here. That is where we have been keeping all of our secret information and I am going to uh paste those in here for now and we will clean this up right here in a bit. So, secret access key. Don't worry that I am putting these here on screen for everybody to see. I'm going to clean all of this up uh once I release this video, so these keys will be deactivated. So, I'll format these how they need to be formatted here in just a second, but I will just say that this is the uh put a comment here of access key and then this other one here was the uh secret access key. So, let me put that in and that is good enough for now because what I have here is in my snippets. Let me clean up my console here a little bit. So, here in my snippets, I have this to where we can go ahead and plug all of this in. So, we can see that what I just grabbed from my snippets, I have a setting here for the S3 bucket name, the S3 region, the S3 access key ID, which is this right here. So, I'm going to paste that in and then the S3 secret access key, which is all of this here. So, let me paste that in as well and now I can get rid of those two comments that I had there. I just didn't want to lose those uh because once you close down this page, those will be gone and you can't get them uh again and it we can see here that we get a warning saying that we haven't downloaded these keys. That's fine because we copied them. Now, I say this every time I show credentials on screen, but be sure that our {dot} env file is in our {dot} gitignore file so that we are never committing credentials like this to version control. So, make sure that {dot} env is in your {dot} gitignore. That is a common mistake that even large companies have made and it's a huge security mistake. Uh but don't worry about me showing mine here because I'll be deleting all of this AWS stuff before the video goes public. Okay, so now that we have these credentials in our {dot} env file, now let's update our settings in our config. py so that we can read these values in our application. So, in config. py here, now this is where we have all of our settings for our application. So, I'm going to grab another snippet here and I will paste this in. I'll put this right below, let's see. I'll put it right below uh right above where our max upload size bytes is. Let me get the indentation correctly there. Okay, so now let's go over this. So, we have the S3 bucket name, which is a string, the S3 region. I have set the default to US East 1. Uh we have the S3 access key ID, which is a secret string or none. Uh the S3 secret access key, which is another secret string. And then we have our S3 endpoint URL. Now, this or none here means that some of these are optional. So let me explain why the credentials are optional. So for local development or if you are hosting on a VPS, we set these

Segment 5 (20:00 - 25:00)

in the dot ENV and boto3 uses them directly. But if you're deploying within the AWS ecosystem like on EC2 or ECS, then you can use IAM roles instead. And in that case, boto3's uh default credential chain picks up temporary credentials automatically and you don't need access keys at all. So by making these optional, the exact same code works in both situations. And also notice that we've added this S3 endpoint URL field for real AWS S3, that is going to be none. But there are S3 compatible services that you can run locally. And if you ever want to use one of those, then you can just point boto3 at this local endpoint with this setting. All right, so now that we have the AWS uh S3 stuff set up, now let's move over to our image utilities here. So let me pull those back up in image utils here. Now up to this point, our process profile image function was doing two jobs. So it's processing the image with pillow and saving it to disk. But now we're switching to S3 and I want to separate those concerns. So the image processing function should only process the image and return the processed bytes and the generated file name. And then the storage layer will take those bytes and put them wherever we want, which in this tutorial is going to be S3. So first, let me update the endpoints here, or I'm sorry, the imports, not the endpoints. So we can remove the pathlib import since we're not working with the file system anymore. So I'm going to remove that. And then we need to add a few imports here. And I will just put these at the bottom and my editor will auto sort these. So I'm going to import boto3 and then from starlette. concurrency, I'm going to import run in threadpool. And then we also need our settings here. So I'll say from config, we want to import our settings. So we are adding boto3 for S3. We are importing run in threadpool so that we can offload the blocking boto3 calls and our settings so that we can access our S3 configuration. And we can also remove this profile pics directory constant here since we won't be saving those to disk. So I will go ahead and remove that. And now let's create a helper function to get our S3 client. And again, this is a bit of typing. So I'm going to grab this from my snippets, but then we will explain this. So let me grab this here and now let me paste this in and we will explain what is going on here. Now this just handles passing our credentials if they exist or letting boto3 find them if they don't. And you'll notice uh I have a leading underscore in the name. That's just a Python convention that signals that this is a private helper. Uh it's meant to be used internally in this module, not imported and called directly from other files. But basically all we are doing here is getting access to our S3 client using those credentials. So now let's modify our process profile image function. Now remember, right now it's saving it to disk. We want to change it so that it just returns the image bytes. So right now it is returning a file name here and we can see that it's returning a string. So I'm going to update this first. Instead of just returning the file name string, I'm instead going to return a tuple. So I'll say uh tuple here and this is going to be bytes and a string because first we are going to return the image bytes and then a string of the file name. Now most of the image processing logic stays the same here. So we are still fixing the orientation, we still resize it and things like that. But down here where we are saving to a file path, this is what is going to change. Instead of creating a file path and calling image. save, we instead save to a bytes IO object. So right here, we no longer need this file path. I am going to instead say that we are just going to have output, which is going to be bytes IO. And now we are going to remove this line here where we are making that profile pics directory if it does not exist. And instead, for image. save, I'm going to save it to that output, which are just bytes. And then I'm going to say output. seek and we will set this back to the beginning here. And then for the return, we are going to return output. read and

Segment 6 (25:00 - 30:00)

also the file name. So instead of image. save going to a file path, we're saving this to this output object, which is just an in-memory file. And the seek zero here just rewinds back to the beginning so that we can read from it. And then we return those bytes and the file name. And now this function is now completely decoupled from the file system. And now we can also delete this uh delete profile image function here since this was deleting from local disk. We're going to replace this with a new function that works with S3. So I'm just going to delete that entire function there. And now we're going to have two new functions here to update and delete to S3. And I'm again going to grab these from my snippets here, but then we will explain these. So let me grab these here and let's paste these in to our image utils. Okay, so we have an upload function and a delete function. Both of these use our get S3 client helper. And in the upload function, we get the client and then we run upload file object. And in the delete function, we get the client and we run delete object. So it's pretty simple. So for the upload function, once we get that client, we're saying upload file object, we are just going to pass in some bytes here to a specific bucket name with a key. Now one thing to note here is that we do have this extra args here of content type image JPEG. If we don't set that, browsers can sometimes download the image instead of displaying it, but setting it to image JPEG, it behaves uh the way that you would expect. Now these boto3 calls are blocking, they're not asynchronous. And if you remember from earlier in the series, when we added pillow image processing, we had the same issue. If we do blocking work directly inside of an async endpoint, then we block the event loop and our server can't handle other requests. So the rule of thumb is pretty simple. If something blocks, you offload it to the threadpool using this run in threadpool uh that we imported earlier. That's the usual pattern for using any blocking library in an async FastAPI app. So we need to create async wrappers that use run in threadpool. So again, this is a bit of typing, so I'm going to grab this from my snippets here. All of this code is available in the description section below. So let me grab this and let's paste this in at the bottom and explain what is going on here. So now we have these async functions here, upload profile image and delete profile image. And then we can just pass in a file name and it creates that key for us that will append that profile pics uh prefix there. And then we are running this upload to S3 uh with our file bytes and that key. And we are running that in a threadpool. Same for delete profile image, uh we are just appending that prefix there, running in threadpool of delete from S3. So now anywhere in our application, we can just await upload profile image and it will happen in the threadpool without blocking our server. And also notice that the delete profile image is now async. Before it was synchronous because it was just deleting a local file. Now it's making an S3 API call, so it needs to be awaited. Now another thing to point out here is that our key, this is matching the S3 uh format that we had in our bucket policy. All right, so now that our storage layer is in place, we need to update our user model. So previously, our image path property returned a local path like, you know, {forward slash} media {forward slash} profile pictures {forward slash} file name. jpeg. Now we want it to return a full S3 URL. So to show you what I mean, let's open up models. py here and we will look at this image path here. Actually first, let's import what we need to import up here. Uh we are going to need to import our settings. So I'll say from config, we want to import our settings. And now down here in our user model and this image path property, we just need to change the return value. Right now it returns the local path. So we can see here if there is an image file, then we return this uh file that lived on local storage. But, now we want it to return the full S3 URL instead. Now, this URL

Segment 7 (30:00 - 35:00)

is a little long here, but let me grab this from my snippets and I'll explain what these S3 image files or paths look like. So, let me go back to models here and let me paste this in and my snippet had the return in there, so I can just delete that. Okay, so this is how a URL to an image on S3 is formatted. So, we have the bucket name {dot} S3, then we have the region, then {dot} amazonaws. com, and then our specific route that we set up on S3. So, our specific route, we set this up to have a prefix of profile pics and then the image name. Now, the default image here, I'm going to keep on local storage. So, this image path stays the same because the default image is part of our application. It is not user uploaded content. So, we can just keep that single image on our local file system. It's not going to hurt anything. Okay, so now we need to update our routes. So, the upload profile picture endpoint, the delete profile picture endpoint, and also the delete user endpoint because if you delete a user, then we should also clean up their profile pictures in S3. So, first, let's open up our users endpoint here. So, that is the users router and let's look at this. So, first in our imports here, I uh we need a new import for S3 error handling and I'll just add this to the bottom of my imports. So, this is from boto core {dot} exceptions. We want to import client error and I will save that. Again, this auto sorts my import, so we can see that it put it up here. And then we need to update our image utils import to include our new upload function. So, if I find our image utils here, we can see that we already have this delete profile image. Uh that one was from the local file system before, but now that's that asynchronous one. So, we don't need to import that again, but I do need to import this upload profile image here. All right, and now let's update these endpoints. So, now let's find the delete user endpoint first. So, I will just search for delete user here and find this endpoint here. Okay. Now, inside of here, we are already storing the old file name before deletion and we already clean up the file after commit. The only change here is that now this delete profile image is now async, so we need to await this. So, I'll just say await delete profile image. And everything else in here, we're not really going to cover because we covered that in what the video where we showed how to do file uploads. So, all of that stays the same. Uh but, for this, we just need to await that delete profile image. Now, we also need to update this upload profile picture endpoint here. So, it's just below the delete one. So, within here. Now, the beginning of this endpoint is the same as before. So, we check authorization, we read the file bytes, we validate the size, but here is where the logic changes. And I need to scroll up here a little bit because this is where the logic changes here we are running uh process profile image here. So, previously, process profile image returned just a file name because it saved to disk. Now, it's a uh it returns a tuple of the processed bytes and the file name. So, I need to add uh processed bytes here as one of those return values and also the new file name. Now, right after this try except block, we need to upload those processed bytes to S3. So, I'm going to grab this as a snippet here from my snippets and I'm not going to overwrite anything. I'm just going to put this after the try except block and before this old file name here. So, let me paste this in. Make sure that the indentation here is correct. Okay. And actually, I had a comment there as well. Let me also grab the comment just so uh that is clear what we are doing here. So, we are uploading to S3 using our upload profile image function that we created earlier. And if something goes wrong with the upload, we catch the client error from boto3 and we return a 500 error here. Now, the rest of this is mostly the same. Uh we store the old file name, we update the database, and

Segment 8 (35:00 - 40:00)

then delete the old file after commit. The only thing down here is that this is now async again. So, we will await that delete profile image for their old profile. So, I mentioned this in the images tutorial itself, but the order of operations here is important. So, we process the image, we upload the image to S3, we then update the database, and then we delete the old image from S3. So, last because if the database update fails, then the worst case that you have is you that you have an orphan file in S3 that you can clean up later, but you didn't delete the old image and leave the user with nothing. So, another endpoint that we need to update here is delete user picture. This is mostly the same here. The only difference is down here where we are uh running this delete profile image, we need to await that. Okay. Okay, and with those changes, at this point, our API is now saving files to S3 and cleaning up from S3, but we still have one more thing to do. So, in main. py, we previously mounted the media directory so that FastAPI could serve those local files, but we're not using local media anymore, so we should remove that mount. So, I opened up main. py here and down here, we can see that we mounted that media directory. I can just remove that. So, let's remove that. We want to keep the the static directory, but we will remove that media one. All right. Now, before we test the full application, I want to show you a quick way to verify that your S3 credentials and permissions are correct because if you're following along and something doesn't work, it's really helpful to have a small script that can confirm whether we can upload and delete. So, in this tutorial, I specifically built this check_s3. py file here. So, let me open this up and again, this will be available for download in the description section below, but all this script does is it uploads a very small test file here. So, we can see that we have a test upload. It's just creating a file called test. txt and then it is just a few bytes here called uh with the value of test. And then we are doing a test delete. So, we're just quickly uploading a file, deleting a file, just making sure that our S3 connection is working. Now, other than that quick walk through, I'm not really going to go over this file in depth uh since it's just meant to test our S3 connection and isn't directly related to our app, but it's there if you want it for testing. So, to run this file, uh I have set this up. I'm going to kill the server here really quick. And within the terminal here, I can run this with UV run check s3. py and let's run this and hopefully it works. Okay, so we can see that it says that our bucket name is uh fast blog uploads. It prints out our region. It says that the upload was a success, the delete was a success, all tests passed, the configuration is working. So, if you're having any issues with S3 connections, then you can uh change whatever you need to do on the AWS side and make sure that this script runs and this very simple script will give you an idea of where your problem lies because if this doesn't work, then everything else in your application isn't going to work either. But, if this works, then you should be good to go with your application working as well. But, it just helps you narrow down some problems. So, what some of those problems could be. So, if the upload fails, then usually your IAM policy is wrong, your bucket name is wrong, or your credentials are wrong. But, if the upload works, but the delete fails, then usually the delete object permission is missing from the IAM policy. And if this works, but your images later load as 403 forbidden in the browser, then usually your bucket policy isn't set correctly or you didn't uncheck those two block public access bucket policy settings when creating the bucket. But, hopefully with those notes, and with this file here, this is working correctly for everyone, and we all get this up and running smoothly. So, now let's test the full application. So, I'm going to run the development server here. So, I'll say UV run fast API dev main. py

Segment 9 (40:00 - 45:00)

and now that is running. So, let's go back to the browser here, and I am going to reload our page. And now you're going to notice that any existing profile pictures are broken. That's expected because those files are still sitting on local disk, but our app is now pointing to S3. Once we upload a new image, it'll go to S3 and load correctly. So, let me go to the account page where we upload a profile picture. So, first I have to log in here, and I will just use one of the test users with test password one. Go ahead and log in, and now let's go to where we can upload a profile picture here. And now let me choose a file, and here I will just choose one of the files from my desktop here. This is within fast API blog populate images, and let's do Corey. png here. So, I will upload that. It says that was uploaded successfully. We can see that profile picture there. And now here's how we can verify that this is coming from S3. If we right click the image, and then go to open image in new tab, then we can see the URL up here is fast API blog uploads. s3. us-east, and so that is coming from an S3 URL. So, now let's go back to AWS here, and let me go to S3, and let's check our bucket directly. So, within here, I'm going to go to our fast API blog uploads bucket here, and we can see that now we have this profile pics. I'm going to look at the object that we have in here. We have our hashed JPEG here. And if I click on this, then I could look at this directly in the browser, but we've already done that. So, I'm just going to look at our object here. So, now let me test uploading a different profile picture, and test that this old one is deleted. So, if I go back to my account here, and I go to choose file, let me pick Let's see, I'll pick a picture of my dog Bronx here, and I will upload this. Profile picture uploaded successfully. Okay, now let's go back to S3, and let's reload this page. And now we can see that we only have one object in here, which is the new profile picture, and this old one was deleted. And now, lastly, let me also check here. Actually, I was going to test one more thing, but I think that is good enough for now. Actually, let's go ahead and test it. What I was going to test is deleting the profile picture. It doesn't look like I've actually put anything here on the front end to delete a profile picture. Maybe that's something I could add in later, but I know that we have a route for that in our API. So, let me go to the docs route here. So, let me log in here. So, I will log in with one of my test accounts, and for the password, it's just test password one. Authorize there. And now we have a delete user route here. Let me check really quick what my ID is. So, let me execute this. So, I have an ID for this user as one. So, for delete image for this user, I'm going to say ID of one. Let's execute that. We got a 200 response. So, now if I reload the page here, then we should see that I now have a default image here. So, this is the default that does live on our local file system. Okay, so all of that is working well. So, let's go back to the code here really quick. Now, you might be wondering about the images here that we already had on our local file system in our media folder. So, if you had existing data that you wanted on S3, then you could write a script to migrate those to S3, but that's more in-depth than what we're covering here. But, what I have done is I have updated the populate DB script to work with S3. So, if you are using the code available for download in the description, then we can run that script, and it'll recreate all the users and posts with their profile pictures uploaded to S3. So, let me show how that works. I will stop the server here really quick. And now let me run that populate db. py script that I've

Segment 10 (45:00 - 47:00)

been using to populate our database throughout the series. Let me run that, and we will wait for that to finish. And now, let me rerun the server again. And now if I look at our home page here, then we can see that now all of these profile pictures are working, and I've updated that script. So, if I go to up open image in new tab here, all of these are now coming from S3. All right, so before we wrap up here, let's do a quick recap of what we changed in this tutorial. So, we set up an S3 bucket with the right permissions. We added boto3 to our project, and we wired up the S3 configuration in our settings. We refactored our image processing to separate it from the storage layer, and we wrote upload and delete functions that run in the thread pool, so that they don't block our async app. We updated our routes and models to work with S3, and we removed the old local media mount since everything is served from S3 now. So, at this point, our stack is starting to look like a real production stack. We have Postgres for the database. We have S3 for file storage, async endpoints for database work, and thread pool offloading for blocking operations. Now, in the next video, we're going to talk about testing. We're going to write tests for our fast API application, and look at the patterns for testing routes, authentication, database interactions, things like that. But, I think that's going to do it for this video. Hopefully now you have a good idea how to make your file storage production ready by using object storage like Amazon AWS S3, and how to integrate boto3 into an existing fast API app. In the next tutorial, we'll learn about testing our fast API application. But, if anyone has any questions about what we covered in this video, then feel free to ask in the comment section below, and I'll do my best to answer those. And if you enjoyed these tutorials, and would like to support them, then there are several ways you can do that. The easiest way is simply like the video, and give it a thumbs up. Also, it's a huge help to share these videos with anyone who you think would find them useful. And if you have the means, you can contribute through Patreon or YouTube, and there are links to those pages in the description section below. Be sure to subscribe for future videos, and thank you all for watching.

Другие видео автора — Corey Schafer

Ctrl+V

Экстракт Знаний в Telegram

Экстракты и дистилляты из лучших YouTube-каналов — сразу после публикации.

Подписаться

Дайджест Экстрактов

Лучшие методички за неделю — каждый понедельник