Managing Private Files With AWS S3

Overview

Back in 2020, at the company I work for, I was told that we were moving our images from Cloudinary to AWS S3 and that I was going to be in charge of performing the migration, long story short, I did the migration, but in the process I was faced with a problem, we have images that must be private and are only meant to be seen by admins. Two years since then, I’m faced with a similar problem, but with a different use case.

In this article, I’ll show you two ways you can manage private files with AWS S3, the one I did in 2020, and the one I’m planning to do now.

Method 1: Get file through a backend endpoint

This solution is what I used when I did the migration of images, it basically consists in creating a backend endpoint which will validate if the user requesting the image has the necessary permissions to see the file.

My Use case

Give access to admins, so they can see private images that are rarely to be requested (3 times as much).

A disadvantage of this approach is that the private file first goes from S3 to our backend and from there it goes to the client, which increase transfer costs, but it was ok for our use case because these images are barely requested once or twice and then forget it, because they are no longer needed unless we need to validate historical data.

Solution 2: Pre-sign URLs

This solution consists in using what is call Pre-sign URLs. The idea behind these URLs is to give access to private files for a period of time, and we do it by signing the object key of our S3 with private credentials in the backend, setting an expiration time in which the pre-sign URL will be valid to be requested.

As you see, for this method, we also make use of a backend endpoint to validate if the user has permissions to see the private file, so we can return the sign URL.

My Use case

Render a list of employees which will have a name and picture (to keep the example simple) that will be frequently visited in the application.

With this, we solve the problem of transfer costs of the first method, here instead of transferring the file from S3 to our backend and then to the client, we just send a signed URL to the client that can be used to obtain the file directly from the S3.

A problem that we’ll face in here is that every time we sign a URL, a new signature is generated, so the browser won’t be able to cache the image, because the URL will always change, so we’ll have something like this every time: In every request, the user will download the file Instead of getting it from cache:

This problem is explained in detail in this article, but basically we’ll need a way to get the same signature for the pre-signed URL, so the browser can cache the file.

The other thing is that I need to render a list of employees in the frontend where everyone has a private image, but having an endpoint for signing just one URL at the time will be an N+1 query problem to our server, so when we return the list of employees we’ll also have to return the pre-signed URLs to avoid this problem.

Securing the S3 Bucket

Something we’ll have to do sooner or later is configuring our S3 to give access to our backend when using the AWS SDK, so we can upload or get files from our S3 bucket.

AWS offer us different methods to give access to our S3 bucket, but because we are working within the same AWS account, we’ll use IAM policies to provide access to our bucket.

IAM Policies

These policies control the access to resources on AWS (S3 in our case) and are attached to entities like users, roles or groups in your AWS account to define what can these entities do in the different services provided by AWS.

Now we’ll proceed with the configuration:

Here I assume you already have an AWS account

Create the bucket to store our objects

If you’ll have public files too, you must deactivate the Block Public Access option, as I did in here.

Create a policy to define the permissions of our user

For my example, I just need 3 permissions:

1. PutObject, to upload files
2. PutObjectAcl, to define if the file will be public or private
3. GetObject, to get files

Also, I specified that this policy is only valid for the bucket we just created with the field: "Resource": "arn:aws:s3:::hrmtest/*"

Create a user to obtain the credentials needed for our backend

Here, we create the user with the policy we just created in the previous step, so we can get the credentials to use them in our application.

After this, we are ready to use our credentials in our application.

Code time!

I prepare a small project where we’ll be able to upload and get files from AWS S3. We’ll use the credentials we downloaded from AWS and will put them inside the .env file that will be read when starting the service, so we can do requests to AWS:

# AWS S3
S3_ACCESS_KEY_ID=AKIAZCF3JGYORXJMZPMY
S3_SECRET_ACCESS_KEY=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
S3_BUCKET_NAME=hrm-article-demo
S3_BUCKET_REGION=us-east-1

After we read the credentials, we’ll use them to get a session to AWS S3:

Upload method

With this, we’ll be able to upload public or private files.

GetFile method

To get the private files, we’ll use this method. For public files AWS give us a URL with the following format: [https://{bucket-name}.s3.amazonaws.com/{object-key.png}](https://bucket-name.s3.amazonaws.com/{object-key.png}) → https://hrm-article-demo.s3.amazonaws.com/employees/1399d998-b505-4ce2-b49b-bd8073de9b9e.jpg

Presign method With this, we’ll be able to sign our private file’s key, so we can see the file for a period of time.

Another way of doing this without getting the same signature every time is:

Here, I just show the code that’s of interest to the article. If you want to see all the project’s code, just go to:

Also, I mentioned that at the backend, we need to verify if the user has the needed permissions to see the requested file, but I didn’t include that code in this article because I’ll make an article on how to handle authentication and authorization in your application.

Play time!

Now that we have the code, we can play with it.

Uploading an image

We’ll upload a private image, so the only way to see it will be through our endpoint to get files or to sign the URL.

Once we send the request, we’ll get a response like this:

As I just told you, the only way to see the image is through our endpoint to get files or signing the URL, so if we try to use the URL that AWS give us it will show us something like this:

But if we use the endpoint we made to get files from AWS, we’ll get the file:

Yeah, that’s me, now let’s try signing the URL:

Now, if we use the pre-signed URL, we’ll also get the file while it is still valid:

GetEmployees method

And for my use case, I told you that I need to generate a signed URL for the employee's pictures, but that calling an endpoint to sign a URL one by one will be an N+1 query problem, so for that, I’ll use a method like this, that when requesting the employees, I can get the pre-signed URL also:

So If we upload some images for our employees, when we do a request to: GET api/v1/employees, we’ll get a signed URL with the same signature during the current day every time we execute a request, and as we can see, the browser cached the image.

Conclusion

Okay, this got longer than I was expecting, but I showed you how to configure your S3 and two methods to get private files from AWS S3 with use cases that I’ve come across at work. So I hope that you can decide what method to use depending on your case.

References

1. For the illustrations, Excalidraw
2. Learn in depth how Pre-sign URLs work
3. If you speak Spanish, here is a good course about AWS S3 https://ed.team/cursos/s3
4. Securing AWS S3 uploads using presigned URLs