AWS API Gateway and S3 Integration (encouraging the correct way)

Amazon S3

Amazon Simple Storage Service (Amazon S3), often referred S3, is a highly scalable, secure, and flexible cloud storage service provided by Amazon Web Services (AWS). It was introduced in March 2006 and has since become one of the most widely used object storage services in the world.

S3 is designed to store and retrieve data of any type or size, making it a fundamental building block for many cloud-based applications and services.

API Gateway

Amazon API Gateway is a fully managed service provided by Amazon Web Services (AWS) that allows you to create, publish, secure, and manage APIs for your applications. It serves a central entry point for your APIs, enabling you to connect to backend services or AWS resources while handling essential tasks such as request routing, authentication, authorization, monitoring, and more.

Requirement

There may be a requirement, where you want to upload images to S3 using APIs and not any SDK or AWS CLI, rather directly from APIs!

Common Mistakes

Image source: https://www.luminis.eu/blog/cloud-en/creating-a-simple-api-stub-with-api-gateway-and-s3/

I’ve seen people using AWS Lambda as the intermediate to receive files via API Gateway and then upload to S3, which according to me isn’t the optimal solution, even if the files are required to be processed before saving, it takes twice the effort to upload the file. (API gateway -> Lambda -> S3), which may lead to unwanted corrupted files in the destination.

Possible Solution

Instead the files can be directly uploaded to the S3 via API Gateway and then, Lambda can perform processing over it and may store the result as and where/when required.

The following steps needs to be performed to achieve the above:

1. Create an S3 bucket to store the files (with public access if required)
2. Create an IAM role for API Gateway with permissions to PutObject in S3.
3. Create an API Gateway with the above created role attached.
4. Create an API with dynamic path parameters to accept the following:

• Bucket Name
• Folder Path (Optional)
• Filename

5. Modify API Gateway settings to accept binary objects in payload.

6. Deploying API Gateway via a stage

7. Create an usage plan, so as to attach API key to avoid unauthorized access to the resources.

Getting started

1. Creating an S3 bucket.

1.1(Optional) Unchecking Block all public access (will allow public read access)

1.2 (Optional) Creating S3 Bucket Policy for the above created bucket to allow end users to access it.

{
  "Id": "Policy1696244459937",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1696244457380",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::api-gateway-s3-tutorial/*",
      "Principal": "*"
    }
  ]
}

2. Create an IAM role with permissions policy to PutObject to the above created S3 bucket

2.1 Creating IAM role

2.2 Creating permission policy to attach to the above created role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::api-gateway-s3-tutorial/*"
        }
    ]
}

Summary of IAM role

3. Creating API Gateway.

3.1 Creating API Gateway as REST API:

Summary for API Gateway Creation

4. Creating API Gateway Resources and Method to accept required parameters to upload object to S3.

4.1 Creating resource {bucket} to accept bucket name in path.

4.2 Creating sub resource {filename} for the above resource to accept the file name from path parameter.

4.3 Creating a PUT Method to the above created resource to upload files and attaching the API Gateway Role created earlier.

4.3 Under Method Request for the above created method, check the API key required field.

4.4 Under Integration Request for the above created method, edit the path parameters as the following to accept bucket name and file name from the path parameters.

5. Modifying API Gateway settings to accept any desired type of file content that we want to upload. (.jpg and .png in our case).

6. Deploying the API with the above made configurations in a new stage.

7. Creating usage plan and API key to add the above created stage.

7.1 Creating API key

7.2 Creating usage plan and associating the previously created stage for API Gateway.

7.3 Associating the API key

Now, the integration process is complete with API key authorization! To test the setup we shall use Postman.

NOTE: Just to be sure, re-deploy the API gateway after making all the changes.

Testing the setup

1. Copy the URI from the deployed stage for API gateway and add the path for the bucket name (along side folder path, if needed) and name of the file, attach the file to be uploaded to the bucket as binary type in the payload, as shown below:

1.1 (Optional) To provide a destination folder path in s3 for a file, you need to use “%2f” instead of “/”. For example if your path is:

api-gateway-s3-tutorial/folder1/wew.png, where

api-gateway-s3-tutorial is your bucket name,

folder1 is your folder and wew.png is your filename,

then the URI to upload shall be:

https://t8mhsswxx6.execute-api.ap-south-1.amazonaws.com/dev/api-gateway-s3-tutorial%2ffolder1/wew.png

2. Add the API key in the header, as x-api-key as the key:

3. As soon as we click on Send, we receive a 200 OK response with empty response body, implies the request was successful. And when we check for the same in the S3 console. We can see that the file has been uploaded!! (to the path if provided or else in the root path of bucket).

Hence marks the completion of this tutorial, hope you find this small tutorial insightful and would have solved your use case.

#aws #apigateway #s3 #cloud #lambda #automation #righteducation #devops #knowledge #information #community #contribution