Python Upload Images to S3 From Facebook From Url

In web and mobile applications, it's common to provide users with the ability to upload data. Your awarding may let users to upload PDFs and documents, or media such as photos or videos. Every modern web server technology has mechanisms to let this functionality. Typically, in the server-based environment, the process follows this period:

1. The user uploads the file to the application server.
2. The awarding server saves the upload to a temporary space for processing.
3. The application transfers the file to a database, file server, or object store for persistent storage.

While the process is simple, information technology tin can have significant side-effects on the performance of the web-server in busier applications. Media uploads are typically large, and then transferring these can represent a large share of network I/O and server CPU time. You lot must too manage the state of the transfer to ensure that the entire object is successfully uploaded, and manage retries and errors.

This is challenging for applications with spiky traffic patterns. For case, in a web application that specializes in sending holiday greetings, it may experience virtually traffic only effectually holidays. If thousands of users attempt to upload media effectually the same fourth dimension, this requires you to scale out the application server and ensure that at that place is sufficient network bandwidth available.

Past directly uploading these files to Amazon S3, you tin can avert proxying these requests through your application server. This can significantly reduce network traffic and server CPU usage, and enable your application server to handle other requests during decorated periods. S3 too is highly available and durable, making it an platonic persistent shop for user uploads.

In this weblog postal service, I walk through how to implement serverless uploads and show the benefits of this arroyo. This pattern is used in the Happy Path web awarding. You tin download the code from this blog postal service in this GitHub repo.

Overview of serverless uploading to S3

When you upload straight to an S3 bucket, you must commencement request a signed URL from the Amazon S3 service. You lot can and then upload directly using the signed URL. This is two-step process for your application front end:

1. Call an Amazon API Gateway endpoint, which invokes the getSignedURL Lambda function. This gets a signed URL from the S3 bucket.
2. Directly upload the file from the awarding to the S3 bucket.

To deploy the S3 uploader example in your AWS account:

1. Navigate to the S3 uploader repo and install the prerequisites listed in the README.doctor.
2. In a concluding window, run:
   git clone https://github.com/aws-samples/amazon-s3-presigned-urls-aws-sam
cd amazon-s3-presigned-urls-aws-sam
sam deploy --guided

3. At the prompts, enter s3uploader for Stack Name and select your preferred Region. Once the deployment is complete, note the APIendpoint output.The API endpoint value is the base URL. The upload URL is the API endpoint with /uploads appended. For example: https://ab123345677.execute-api.u.s.a.-west-2.amazonaws.com/uploads.

Testing the application

I show 2 ways to examination this application. The get-go is with Postman, which allows y'all to directly call the API and upload a binary file with the signed URL. The 2nd is with a bones frontend awarding that demonstrates how to integrate the API.

To exam using Postman:

1. Offset, copy the API endpoint from the output of the deployment.
2. In the Postman interface, paste the API endpoint into the box labeled Enter asking URL.
3. Choose Send.
4. Subsequently the request is complete, the Body section shows a JSON response. The uploadURL aspect contains the signed URL. Copy this aspect to the clipboard.
5. Select the + icon next to the tabs to create a new request.
6. Using the dropdown, change the method from GET to PUT. Paste the URL into the Enter request URL box.
7. Cull the Body tab, so the binary radio push.
8. Choose Select file and choose a JPG file to upload.
   Choose Send. You see a 200 OK response later on the file is uploaded.
9. Navigate to the S3 console, and open up the S3 bucket created by the deployment. In the bucket, y'all meet the JPG file uploaded via Postman.

To test with the sample frontend application:

1. Copy alphabetize.html from the instance's repo to an S3 bucket.
2. Update the object's permissions to get in publicly readable.
3. In a browser, navigate to the public URL of index.html file.
4. Select Choose file and and then select a JPG file to upload in the file picker. Choose Upload prototype. When the upload completes, a confirmation message is displayed.
5. Navigate to the S3 console, and open the S3 bucket created past the deployment. In the bucket, you see the second JPG file y'all uploaded from the browser.

Agreement the S3 uploading process

When uploading objects to S3 from a web application, you must configure S3 for Cross-Origin Resource Sharing (CORS). CORS rules are defined as an XML document on the bucket. Using AWS SAM, you lot can configure CORS equally part of the resource definition in the AWS SAM template:

                      S3UploadBucket:     Type: AWS::S3::Bucket     Properties:       CorsConfiguration:         CorsRules:         - AllowedHeaders:             - "*"           AllowedMethods:             - GET             - PUT             - Head           AllowedOrigins:             - "*"                  

The preceding policy allows all headers and origins – it's recommended that you apply a more restrictive policy for production workloads.

In the beginning step of the process, the API endpoint invokes the Lambda function to make the signed URL asking. The Lambda function contains the following code:

          const AWS = require('aws-sdk') AWS.config.update({ region: process.env.AWS_REGION }) const s3 = new AWS.S3() const URL_EXPIRATION_SECONDS = 300  // Main Lambda entry betoken exports.handler = async (outcome) => {   return wait getUploadURL(event) }  const getUploadURL = async function(effect) {   const randomID = parseInt(Math.random() * 10000000)   const Key = `${randomID}.jpg`    // Get signed URL from S3   const s3Params = {     Bucket: process.env.UploadBucket,     Cardinal,     Expires: URL_EXPIRATION_SECONDS,     ContentType: 'image/jpeg'   }   const uploadURL = await s3.getSignedUrlPromise('putObject', s3Params)   return JSON.stringify({     uploadURL: uploadURL,     Key   }) }                  

This office determines the name, or primal, of the uploaded object, using a random number. The s3Params object defines the accepted content blazon and besides specifies the expiration of the key. In this instance, the key is valid for 300 seconds. The signed URL is returned as part of a JSON object including the central for the calling application.

The signed URL contains a security token with permissions to upload this single object to this bucket. To successfully generate this token, the code calling getSignedUrlPromise must have s3:putObject permissions for the bucket. This Lambda office is granted the S3WritePolicy policy to the bucket by the AWS SAM template.

The uploaded object must match the same file name and content type equally defined in the parameters. An object matching the parameters may exist uploaded multiple times, providing that the upload process starts before the token expires. The default expiration is fifteen minutes but you may want to specify shorter expirations depending upon your utilise case.

In one case the frontend application receives the API endpoint response, information technology has the signed URL. The frontend awarding then uses the PUT method to upload binary data directly to the signed URL:

          allow blobData = new Blob([new Uint8Array(array)], {type: 'image/jpeg'}) const issue = await fetch(signedURL, {   method: 'PUT',   trunk: blobData })                  

At this point, the caller application is interacting directly with the S3 service and not with your API endpoint or Lambda part. S3 returns a 200 HTML status code once the upload is consummate.

For applications expecting a big number of user uploads, this provides a uncomplicated way to offload a large amount of network traffic to S3, away from your backend infrastructure.

Adding authentication to the upload process

The current API endpoint is open, available to any service on the internet. This means that anyone can upload a JPG file once they receive the signed URL. In virtually product systems, developers want to apply authentication to control who has access to the API, and who tin can upload files to your S3 buckets.

You lot can restrict access to this API by using an authorizer. This sample uses HTTP APIs, which support JWT authorizers. This allows y'all to command access to the API via an identity provider, which could be a service such equally Amazon Cognito or Auth0.

The Happy Path awarding only allows signed-in users to upload files, using Auth0 every bit the identity provider. The sample repo contains a 2d AWS SAM template, templateWithAuth.yaml, which shows how you can add an authorizer to the API:

                      MyApi:     Type: AWS::Serverless::HttpApi     Backdrop:       Auth:         Authorizers:           MyAuthorizer:             JwtConfiguration:               issuer: !Ref Auth0issuer               audience:                 - https://auth0-jwt-authorizer             IdentitySource: "$request.header.Authorization"         DefaultAuthorizer: MyAuthorizer                  

Both the issuer and audience attributes are provided by the Auth0 configuration. By specifying this authorizer as the default authorizer, information technology is used automatically for all routes using this API. Read office 1 of the Ask Around Me serial to learn more about configuring Auth0 and authorizers with HTTP APIs.

Afterwards authentication is added, the calling web awarding provides a JWT token in the headers of the request:

          const response = await axios.get(API_ENDPOINT_URL, {   headers: {     Authorisation: `Bearer ${token}`         } })                  

API Gateway evaluates this token earlier invoking the getUploadURL Lambda function. This ensures that only authenticated users tin upload objects to the S3 bucket.

Modifying ACLs and creating publicly readable objects

In the current implementation, the uploaded object is not publicly accessible. To make an uploaded object publicly readable, you lot must fix its admission command listing (ACL). In that location are preconfigured ACLs bachelor in S3, including a public-read option, which makes an object readable past anyone on the internet. Set the appropriate ACL in the params object earlier calling s3.getSignedUrl:

          const s3Params = {   Bucket: process.env.UploadBucket,   Key,   Expires: URL_EXPIRATION_SECONDS,   ContentType: 'image/jpeg',   ACL: 'public-read' }                  

Since the Lambda role must accept the appropriate bucket permissions to sign the request, yous must also ensure that the function has PutObjectAcl permission. In AWS SAM, you can add the permission to the Lambda part with this policy:

                      - Statement:           - Issue: Allow             Resource: !Sub 'arn:aws:s3:::${S3UploadBucket}/'             Action:               - s3:putObjectAcl                  

Conclusion

Many web and mobile applications allow users to upload data, including large media files like images and videos. In a traditional server-based awarding, this can create heavy load on the awarding server, and likewise use a considerable corporeality of network bandwidth.

By enabling users to upload files to Amazon S3, this serverless blueprint moves the network load away from your service. This tin make your application much more scalable, and capable of handling spiky traffic.

This blog mail service walks through a sample application repo and explains the process for retrieving a signed URL from S3. It explains how to the test the URLs in both Postman and in a web application. Finally, I explain how to add authentication and make uploaded objects publicly accessible.

To larn more, see this video walkthrough that shows how to upload directly to S3 from a frontend spider web application. For more serverless learning resource, visit https://serverlessland.com.

hackthrooderfe.blogspot.com

Source: https://aws.amazon.com/blogs/compute/uploading-to-amazon-s3-directly-from-a-web-or-mobile-application/

Share this post