Have you considered doing it from an Amazon EC2 instance instead? Why are you wanting to do this in a Lambda function? I have several objects decompressed in S3, I want to compress them all into a single file and make it available in another location in the same bucket and then send the download link to my api and then to the client, and in this way later download it.
This functionality can be used frequently, possibly every day there are people who download assets in my app. I currently have the memory set to the maximum in the configuration I have little experience with aws and would have to investigate how it would be worth trying from EC2.
The idea of doing it with a lambda function arises by freeing the api of this load, and initially the behavior was very good, but after many large files it began to fail. Add a comment. Active Oldest Votes. Improve this answer. John Rotenstein John Rotenstein k 17 17 gold badges silver badges bronze badges. Thanks for your support friend, I am following your recommendations.!
Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. It is entirely pay as you go and you only pay for what you need, implicating the ability to store massive amounts of data for cheap. Regarding zip files, there is no need to upload files to S3 as such because of its cost effective storage. In other words, S3 stores static assets in a very cost effective way. Users often use it as their primary way of performing operations against their Amazon S3 buckets and objects.
This can be accomplished with a variety of commands — the cp and sync commands. The cp command is very simple to understand. It is basically a way for a user to copy their contents in one directory from another. It is flexible, and can be performed between two Amazon S3 buckets, or between a local directory and an Amazon S3 Bucket. Since it performs operations between two directories, it can be implemented when wanting to upload or download contents.
Also important to note, since it only copies one file or object at a time, users will have to add in the —recursive command to make it transfer all assets under the specified prefix. This extra step signifies just how flexible the cp command is.
The main benefit that users can expect to receive from the AWS sync command as opposed to the cp command, is that by default, the sync command will effectively sync or download multiple files between two specified directories. AWS notates that users only have the ability to download one object at a time, and not multiple at once. Check the list of CLI commands that can be used to accomplish downloading multiple assets.
Depending on the method of encryption for your amazon s3 objects, users have the ability to download and decrypt them. Furthermore, users that are attempting to download multiple objects that are encrypted, it is important that their accounts have the appropriate permissions necessary to decrypt the objects [5].
Notice that this operation in particular is using the get-object command and not the s3 sync or cp command. AWS S3 is a fully redundant, resilient, and highly available storage service that has a pay as you go pricing model.
I had a program last year that dynamically generated lots of images and needed a way to share those images with third parties later. After looking around for a while, it looked like most web developers faced this same problem and somehow fixed it but the only available information online was not satisfactory to me.
The old way of sharing files in Amazon S3 is to download files from S3 storage, zip them locally, and upload them back to S3. While this method works, it is definitely not good if you have less storage and need to create a large zip file.
It is also very resource intensive and time consuming. It also requires cleanup at the end and the entire process consumes almost twice the storage needed for the zipped file.
Another possible option is to compress the files before putting them in S3 storage. This method works well for static files but does not address the case where clients need an assortment of files. It also does not cover dynamically generated files. All those solutions are good, but they all fail to address the real world issue where files are generated dynamically. There is no way to know upfront what clients require or which files need sharing.
0コメント