{"_id":"s3-node","_rev":"3212872","name":"s3-node","description":"Made compatible for node 12 and later as this was in open issues for s3 library","dist-tags":{"latest":"0.0.1"},"maintainers":[{"name":"ncnipunchawla","email":""}],"time":{"modified":"2024-02-28T06:15:29.000Z","created":"2021-01-14T13:51:01.607Z","0.0.1":"2021-01-14T13:51:01.607Z"},"users":{},"author":{"name":"Nipun Chawla"},"repository":{"type":"git","url":"git+https://github.com/ncnipunchawla/node-s3-client.git"},"versions":{"0.0.1":{"name":"s3-node","version":"0.0.1","description":"Made compatible for node 12 and later as this was in open issues for s3 library","main":"lib/index.js","scripts":{"test":"mocha"},"repository":{"type":"git","url":"git+https://github.com/ncnipunchawla/node-s3-client.git"},"keywords":["amazon","s3","sync","folder","directory","retry","limit","stream","async","parallel","multipart","size"],"author":{"name":"Nipun Chawla"},"license":"MIT","engines":{"node":">=8.0.0"},"devDependencies":{"mocha":"~2.0.1","ncp":"~1.0.1"},"dependencies":{"aws-sdk":"~2.4.9","fd-slicer":"~1.0.0","findit2":"~2.2.3","graceful-fs":"~4.1.4","mime":"~1.2.11","mkdirp":"~0.5.0","pend":"~1.2.0","rimraf":"~2.2.8","streamsink":"~1.2.0"},"bugs":{"url":"https://github.com/ncnipunchawla/node-s3-client.git/issues"},"directories":{"test":"test"},"gitHead":"dc968c300a65b381efbe69541e6e8c976a8db060","homepage":"https://github.com/ncnipunchawla/node-s3-client#readme","_id":"s3-node@0.0.1","_nodeVersion":"14.15.4","_npmVersion":"6.14.10","dist":{"shasum":"675446263c0c17a354879430e4ac2ea6be2e145e","size":16257,"noattachment":false,"key":"/s3-node/-/s3-node-0.0.1.tgz","tarball":"http://registry.cnpm.dingdandao.com/s3-node/download/s3-node-0.0.1.tgz"},"_npmUser":{"name":"ncnipunchawla","email":"ncnipunchawla@gmail.com"},"maintainers":[{"name":"ncnipunchawla","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/s3-node_0.0.1_1610632261474_0.325602926477514"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2021-12-26T21:20:39.823Z","publish_time":1610632261607,"_cnpm_publish_time":1610632261607}},"readme":"# High Level Amazon S3 Client\n\n## Installation\n\n`npm install s3-node --save`\n\n## Features\n\n * Automatically retry a configurable number of times when S3 returns an error.\n * Includes logic to make multiple requests when there is a 1000 object limit.\n * Ability to set a limit on the maximum parallelization of S3 requests.\n   Retries get pushed to the end of the parallelization queue.\n * Ability to sync a dir to and from S3.\n * Progress reporting.\n * Supports files of any size (up to S3's maximum 5 TB object size limit).\n * Uploads large files quickly using parallel multipart uploads.\n * Uses heuristics to compute multipart ETags client-side to avoid uploading\n   or downloading files unnecessarily.\n * Automatically provide Content-Type for uploads based on file extension.\n * Support third-party S3-compatible platform services like Ceph\n\nSee also the companion CLI tool which is meant to be a drop-in replacement for\ns3cmd: [s3-cli](https://github.com/ncnipunchawla/node-s3-client.git).\n\n## Synopsis\n\n### Create a client\n\n```js\nvar s3 = require('s3-node');\n\nvar client = s3.createClient({\n  maxAsyncS3: 20,     // this is the default\n  s3RetryCount: 3,    // this is the default\n  s3RetryDelay: 1000, // this is the default\n  multipartUploadThreshold: 20971520, // this is the default (20 MB)\n  multipartUploadSize: 15728640, // this is the default (15 MB)\n  s3Options: {\n    accessKeyId: \"your s3 key\",\n    secretAccessKey: \"your s3 secret\",\n    region: \"your region\",\n    // endpoint: 's3.yourdomain.com',\n    // sslEnabled: false\n    // any other options are passed to new AWS.S3()\n    // See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#constructor-property\n  },\n});\n```\n\n### Create a client from existing AWS.S3 object\n\n```js\nvar s3 = require('s3-node');\nvar awsS3Client = new AWS.S3(s3Options);\nvar options = {\n  s3Client: awsS3Client,\n  // more options available. See API docs below.\n};\nvar client = s3.createClient(options);\n```\n\n### Upload a file to S3\n\n```js\nvar params = {\n  localFile: \"some/local/file\",\n\n  s3Params: {\n    Bucket: \"s3 bucket name\",\n    Key: \"some/remote/file\",\n    // other options supported by putObject, except Body and ContentLength.\n    // See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property\n  },\n};\nvar uploader = client.uploadFile(params);\nuploader.on('error', function(err) {\n  console.error(\"unable to upload:\", err.stack);\n});\nuploader.on('progress', function() {\n  console.log(\"progress\", uploader.progressMd5Amount,\n            uploader.progressAmount, uploader.progressTotal);\n});\nuploader.on('end', function() {\n  console.log(\"done uploading\");\n});\n```\n\n### Download a file from S3\n\n```js\nvar params = {\n  localFile: \"some/local/file\",\n\n  s3Params: {\n    Bucket: \"s3 bucket name\",\n    Key: \"some/remote/file\",\n    // other options supported by getObject\n    // See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property\n  },\n};\nvar downloader = client.downloadFile(params);\ndownloader.on('error', function(err) {\n  console.error(\"unable to download:\", err.stack);\n});\ndownloader.on('progress', function() {\n  console.log(\"progress\", downloader.progressAmount, downloader.progressTotal);\n});\ndownloader.on('end', function() {\n  console.log(\"done downloading\");\n});\n```\n\n### Sync a directory to S3\n\n```js\nvar params = {\n  localDir: \"some/local/dir\",\n  deleteRemoved: true, // default false, whether to remove s3 objects\n                       // that have no corresponding local file.\n\n  s3Params: {\n    Bucket: \"s3 bucket name\",\n    Prefix: \"some/remote/dir/\",\n    // other options supported by putObject, except Body and ContentLength.\n    // See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property\n  },\n};\nvar uploader = client.uploadDir(params);\nuploader.on('error', function(err) {\n  console.error(\"unable to sync:\", err.stack);\n});\nuploader.on('progress', function() {\n  console.log(\"progress\", uploader.progressAmount, uploader.progressTotal);\n});\nuploader.on('end', function() {\n  console.log(\"done uploading\");\n});\n```\n\n## Tips\n\n * Consider increasing the socket pool size in the `http` and `https` global\n   agents. This will improve bandwidth when using `uploadDir` and `downloadDir`\n   functions. For example:\n\n   ```js\n   http.globalAgent.maxSockets = https.globalAgent.maxSockets = 20;\n   ```\n\n## API Documentation\n\n### s3.AWS\n\nThis contains a reference to the aws-sdk module. It is a valid use case to use\nboth this module and the lower level aws-sdk module in tandem.\n\n### s3.createClient(options)\n\nCreates an S3 client.\n\n`options`:\n\n * `s3Client` - optional, an instance of `AWS.S3`. Leave blank if you provide `s3Options`.\n * `s3Options` - optional. leave blank if you provide `s3Client`.\n   - See AWS SDK documentation for available options which are passed to `new AWS.S3()`:\n     http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#constructor-property\n * `maxAsyncS3` - maximum number of simultaneous requests this client will\n   ever have open to S3. defaults to `20`.\n * `s3RetryCount` - how many times to try an S3 operation before giving up.\n   Default 3.\n * `s3RetryDelay` - how many milliseconds to wait before retrying an S3\n   operation. Default 1000.\n * `multipartUploadThreshold` - if a file is this many bytes or greater, it\n   will be uploaded via a multipart request. Default is 20MB. Minimum is 5MB.\n   Maximum is 5GB.\n * `multipartUploadSize` - when uploading via multipart, this is the part size.\n   The minimum size is 5MB. The maximum size is 5GB. Default is 15MB. Note that\n   S3 has a maximum of 10000 parts for a multipart upload, so if this value is\n   too small, it will be ignored in favor of the minimum necessary value\n   required to upload the file.\n\n### s3.getPublicUrl(bucket, key, [bucketLocation])\n\n * `bucket` S3 bucket\n * `key` S3 key\n * `bucketLocation` string, one of these:\n   - \"\" (default) - US Standard\n   - \"eu-west-1\"\n   - \"us-west-1\"\n   - \"us-west-2\"\n   - \"ap-southeast-1\"\n   - \"ap-southeast-2\"\n   - \"ap-northeast-1\"\n   - \"sa-east-1\"\n\nYou can find out your bucket location programatically by using this API:\nhttp://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getBucketLocation-property\n\nreturns a string which looks like this:\n\n`https://s3.amazonaws.com/bucket/key`\n\nor maybe this if you are not in US Standard:\n\n`https://s3-eu-west-1.amazonaws.com/bucket/key`\n\n### s3.getPublicUrlHttp(bucket, key)\n\n * `bucket` S3 Bucket\n * `key` S3 Key\n\nWorks for any region, and returns a string which looks like this:\n\n`http://bucket.s3.amazonaws.com/key`\n\n### client.uploadFile(params)\n\nSee http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property\n\n`params`:\n\n * `s3Params`: params to pass to AWS SDK `putObject`.\n * `localFile`: path to the file on disk you want to upload to S3.\n * (optional) `defaultContentType`: Unless you explicitly set the `ContentType`\n   parameter in `s3Params`, it will be automatically set for you based on the\n   file extension of `localFile`. If the extension is unrecognized,\n   `defaultContentType` will be used instead. Defaults to\n   `application/octet-stream`.\n\nThe difference between using AWS SDK `putObject` and this one:\n\n * This works with files, not streams or buffers.\n * If the reported MD5 upon upload completion does not match, it retries.\n * If the file size is large enough, uses multipart upload to upload parts in\n   parallel.\n * Retry based on the client's retry settings.\n * Progress reporting.\n * Sets the `ContentType` based on file extension if you do not provide it.\n\nReturns an `EventEmitter` with these properties:\n\n * `progressMd5Amount`\n * `progressAmount`\n * `progressTotal`\n\nAnd these events:\n\n * `'error' (err)`\n * `'end' (data)` - emitted when the file is uploaded successfully\n   - `data` is the same object that you get from `putObject` in AWS SDK\n * `'progress'` - emitted when `progressMd5Amount`, `progressAmount`, and\n   `progressTotal` properties change. Note that it is possible for progress to\n   go backwards when an upload fails and must be retried.\n * `'fileOpened' (fdSlicer)` - emitted when `localFile` has been opened. The file\n   is opened with the [fd-slicer](https://github.com/ncnipunchawla/node-s3-client.git)\n   module because we might need to read from multiple locations in the file at\n   the same time. `fdSlicer` is an object for which you can call\n   `createReadStream(options)`. See the fd-slicer README for more information.\n * `'fileClosed'` - emitted when `localFile` has been closed.\n\nAnd these methods:\n\n * `abort()` - call this to stop the find operation.\n\n### client.downloadFile(params)\n\nSee http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property\n\n`params`:\n\n * `localFile` - the destination path on disk to write the s3 object into\n * `s3Params`: params to pass to AWS SDK `getObject`.\n\nThe difference between using AWS SDK `getObject` and this one:\n\n * This works with a destination file, not a stream or a buffer.\n * If the reported MD5 upon download completion does not match, it retries.\n * Retry based on the client's retry settings.\n * Progress reporting.\n\nReturns an `EventEmitter` with these properties:\n\n * `progressAmount`\n * `progressTotal`\n\nAnd these events:\n\n * `'error' (err)`\n * `'end'` - emitted when the file is downloaded successfully\n * `'progress'` - emitted when `progressAmount` and `progressTotal`\n   properties change.\n\n### client.downloadBuffer(s3Params)\n\nhttp://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property\n\n * `s3Params`: params to pass to AWS SDK `getObject`.\n\nThe difference between using AWS SDK `getObject` and this one:\n\n * This works with a buffer only.\n * If the reported MD5 upon download completion does not match, it retries.\n * Retry based on the client's retry settings.\n * Progress reporting.\n\nReturns an `EventEmitter` with these properties:\n\n * `progressAmount`\n * `progressTotal`\n\nAnd these events:\n\n * `'error' (err)`\n * `'end' (buffer)` - emitted when the file is downloaded successfully.\n   `buffer` is a `Buffer` containing the object data.\n * `'progress'` - emitted when `progressAmount` and `progressTotal`\n   properties change.\n\n### client.downloadStream(s3Params)\n\nhttp://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property\n\n * `s3Params`: params to pass to AWS SDK `getObject`.\n\nThe difference between using AWS SDK `getObject` and this one:\n\n * This works with a stream only.\n\nIf you want retries, progress, or MD5 checking, you must code it yourself.\n\nReturns a `ReadableStream` with these additional events:\n\n * `'httpHeaders' (statusCode, headers)` - contains the HTTP response\n   headers and status code.\n\n### client.listObjects(params)\n\nSee http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjects-property\n\n`params`:\n\n * `s3Params` - params to pass to AWS SDK `listObjects`.\n * (optional) `recursive` - `true` or `false` whether or not you want to recurse\n   into directories. Default `false`.\n\nNote that if you set `Delimiter` in `s3Params` then you will get a list of\nobjects and folders in the directory you specify. You probably do not want to\nset `recursive` to `true` at the same time as specifying a `Delimiter` because\nthis will cause a request per directory. If you want all objects that share a\nprefix, leave the `Delimiter` option `null` or `undefined`.\n\nBe sure that `s3Params.Prefix` ends with a trailing slash (`/`) unless you\nare requesting the top-level listing, in which case `s3Params.Prefix` should\nbe empty string.\n\nThe difference between using AWS SDK `listObjects` and this one:\n\n * Retries based on the client's retry settings.\n * Supports recursive directory listing.\n * Makes multiple requests if the number of objects to list is greater than 1000.\n\nReturns an `EventEmitter` with these properties:\n\n * `progressAmount`\n * `objectsFound`\n * `dirsFound`\n\nAnd these events:\n\n * `'error' (err)`\n * `'end'` - emitted when done listing and no more 'data' events will be emitted.\n * `'data' (data)` - emitted when a batch of objects are found. This is\n   the same as the `data` object in AWS SDK.\n * `'progress'` - emitted when `progressAmount`, `objectsFound`, and\n   `dirsFound` properties change.\n\nAnd these methods:\n\n * `abort()` - call this to stop the find operation.\n\n### client.deleteObjects(s3Params)\n\nSee http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#deleteObjects-property\n\n`s3Params` are the same.\n\nThe difference between using AWS SDK `deleteObjects` and this one:\n\n * Retry based on the client's retry settings.\n * Make multiple requests if the number of objects you want to delete is\n   greater than 1000.\n\nReturns an `EventEmitter` with these properties:\n\n * `progressAmount`\n * `progressTotal`\n\nAnd these events:\n\n * `'error' (err)`\n * `'end'` - emitted when all objects are deleted.\n * `'progress'` - emitted when the `progressAmount` or `progressTotal` properties change.\n * `'data' (data)` - emitted when a request completes. There may be more.\n\n### client.uploadDir(params)\n\nSyncs an entire directory to S3.\n\n`params`:\n\n * `localDir` - source path on local file system to sync to S3\n * `s3Params`\n   - `Prefix` (required)\n   - `Bucket` (required)\n * (optional) `deleteRemoved` - delete s3 objects with no corresponding local file.\n   default false\n * (optional) `getS3Params` - function which will be called for every file that\n   needs to be uploaded. You can use this to skip some files. See below.\n * (optional) `defaultContentType`: Unless you explicitly set the `ContentType`\n   parameter in `s3Params`, it will be automatically set for you based on the\n   file extension of `localFile`. If the extension is unrecognized,\n   `defaultContentType` will be used instead. Defaults to\n   `application/octet-stream`.\n * (optional) `followSymlinks` - Set this to `false` to ignore symlinks.\n   Defaults to `true`.\n\n```js\nfunction getS3Params(localFile, stat, callback) {\n  // call callback like this:\n  var err = new Error(...); // only if there is an error\n  var s3Params = { // if there is no error\n    ContentType: getMimeType(localFile), // just an example\n  };\n  // pass `null` for `s3Params` if you want to skip uploading this file.\n  callback(err, s3Params);\n}\n```\n\nReturns an `EventEmitter` with these properties:\n\n * `progressAmount`\n * `progressTotal`\n * `progressMd5Amount`\n * `progressMd5Total`\n * `deleteAmount`\n * `deleteTotal`\n * `filesFound`\n * `objectsFound`\n * `doneFindingFiles`\n * `doneFindingObjects`\n * `doneMd5`\n\nAnd these events:\n\n * `'error' (err)`\n * `'end'` - emitted when all files are uploaded\n * `'progress'` - emitted when any of the above progress properties change.\n * `'fileUploadStart' (localFilePath, s3Key)` - emitted when a file begins\n   uploading.\n * `'fileUploadEnd' (localFilePath, s3Key)` - emitted when a file successfully\n   finishes uploading.\n\n`uploadDir` works like this:\n\n 0. Start listing all S3 objects for the target `Prefix`. S3 guarantees\n    returned objects to be in sorted order.\n 0. Meanwhile, recursively find all files in `localDir`.\n 0. Once all local files are found, we sort them (the same way that S3 sorts).\n 0. Next we iterate over the sorted local file list one at a time, computing\n    MD5 sums.\n 0. Now S3 object listing and MD5 sum computing are happening in parallel. As\n    each operation progresses we compare both sorted lists side-by-side,\n    iterating over them one at a time, uploading files whose MD5 sums don't\n    match the remote object (or the remote object is missing), and, if\n    `deleteRemoved` is set, deleting remote objects whose corresponding local\n    files are missing.\n\n### client.downloadDir(params)\n\nSyncs an entire directory from S3.\n\n`params`:\n\n * `localDir` - destination directory on local file system to sync to\n * `s3Params`\n   - `Prefix` (required)\n   - `Bucket` (required)\n * (optional) `deleteRemoved` - delete local files with no corresponding s3 object. default `false`\n * (optional) `getS3Params` - function which will be called for every object that\n   needs to be downloaded. You can use this to skip downloading some objects.\n   See below.\n * (optional) `followSymlinks` - Set this to `false` to ignore symlinks.\n   Defaults to `true`.\n\n```js\nfunction getS3Params(localFile, s3Object, callback) {\n  // localFile is the destination path where the object will be written to\n  // s3Object is same as one element in the `Contents` array from here:\n  // http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjects-property\n\n  // call callback like this:\n  var err = new Error(...); // only if there is an error\n  var s3Params = { // if there is no error\n    VersionId: \"abcd\", // just an example\n  };\n  // pass `null` for `s3Params` if you want to skip downloading this object.\n  callback(err, s3Params);\n}\n```\n\nReturns an `EventEmitter` with these properties:\n\n * `progressAmount`\n * `progressTotal`\n * `progressMd5Amount`\n * `progressMd5Total`\n * `deleteAmount`\n * `deleteTotal`\n * `filesFound`\n * `objectsFound`\n * `doneFindingFiles`\n * `doneFindingObjects`\n * `doneMd5`\n\nAnd these events:\n\n * `'error' (err)`\n * `'end'` - emitted when all files are downloaded\n * `'progress'` - emitted when any of the progress properties above change\n * `'fileDownloadStart' (localFilePath, s3Key)` - emitted when a file begins\n   downloading.\n * `'fileDownloadEnd' (localFilePath, s3Key)` - emitted when a file successfully\n   finishes downloading.\n\n`downloadDir` works like this:\n\n 0. Start listing all S3 objects for the target `Prefix`. S3 guarantees\n    returned objects to be in sorted order.\n 0. Meanwhile, recursively find all files in `localDir`.\n 0. Once all local files are found, we sort them (the same way that S3 sorts).\n 0. Next we iterate over the sorted local file list one at a time, computing\n    MD5 sums.\n 0. Now S3 object listing and MD5 sum computing are happening in parallel. As\n    each operation progresses we compare both sorted lists side-by-side,\n    iterating over them one at a time, downloading objects whose MD5 sums don't\n    match the local file (or the local file is missing), and, if\n    `deleteRemoved` is set, deleting local files whose corresponding objects\n    are missing.\n\n### client.deleteDir(s3Params)\n\nDeletes an entire directory on S3.\n\n`s3Params`:\n\n * `Bucket`\n * `Prefix`\n * (optional) `MFA`\n\nReturns an `EventEmitter` with these properties:\n\n * `progressAmount`\n * `progressTotal`\n\nAnd these events:\n\n * `'error' (err)`\n * `'end'` - emitted when all objects are deleted.\n * `'progress'` - emitted when the `progressAmount` or `progressTotal` properties change.\n\n`deleteDir` works like this:\n\n 0. Start listing all objects in a bucket recursively. S3 returns 1000 objects\n    per response.\n 0. For each response that comes back with a list of objects in the bucket,\n    immediately send a delete request for all of them.\n\n### client.copyObject(s3Params)\n\nSee http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#copyObject-property\n\n`s3Params` are the same. Don't forget that `CopySource` must contain the\nsource bucket name as well as the source key name.\n\nThe difference between using AWS SDK `copyObject` and this one:\n\n * Retry based on the client's retry settings.\n\nReturns an `EventEmitter` with these events:\n\n * `'error' (err)`\n * `'end' (data)`\n\n### client.moveObject(s3Params)\n\nSee http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#copyObject-property\n\n`s3Params` are the same. Don't forget that `CopySource` must contain the\nsource bucket name as well as the source key name.\n\nUnder the hood, this uses `copyObject` and then `deleteObjects` only if the\ncopy succeeded.\n\nReturns an `EventEmitter` with these events:\n\n * `'error' (err)`\n * `'copySuccess' (data)`\n * `'end' (data)`\n\n## Examples\n\n### Check if a file exists in S3\n\nUsing the AWS SDK, you can send a HEAD request, which will tell you if a file exists at `Key`.\n\nSee http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property\n\n```js\nvar client = require('s3-node').createClient({ /* options */ });\nclient.s3.headObject({\n  Bucket: 's3 bucket name',\n  Key: 'some/remote/file'\n}, function(err, data) {\n  if (err) {\n    // file does not exist (err.statusCode == 404)\n    return;\n  }\n  // file exists\n});\n```\n\n## Testing\n\n`S3_KEY=<valid_s3_key> S3_SECRET=<valid_s3_secret> S3_BUCKET=<valid_s3_bucket> npm test`\n\nTests upload and download large amounts of data to and from S3. The test\ntimeout is set to 40 seconds because Internet connectivity waries wildly.\n","_attachments":{},"homepage":"https://github.com/ncnipunchawla/node-s3-client#readme","bugs":{"url":"https://github.com/ncnipunchawla/node-s3-client.git/issues"},"license":"MIT"}