{"_id":"chunk-data","_rev":"4298135","name":"chunk-data","description":"Split buffers and streams into smaller chunks","dist-tags":{"latest":"0.1.0"},"maintainers":[{"name":"sindresorhus","email":""}],"time":{"modified":"2026-04-07T21:03:44.000Z","created":"2025-10-20T14:07:28.937Z","0.1.0":"2025-10-20T14:07:28.937Z"},"users":{},"author":{"name":"Sindre Sorhus","email":"sindresorhus@gmail.com","url":"https://sindresorhus.com"},"repository":{"type":"git","url":"git+https://github.com/sindresorhus/chunk-data.git"},"versions":{"0.1.0":{"name":"chunk-data","version":"0.1.0","description":"Split buffers and streams into smaller chunks","license":"MIT","repository":{"type":"git","url":"git+https://github.com/sindresorhus/chunk-data.git"},"funding":"https://github.com/sponsors/sindresorhus","author":{"name":"Sindre Sorhus","email":"sindresorhus@gmail.com","url":"https://sindresorhus.com"},"type":"module","exports":{"types":"./index.d.ts","default":"./index.js"},"sideEffects":false,"engines":{"node":">=20"},"scripts":{"test":"xo && node --test && tsd"},"keywords":["chunk","chunks","chunking","split","buffer","stream","readable","iterable","generator","async","progress","upload","uint8array","typed-array","http","request"],"devDependencies":{"@types/node":"^24.8.1","tsd":"^0.33.0","xo":"^1.2.2"},"gitHead":"aecf2a66ec25f0286fe2a32f97e2ffc78c6b6cbb","types":"./index.d.ts","_id":"chunk-data@0.1.0","bugs":{"url":"https://github.com/sindresorhus/chunk-data/issues"},"homepage":"https://github.com/sindresorhus/chunk-data#readme","_nodeVersion":"24.9.0","_npmVersion":"11.6.1","dist":{"shasum":"086f79f356b6c44d0cf0a36db2cec3e1daa40bb1","size":4208,"noattachment":false,"key":"/chunk-data/-/chunk-data-0.1.0.tgz","tarball":"http://registry.cnpm.dingdandao.com/chunk-data/download/chunk-data-0.1.0.tgz"},"_npmUser":{"name":"sindresorhus","email":"sindresorhus@gmail.com"},"directories":{},"maintainers":[{"name":"sindresorhus","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/chunk-data_0.1.0_1760969248742_0.1229937984074907"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-10-20T14:07:28.937Z","publish_time":1760969248937,"_source_registry_name":"default","_cnpm_publish_time":1760969248937}},"readme":"# chunk-data\n\n> Split buffers and streams into smaller chunks\n\nUse case: When working with HTTP uploads and progress tracking, you need the request body to be sent in multiple chunks. However, data sources like `Readable.from(buffer)` emit the entire buffer as a single chunk, causing upload progress to jump from 0% to 100% instantly with no intermediate updates.\n\n## Install\n\n```sh\nnpm install chunk-data\n```\n\n## Usage\n\n### Chunking a buffer\n\n```js\nimport {chunk} from 'chunk-data';\n\nconst largeBuffer = new Uint8Array(10_000_000); // 10 MB\n\n// Split into 64 KB chunks\nfor (const piece of chunk(largeBuffer, 65_536)) {\n\tconsole.log(piece.length);\n\t//=> 65536 (except last chunk)\n}\n```\n\n### Chunking an iterable of buffers\n\n```js\nimport {chunkFrom} from 'chunk-data';\n\nconst buffers = [new Uint8Array(1000), new Uint8Array(2000)];\n\n// Re-chunk into 500-byte chunks\nfor (const piece of chunkFrom(buffers, 500)) {\n\tconsole.log(piece.length);\n\t//=> 500, 500, 500, 500, 500, 1000\n}\n```\n\n### Chunking a stream\n\n```js\nimport {chunkFromAsync} from 'chunk-data';\nimport {Readable} from 'node:stream';\n\nconst largeBuffer = new Uint8Array(10_000_000); // 10 MB\nconst stream = Readable.from([largeBuffer]);\n\n// Split into 64 KB chunks\nfor await (const piece of chunkFromAsync(stream, 65_536)) {\n\tconsole.log(piece.length);\n\t//=> 65536 (except last chunk)\n}\n```\n\n> [!TIP]\n> Node.js streams (like `Readable`) are async iterables and work directly with `chunkFromAsync()`.\n\n## API\n\n### chunk(data, chunkSize)\n\nSplits a typed array into smaller chunks.\n\nThis is a generator function that yields chunks of the specified size. Returns zero-copy views of the original buffer for optimal performance.\n\n#### data\n\nType: `ArrayBufferView`\n\nThe typed array to split into chunks. Can be `Uint8Array`, `Uint16Array`, `Int32Array`, or any other typed array.\n\n#### chunkSize\n\nType: `number`\n\nThe maximum size of each chunk in bytes. Must be a positive integer.\n\nReturns: `Generator<Uint8Array>`\n\n### chunkFrom(iterable, chunkSize)\n\nSplits an iterable of typed arrays into smaller chunks.\n\nThis is a generator function that re-chunks data from sources that emit inconsistently-sized chunks. Accumulates small chunks and splits large chunks to ensure consistent output chunk sizes.\n\n#### iterable\n\nType: `Iterable<ArrayBufferView>`\n\nThe iterable of typed arrays to re-chunk. Can be an array of buffers, a generator, or any iterable that yields typed arrays.\n\n#### chunkSize\n\nType: `number`\n\nThe maximum size of each chunk in bytes. Must be a positive integer.\n\nReturns: `Generator<Uint8Array>`\n\n### chunkFromAsync(iterable, chunkSize)\n\nSplits an async iterable of typed arrays into smaller chunks.\n\nThis async generator consumes an async iterable or iterable and re-emits its data as smaller chunks. Accumulates small chunks and splits large chunks to ensure consistent output chunk sizes.\n\n#### iterable\n\nType: `AsyncIterable<ArrayBufferView> | Iterable<ArrayBufferView>`\n\nThe async iterable to read from and re-chunk. Can be any async iterable or iterable that yields typed arrays. Node.js streams are async iterables and work directly.\n\n#### chunkSize\n\nType: `number`\n\nThe maximum size of each output chunk in bytes. Must be a positive integer.\n\nReturns: `AsyncGenerator<Uint8Array>`\n\n## Why?\n\n**Problem:** Upload progress tracking requires sending data in multiple chunks, but many data sources emit large single chunks:\n\n```js\nimport {Readable} from 'node:stream';\n\nconst buffer = new Uint8Array(10_000_000);\nconst stream = Readable.from([buffer]);\n\n// This emits ONE chunk of 10 MB, not multiple smaller chunks\nfor await (const chunk of stream) {\n\tconsole.log(chunk.length);\n\t//=> 10000000 (entire buffer in one go)\n}\n```\n\n**Solution:** Use `chunkFromAsync()` to split it:\n\n```js\nimport {Readable} from 'node:stream';\nimport {chunkFromAsync} from 'chunk-data';\n\nconst buffer = new Uint8Array(10_000_000);\nconst stream = Readable.from([buffer]);\n\n// Now we get many smaller chunks\nfor await (const piece of chunkFromAsync(stream, 65_536)) {\n\tconsole.log(piece.length);\n\t//=> 65536\n\t//=> 65536\n\t//=> 65536\n\t//=> ... (152 chunks of 64 KB each, plus remainder)\n}\n```\n\nThis enables HTTP clients like Got to report incremental upload progress instead of jumping from 0% to 100%.\n\n## Performance\n\n### `chunk()` - Zero-copy views\n\nUses `subarray()` which creates views of the original data without copying. This means chunking is extremely fast and memory-efficient, even for large buffers.\n\n```js\nimport {chunk} from 'chunk-data';\n\nconst buffer = new Uint8Array(100_000_000); // 100 MB\n\nfor (const piece of chunk(buffer, 65_536)) {\n\t// No data copying - chunks are views into the original buffer\n\t// Chunking 100 MB takes <1ms\n}\n```\n\n### `chunkFrom()` and `chunkFromAsync()` - Optimized copying\n\nMay copy data to coalesce small input chunks, but optimized to minimize copying:\n- Emit zero-copy views from large input chunks whenever possible\n- Only copy when combining small chunks to reach the desired chunk size\n- Avoid O(n²) behavior by copying at most once per input chunk\n\nThis makes them efficient for both scenarios:\n- Sources that emit large chunks (mostly zero-copy)\n- Sources that emit many small chunks (minimal copying)\n\n## When to use which?\n\n- **`chunk()`** - When you have a single buffer in memory that you want to split into smaller pieces\n- **`chunkFrom()`** - When you have multiple buffers (like an array or generator) that you want to re-chunk synchronously\n- **`chunkFromAsync()`** - When you have a stream or async source that you want to re-chunk asynchronously\n\n## Related\n\n- [chunkify](https://github.com/sindresorhus/chunkify) - Split an iterable into evenly sized chunks\n","_attachments":{},"homepage":"https://github.com/sindresorhus/chunk-data#readme","bugs":{"url":"https://github.com/sindresorhus/chunk-data/issues"},"license":"MIT"}