{"_id":"batchflow","_rev":"93918","name":"batchflow","description":"Batch process collections in parallel or sequentially.","dist-tags":{"latest":"0.4.0"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"time":{"modified":"2021-06-03T10:31:14.000Z","created":"2012-08-14T18:14:22.936Z","0.4.0":"2013-04-24T23:41:00.530Z","0.3.2":"2013-02-05T17:52:16.821Z","0.3.1":"2013-01-23T05:24:10.065Z","0.3.0":"2012-10-13T07:52:09.610Z","0.2.0":"2012-09-16T20:30:21.255Z","0.1.0":"2012-09-06T05:28:23.979Z","0.0.4":"2012-09-05T22:03:31.419Z","0.0.3":"2012-08-15T15:50:25.185Z","0.0.2":"2012-08-14T19:17:07.657Z","0.0.1":"2012-08-14T18:14:22.936Z"},"users":{"andrew.finnell":true,"mrded":true,"andriecool":true,"ahsanshafiq":true},"author":{"name":"JP","email":"jprichardson@gmail.com"},"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"versions":{"0.4.0":{"name":"batchflow","version":"0.4.0","description":"Batch process collections in parallel or sequentially.","homepage":["http://jprichardson.github.com/node-batchflow/"],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"devDependencies":{"testutil":"~0.4.0","mocha":"*","autoresolve":"0.0.3","string":"~1.2.0"},"readmeFilename":"README.md","_id":"batchflow@0.4.0","dist":{"shasum":"7d419df79b6b7587b06f9ea34f96ccef6f74e5b5","size":7481,"noattachment":false,"key":"/batchflow/-/batchflow-0.4.0.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.4.0.tgz"},"_from":".","_npmVersion":"1.2.17","_npmUser":{"name":"jp","email":"jprichardson@gmail.com"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1366846860530,"_cnpm_publish_time":1366846860530,"_hasShrinkwrap":false},"0.3.2":{"name":"batchflow","version":"0.3.2","description":"Batch process collections in parallel or sequentially.","homepage":["http://jprichardson.github.com/node-batchflow/"],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"devDependencies":{"testutil":"~0.4.0","mocha":"*","autoresolve":"0.0.3","string":"~1.2.0"},"_id":"batchflow@0.3.2","dist":{"shasum":"eaa7b14e7558c0047beac97154da512c82c59d61","size":7405,"noattachment":false,"key":"/batchflow/-/batchflow-0.3.2.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.3.2.tgz"},"_npmVersion":"1.1.62","_npmUser":{"name":"jp","email":"jprichardson@gmail.com"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1360086736821,"_cnpm_publish_time":1360086736821,"_hasShrinkwrap":false},"0.3.1":{"name":"batchflow","version":"0.3.1","description":"Batch process collections in parallel or sequentially.","homepage":["http://jprichardson.github.com/node-batchflow/"],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"devDependencies":{"testutil":"~0.4.0","mocha":"*","autoresolve":"0.0.3"},"readmeFilename":"README.md","_id":"batchflow@0.3.1","dist":{"shasum":"1649a5f0fe0a1754d73203d31287244bfbadf251","size":7107,"noattachment":false,"key":"/batchflow/-/batchflow-0.3.1.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.3.1.tgz"},"_npmVersion":"1.1.69","_npmUser":{"name":"jp","email":"jprichardson@gmail.com"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1358918650065,"_cnpm_publish_time":1358918650065,"_hasShrinkwrap":false},"0.3.0":{"name":"batchflow","version":"0.3.0","description":"Batch process collections in parallel or sequentially.","homepage":["http://jprichardson.github.com/node-batchflow/"],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"devDependencies":{"testutil":"~0.2.2","mocha":"*"},"_id":"batchflow@0.3.0","dist":{"shasum":"d620c672505abe6f0128d1055ae10ecf45448909","size":6462,"noattachment":false,"key":"/batchflow/-/batchflow-0.3.0.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.3.0.tgz"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1350114729610,"_cnpm_publish_time":1350114729610,"_hasShrinkwrap":false},"0.2.0":{"name":"batchflow","version":"0.2.0","description":"Batch process collections in parallel or sequentially.","homepage":[""],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"devDependencies":{"testutil":"~0.2.2","mocha":"*"},"_id":"batchflow@0.2.0","dist":{"shasum":"81d83209220310ef211ff717eab75f9963df1b14","size":5283,"noattachment":false,"key":"/batchflow/-/batchflow-0.2.0.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.2.0.tgz"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1347827421255,"_cnpm_publish_time":1347827421255,"_hasShrinkwrap":false},"0.1.0":{"name":"batchflow","version":"0.1.0","description":"Batch process collections in parallel or sequentially.","homepage":[""],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"devDependencies":{"testutil":"~0.2.2","mocha":"*"},"_id":"batchflow@0.1.0","dist":{"shasum":"e72a9ccb69347b11884061248307439f4b725ed4","size":5111,"noattachment":false,"key":"/batchflow/-/batchflow-0.1.0.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.1.0.tgz"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1346909303979,"_cnpm_publish_time":1346909303979,"_hasShrinkwrap":false},"0.0.4":{"name":"batchflow","version":"0.0.4","description":"Batch process collections in parallel or sequentially.","homepage":[""],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"devDepdencies":{"mocha":"1.3.x"},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"devDependencies":{"testutil":"~0.2.2"},"_id":"batchflow@0.0.4","dist":{"shasum":"623a8c4c9fcdbb82253c36e1964f23b08ccd029d","size":4905,"noattachment":false,"key":"/batchflow/-/batchflow-0.0.4.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.0.4.tgz"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1346882611419,"_cnpm_publish_time":1346882611419,"_hasShrinkwrap":false},"0.0.3":{"name":"batchflow","version":"0.0.3","description":"Batch process collections in parallel or sequentially.","homepage":[""],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"devDepdencies":{"mocha":"1.3.x"},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"_id":"batchflow@0.0.3","dist":{"shasum":"ad586534090c12fc55aa8ace458bc07f5b884bb2","size":4555,"noattachment":false,"key":"/batchflow/-/batchflow-0.0.3.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.0.3.tgz"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1345045825185,"_cnpm_publish_time":1345045825185,"_hasShrinkwrap":false},"0.0.2":{"name":"batchflow","version":"0.0.2","description":"Batch process collections in parallel or sequentially.","homepage":[""],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"devDepdencies":{"mocha":"1.3.x"},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"_id":"batchflow@0.0.2","dist":{"shasum":"3466b0456b5f7dadb39eb8ac5693e629498f53fc","size":4216,"noattachment":false,"key":"/batchflow/-/batchflow-0.0.2.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.0.2.tgz"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1344971827657,"_cnpm_publish_time":1344971827657,"_hasShrinkwrap":false},"0.0.1":{"name":"batchflow","version":"0.0.1","description":"Batch process collections in parallel or sequentially.","homepage":[""],"repository":{"type":"git","url":"git@github.com:jprichardson/node-batchflow.git"},"keywords":["batch","flow","parallel","control","sequence","sequentially","arrays","async","step","seq"],"author":{"name":"JP","email":"jprichardson@gmail.com"},"licenses":[{"type":"MIT","url":""}],"dependencies":{},"devDepdencies":{"mocha":"1.3.x"},"main":"./lib/batchflow.js","scripts":{"test":"mocha test"},"_id":"batchflow@0.0.1","dist":{"shasum":"a68f9487ddc1f260640fe4847ab2bb40365276c9","size":3549,"noattachment":false,"key":"/batchflow/-/batchflow-0.0.1.tgz","tarball":"http://registry.cnpm.dingdandao.com/batchflow/download/batchflow-0.0.1.tgz"},"maintainers":[{"name":"jp","email":"j.p@hotmail.com"}],"directories":{},"publish_time":1344968062936,"_cnpm_publish_time":1344968062936,"_hasShrinkwrap":false}},"readme":"\nNode.js - batchflow\n===================\n\n[![build status](https://secure.travis-ci.org/jprichardson/node-batchflow.png)](http://travis-ci.org/jprichardson/node-batchflow)\n\nBatch process collections in parallel or sequentially.\n\n\nWhy?\n----\n\nI really got tired of writing the following patterns over and over again:\n\n**Sequential:**\n\n```javascript\nvar files = [... list of files ...];\nfunction again(x) {\n\tif (x < files.length) {\n\t\tfs.readFile(files[x], function(err, data) {\n\t\t\t//... do something with data ...\n\t\t\tagain(x + 1);\n\t\t});\n\t} else {\n\t\tconsole.log('Done.');\n\t}\n}\n\nagain(0);\n```\n\nor..\n\n**Parallel:**\n\n```javascript\nvar files = [... list of files ...];\nvar pending = 0;\nfiles.forEach(function(file, i) {\n\tpending += 1;\n\tfs.readFile(file, function(err, data) {\n\t\t//... do something with data ....\n\t\t\n\t\tpending -= 1;\n\t\tif (pending === 0 && i === files.length -1) {\n\t\t\tconsole.log('Done.');\n\t\t}\n\t});\n});\n```\n\nThat's ugly. For more complicated examples it requires a bit more thinking.\n\nWhy don't I use the wonderful library [async][1]? Well, `async` tries to do way too much. I also suffer from a severe case of NIH syndrome. Kidding, or else I'd rewrite Express.js. Or, am I? Muahahhaa. `async` syntax is also very ugly and not CoffeeScript friendly.\n\n\n\n\nInstallation\n------------\n\n    npm install batchflow\n\n\n\nExamples\n--------\n\n### 50 Foot Overview\n\nSimple Sequential Example:\n\n```javascript\nvar a = [\n        function(finished) { setTimeout(function(){finished(1)}, 1); }, //executes in 1 ms\n        function(finished) { setTimeout(function(){finished(2)}, 20); }, //executes in 20 ms\n        function(finished) { setTimeout(function(){finished(3)}, 2); } //executes in 2 ms\n    ];\n\n//sequential\nbatch(a).sequential()\n.each(function(i, item, done) {\n  item(done);\n}).end(function(results) {\n  for (var i = 0; i < results.length; ++i) {\n    console.log(results[i]);\n  }\n});\n\n/*\n  1\n  2\n  3\n*/\n```\n\nSimple Parallel Example:\n\n```javascript\n//parallel\nbatch(a).parallel()\n.each(function(i, item, done) {\n  item(done);\n}).end(function(results) {\n  for (var i = 0; i < results.length; ++i) {\n    console.log(results[i]);\n  }\n});\n\n/*\n  1\n  3\n  2\n*/\n```\n\n### Arrays\n\nLet's rewrite the previous file patterns mentioned in **Why?** into a sequential example:\n\n**Sequential:**\n\n```javascript\nvar batch = require('batchflow');\n\nvar files = [... list of files ...];\nbatch(files).sequential()\n.each(function(i, item, next) {\n\tfs.readFile(item, function(err, data) {\n\t\t//do something with data\n\t\tnext(someResult);\n\t});\n}).end(function(results) {\n\t//analyze results\n});\n```\n\nHow about the parallel example?\n\n**Parallel:**\n \n ```javascript\nvar batch = require('batchflow');\n\nvar files = [... list of files ...];\nbatch(files).parallel()\n.each(function(i, item, done) {\n\tfs.readFile(item, function(err, data) {\n\t\t//do something with data\n\t\tdone(someResult); //<---- yes, you must still call done() in parallel, this way we can know when to trigger `end()`.\n\t});\n}).end(function(results) {\n\t//analyze results\n});\n```\n\nWhat's that, your data is not stored in an array? Oh, you say it's stored in an object? That's OK too...\n\n### Objects\n\n**Sequential:**\n\n```javascript\nvar batch = require('batchflow');\n\nvar files = {'file1': 'path'.... 'filen': 'pathn'}\nbatch(files).sequential()\n.each(function(key, val, next) {\n\tfs.readFile(val, function(err, data) {\n\t\t//do something with data\n\t\tnext(someResult);\n\t});\n}).end(function(results) {\n\t//analyze results\n});\n```\n\nHow about the parallel example?\n\n**Parallel:**\n\n ```javascript\nvar batch = require('batchflow');\n\nvar files = {'file1': 'path'.... 'filen': 'pathn'}\nbatch(files).parallel()\n.each(function(key, val, done) {\n\tfs.readFile(val, function(err, data) {\n\t\t//do something with data\n\t\tdone(someResult);\n\t});\n}).end(function(results) {\n\t//analyze results\n});\n```\n\n### Misc\n\n1. Is `sequential()` or `parallel()` too long? Fine. `series()` and `seq()` are aliases for `sequential()`. `par()` is an alias for `parallel()`.\n2. You don't like the fluent API? That's OK too:\n\nNon-fluent API BatchFlow\n\n```javascript\nvar batch = require('batchflow');\nvar bf = batch(files);\nbf.sequential()\n\nbf.each(function(i, file, next) {\n\tnext(someResult);\n});\n \nbf.end(function(results) {\n\t//blah blah\n});\n```\n\n\n### CoffeeScript Friendly\n\n```coffee\nbatch = require('batchflow')\nfiles = [... list of files ...]\nbf = batch(files).seq().each (i, file, done) ->\n  fs.readFile file, done\nbf.error (err) ->\n  console.log(err);\nbf.end (results) ->\n  console.log fr.toString() for fr in results\n```\n\n\n### Error Handling\n\nWhat's that, you want error handling? Well, you might as well call me Burger King... have it your way.\n\nNote that before version `0.3`, it would exit prematurely if an error happened. This was a boneheaded\ndesign decision. After `0.3`, it'll keep happily processing even if an error occured.\n\nCatch an error in the callback parameter...\n\n```javascript\nvar a = {'f': '/tmp/file_DOES_NOT_exist_hopefully' + Math.random()};\nbatch(a).parallel().each(function(i, item, done) {\n  fs.readFile(item, done);\n}).error(function(err) {\n  console.error(err);\n}).end(function(fileData) {\n  //do something with file data\n});\n```\n\nCatch an error in the function...\n\n```javascript\nvar a = ['/tmp/file_DOES_NOT_exist_hopefully' + Math.random()];\nbatch(a).series().each(function(i, item, done) {\n  throw new Error('err');\n}).error(function(err) {\n  console.error(err);\n}).end(function() {\n  //do something\n});\n\n```\n\n\n### Limits\n\nYou can set a limit to how many items can be processed in parallel. In fact, `sequential` mode is the same as having the `limit` set to `1` and calling `parallel`. In other words: `batch(myArray).sequential() ....` is the same as `batch(myArray).parallel(1)`.\n\nTo set the limit, just pass the limit as a parameter to `parallel()`. The **default is 2^53** which is the max integer size in JavaScript.\n\nExample:\n\n```javascript\nbatch(myArray).parallel(8)\n.each(function(i, item, done){\n  // ... code here ... \n}).end(function(results){\n  // ... code here ...\n})\n```\n\n\n### Difference between done() and next()\n\nSo you noticed that in all of the examples where I was calling `sequential()` the third parameter is named `next` and in the examples where I was calling `parallel()` the third parameter is named `done`?  This is really just a matter of convention. It could be named `fruitypebbles`. But in sequential processing, it makes sense for it to be `next` because you want to process the next one. However, in parallel processing, you want to alert the system that the callback is `done`.\n\nSequential...\n\n```javascript\nbatch(myArray).sequential()\n.each(function(i, item, next) {\n  // ... code here ...\n}).end();\n```\n\nParallel...\n\n```javascript\nbatch(myArray).parallel()\n.each(function(i, item, done) {\n  // ... code here ...\n}).end();\n```\n\n\n### Progress\n\nYou can keep track of progress by accessing the `finished` field.\n\nCompute percentage by this formula: `(this.finished / this.total) * 100.0`.\n\nExample:\n\n```javascript\nvar myar = {w: 'hi', b: 'hello', c: 'hola', z: 'gutentag'}\nbatch(myar).sequential()\n.each(function(i, item, next) {\n  console.log(this.finished) //the number finished.\n  console.log(this.total) //4\n  console.log((this.finished / this.total) * 100.0) //{percent complete}\n})\n.end();\n```\n\n\nAuthor\n------\n\n`node-batchflow` was written by [JP Richardson][aboutjp]. You should follow him on Twitter [@jprichardson][twitter]. Also read his coding blog [Procbits][procbits]. If you write software with others, you should checkout [Gitpilot][gitpilot] to make collaboration with Git simple.\n\n\n\nLicense\n-------\n\n(MIT License)\n\nCopyright 2012, JP  <jprichardson@gmail.com>\n\n\n\n[1]: https://github.com/caolan/async/\n\n[aboutjp]: http://about.me/jprichardson\n[twitter]: http://twitter.com/jprichardson\n[procbits]: http://procbits.com\n[gitpilot]: http://gitpilot.com\n\n\n","_attachments":{},"readmeFilename":"README.md","homepage":["http://jprichardson.github.com/node-batchflow/"]}