{"_id":"tiktoken","_rev":"4109616","name":"tiktoken","description":"JS/WASM bindings for tiktoken","dist-tags":{"devel":"1.0.8","latest":"1.0.22"},"maintainers":[{"name":"davidduong","email":""}],"time":{"modified":"2026-02-01T17:23:40.000Z","created":"2023-02-13T20:40:40.212Z","1.0.22":"2025-08-09T01:00:31.943Z","1.0.21":"2025-04-22T09:02:20.314Z","1.0.20":"2025-02-04T01:58:23.595Z","1.0.19":"2025-02-02T21:19:25.062Z","1.0.18":"2024-12-19T17:06:16.276Z","1.0.17":"2024-10-04T09:35:14.771Z","1.0.16":"2024-08-15T23:08:43.067Z","1.0.15":"2024-05-13T22:17:26.131Z","1.0.14":"2024-04-12T23:06:22.583Z","1.0.13":"2024-01-28T07:50:59.155Z","1.0.12":"2024-01-28T07:13:06.395Z","1.0.11":"2023-11-15T23:04:33.802Z","1.0.10":"2023-06-24T23:21:22.953Z","1.0.9":"2023-06-22T23:53:54.867Z","1.0.8":"2023-06-15T20:57:52.006Z","1.0.7":"2023-05-13T11:42:38.116Z","0.0.1":"2023-02-13T20:42:17.095Z","1.0.0":"2023-02-13T20:40:40.212Z"},"users":{},"repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"versions":{"1.0.22":{"name":"tiktoken","version":"1.0.22","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"},"./encoders/o200k_base.json":"./encoders/o200k_base.json","./encoders/o200k_base":{"types":"./encoders/o200k_base.d.ts","edge-light":"./encoders/o200k_base.js","node":"./encoders/o200k_base.cjs","default":"./encoders/o200k_base.js"}},"_id":"tiktoken@1.0.22","gitHead":"4c8b748e07992c00386f3180af5c574b27b65139","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.20.8","_npmVersion":"10.8.2","dist":{"shasum":"a6c674839228bb88f32dfe646dff47193762f7d3","size":10611708,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.22.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.22.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tiktoken_1.0.22_1754701231604_0.792492783270798"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-08-09T01:00:31.943Z","publish_time":1754701231943,"_source_registry_name":"default","_cnpm_publish_time":1754701231943},"1.0.21":{"name":"tiktoken","version":"1.0.21","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"},"./encoders/o200k_base.json":"./encoders/o200k_base.json","./encoders/o200k_base":{"types":"./encoders/o200k_base.d.ts","edge-light":"./encoders/o200k_base.js","node":"./encoders/o200k_base.cjs","default":"./encoders/o200k_base.js"}},"_id":"tiktoken@1.0.21","gitHead":"9dd7d6ee3825b2cb01ec27b7a71427583ac6c7d9","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.20.8","_npmVersion":"10.8.2","dist":{"shasum":"434f4c67ccda114cdfb19a180b93d9be8bc396be","size":10602749,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.21.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.21.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tiktoken_1.0.21_1745312539958_0.77577814947034"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-04-22T09:02:20.314Z","publish_time":1745312540314,"_source_registry_name":"default","_cnpm_publish_time":1745312540314},"1.0.20":{"name":"tiktoken","version":"1.0.20","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"},"./encoders/o200k_base.json":"./encoders/o200k_base.json","./encoders/o200k_base":{"types":"./encoders/o200k_base.d.ts","edge-light":"./encoders/o200k_base.js","node":"./encoders/o200k_base.cjs","default":"./encoders/o200k_base.js"}},"_id":"tiktoken@1.0.20","gitHead":"e3437a351b425664294b51ce6b039b6345893058","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.20.6","_npmVersion":"10.8.2","dist":{"shasum":"a6d7ce757c009e7b4bc757aee7d512187b4b92a6","size":10615109,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.20.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.20.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tiktoken_1.0.20_1738634303215_0.5879569176006003"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-02-04T01:58:23.595Z","publish_time":1738634303595,"_source_registry_name":"default","_cnpm_publish_time":1738634303595},"1.0.19":{"name":"tiktoken","version":"1.0.19","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"},"./encoders/o200k_base.json":"./encoders/o200k_base.json","./encoders/o200k_base":{"types":"./encoders/o200k_base.d.ts","edge-light":"./encoders/o200k_base.js","node":"./encoders/o200k_base.cjs","default":"./encoders/o200k_base.js"}},"_id":"tiktoken@1.0.19","gitHead":"37c993f3f714dff6cb5975417b68a58eb859347a","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.20.6","_npmVersion":"10.8.2","dist":{"shasum":"7637d4ce871f8cde568723860fb0ad930ad32eeb","size":10615058,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.19.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.19.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tiktoken_1.0.19_1738531164135_0.5003265124450169"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2025-02-02T21:19:25.062Z","publish_time":1738531165062,"_source_registry_name":"default","_cnpm_publish_time":1738531165062},"1.0.18":{"name":"tiktoken","version":"1.0.18","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"},"./encoders/o200k_base.json":"./encoders/o200k_base.json","./encoders/o200k_base":{"types":"./encoders/o200k_base.d.ts","edge-light":"./encoders/o200k_base.js","node":"./encoders/o200k_base.cjs","default":"./encoders/o200k_base.js"}},"_id":"tiktoken@1.0.18","gitHead":"638e6a253e940bc78883f45fdbef0415cf2ff735","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.20.5","_npmVersion":"10.8.2","dist":{"shasum":"41c7d776ee5da27642e731d732fee3dde3fc7a2b","size":10606105,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.18.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.18.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tiktoken_1.0.18_1734627975600_0.42263614781551295"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2024-12-19T17:06:16.276Z","publish_time":1734627976276,"_source_registry_name":"default","_cnpm_publish_time":1734627976276},"1.0.17":{"name":"tiktoken","version":"1.0.17","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"},"./encoders/o200k_base.json":"./encoders/o200k_base.json","./encoders/o200k_base":{"types":"./encoders/o200k_base.d.ts","edge-light":"./encoders/o200k_base.js","node":"./encoders/o200k_base.cjs","default":"./encoders/o200k_base.js"}},"_id":"tiktoken@1.0.17","gitHead":"b742a86f032dde5d19f0bbb78866827139df609f","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.20.4","_npmVersion":"10.7.0","dist":{"shasum":"53b9b38b8b1a9c6996cded21f6ef0ff473fae265","size":10603412,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.17.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.17.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.17_1728034514385_0.1439731456404698"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2024-10-04T09:35:14.771Z","publish_time":1728034514771,"_source_registry_name":"default","_cnpm_publish_time":1728034514771},"1.0.16":{"name":"tiktoken","version":"1.0.16","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"},"./encoders/o200k_base.json":"./encoders/o200k_base.json","./encoders/o200k_base":{"types":"./encoders/o200k_base.d.ts","edge-light":"./encoders/o200k_base.js","node":"./encoders/o200k_base.cjs","default":"./encoders/o200k_base.js"}},"_id":"tiktoken@1.0.16","gitHead":"60ff9bb8aa126a6ded2218fb424edee01b8c11d5","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.20.4","_npmVersion":"10.7.0","dist":{"shasum":"efa91e6bb653dfbe81455865ea90ee845de915ad","size":10577861,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.16.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.16.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.16_1723763322696_0.6881845602153842"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2024-08-15T23:08:43.067Z","publish_time":1723763323067,"_source_registry_name":"default","_cnpm_publish_time":1723763323067},"1.0.15":{"name":"tiktoken","version":"1.0.15","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"},"./encoders/o200k_base.json":"./encoders/o200k_base.json","./encoders/o200k_base":{"types":"./encoders/o200k_base.d.ts","edge-light":"./encoders/o200k_base.js","node":"./encoders/o200k_base.cjs","default":"./encoders/o200k_base.js"}},"_id":"tiktoken@1.0.15","gitHead":"a7cce9922b10bca567be8453f1ef0489428fa02f","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.20.2","_npmVersion":"10.5.0","dist":{"shasum":"a1e11681fa51b50c81bb7eaaee53b7a66e844a23","size":10605988,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.15.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.15.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.15_1715638645787_0.494000620126285"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2024-05-13T22:17:26.131Z","publish_time":1715638646131,"_source_registry_name":"default","_cnpm_publish_time":1715638646131},"1.0.14":{"name":"tiktoken","version":"1.0.14","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"}},"_id":"tiktoken@1.0.14","gitHead":"045f0e431b29729c833c2ae6e458ac3320b4264a","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.20.1","_npmVersion":"10.5.0","dist":{"shasum":"1263821f4ba0a4ec71604db8608a3accd43001c9","size":6077922,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.14.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.14.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.14_1712963182270_0.14606268484074758"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2024-04-12T23:06:22.583Z","publish_time":1712963182583,"_source_registry_name":"default","_cnpm_publish_time":1712963182583},"1.0.13":{"name":"tiktoken","version":"1.0.13","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"}},"_id":"tiktoken@1.0.13","gitHead":"ffa82838d2532b267fccd7ac2c8dd77e2785d1f9","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.19.0","_npmVersion":"10.2.3","dist":{"shasum":"39e3d10a1c134d8dd98d88b023a9963c0f061b21","size":6078360,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.13.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.13.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.13_1706428258864_0.39298223200340865"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2024-01-28T07:50:59.155Z","publish_time":1706428259155,"_source_registry_name":"default","_cnpm_publish_time":1706428259155},"1.0.12":{"name":"tiktoken","version":"1.0.12","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"}},"_id":"tiktoken@1.0.12","gitHead":"a020c6af42372610f851b41cefa03454a217ab7e","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.19.0","_npmVersion":"10.2.3","dist":{"shasum":"48df45eb63b7d0b0839ffabbff8637efee9d9531","size":6078359,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.12.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.12.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.12_1706425986041_0.6912431545595323"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2024-01-28T07:13:06.395Z","publish_time":1706425986395,"_source_registry_name":"default","_cnpm_publish_time":1706425986395},"1.0.11":{"name":"tiktoken","version":"1.0.11","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"}},"_id":"tiktoken@1.0.11","gitHead":"74c147e19584a3a1acea0c8e0da4d39415cd33e0","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_nodeVersion":"18.18.2","_npmVersion":"9.8.1","dist":{"shasum":"a2d1926133a5c0ff21fec732ef6af49c08f332b6","size":6030803,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.11.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.11.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.11_1700089473426_0.5899251173705369"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2023-11-15T23:04:33.802Z","publish_time":1700089473802,"_source_registry_name":"default","_cnpm_publish_time":1700089473802},"1.0.10":{"name":"tiktoken","version":"1.0.10","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"}},"gitHead":"c86963f5222caf32dcd7b50d3137a8053d710c61","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_id":"tiktoken@1.0.10","_nodeVersion":"18.16.0","_npmVersion":"9.5.1","dist":{"shasum":"d5ade07bc19f4424c404646b606e64674b0df5ef","size":5869739,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.10.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.10.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.10_1687648882653_0.7352088497524638"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2023-06-24T23:21:22.953Z","publish_time":1687648882953,"_source_registry_name":"default","_cnpm_publish_time":1687648882953},"1.0.9":{"name":"tiktoken","version":"1.0.9","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"}},"gitHead":"71db4346ae33286adcedad491e8c7573352ad562","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_id":"tiktoken@1.0.9","_nodeVersion":"18.16.0","_npmVersion":"9.5.1","dist":{"shasum":"915969c47e460c461017899b091a6208e1d5ea8b","size":5869271,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.9.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.9.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.9_1687478034560_0.3850912256795096"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2023-06-22T23:53:54.867Z","publish_time":1687478034867,"_source_registry_name":"default","_cnpm_publish_time":1687478034867},"1.0.8":{"name":"tiktoken","version":"1.0.8","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist","access":"public"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"}},"readmeFilename":"README.md","gitHead":"0d7e0512565993d1109c88568182f1d5a55ce6c2","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_id":"tiktoken@1.0.8","_nodeVersion":"19.8.1","_npmVersion":"9.6.2","dist":{"shasum":"0a8513f4cadec57aa9bb7bf0649bb9ef5dd19431","size":5870377,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.8.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.8.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.8_1686862671778_0.4023858197879555"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2023-06-15T20:57:52.006Z","publish_time":1686862672006,"_source_registry_name":"default","_cnpm_publish_time":1686862672006},"1.0.7":{"name":"tiktoken","version":"1.0.7","description":"JS/WASM bindings for tiktoken","license":"MIT","repository":{"type":"git","url":"git+https://github.com/dqbd/tiktoken.git"},"dependencies":{},"publishConfig":{"directory":"./dist"},"main":"tiktoken.cjs","types":"tiktoken.d.ts","exports":{".":{"types":"./tiktoken.d.ts","edge-light":"./tiktoken.js","node":"./tiktoken.cjs","default":"./tiktoken.js"},"./init":{"types":"./init.d.ts","edge-light":"./init.js","node":"./init.cjs","default":"./init.js"},"./load":{"types":"./load.d.ts","edge-light":"./load.js","node":"./load.cjs","default":"./load.js"},"./tiktoken_bg.wasm":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm"},"./tiktoken_bg.wasm?module":{"types":"./tiktoken_bg.wasm.d.ts","default":"./tiktoken_bg.wasm?module"},"./lite":{"types":"./lite/tiktoken.d.ts","edge-light":"./lite/tiktoken.js","node":"./lite/tiktoken.cjs","default":"./lite/tiktoken.js"},"./lite/init":{"types":"./lite/init.d.ts","edge-light":"./lite/init.js","node":"./lite/init.cjs","default":"./lite/init.js"},"./lite/load":{"types":"./lite/load.d.ts","edge-light":"./lite/load.js","node":"./lite/load.cjs","default":"./lite/load.js"},"./lite/tiktoken_bg.wasm":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm"},"./lite/tiktoken_bg.wasm?module":{"types":"./lite/tiktoken_bg.wasm.d.ts","default":"./lite/tiktoken_bg.wasm?module"},"./model_to_encoding.json":"./model_to_encoding.json","./registry.json":"./registry.json","./encoders/gpt2.json":"./encoders/gpt2.json","./encoders/gpt2":{"types":"./encoders/gpt2.d.ts","edge-light":"./encoders/gpt2.js","node":"./encoders/gpt2.cjs","default":"./encoders/gpt2.js"},"./encoders/r50k_base.json":"./encoders/r50k_base.json","./encoders/r50k_base":{"types":"./encoders/r50k_base.d.ts","edge-light":"./encoders/r50k_base.js","node":"./encoders/r50k_base.cjs","default":"./encoders/r50k_base.js"},"./encoders/p50k_base.json":"./encoders/p50k_base.json","./encoders/p50k_base":{"types":"./encoders/p50k_base.d.ts","edge-light":"./encoders/p50k_base.js","node":"./encoders/p50k_base.cjs","default":"./encoders/p50k_base.js"},"./encoders/p50k_edit.json":"./encoders/p50k_edit.json","./encoders/p50k_edit":{"types":"./encoders/p50k_edit.d.ts","edge-light":"./encoders/p50k_edit.js","node":"./encoders/p50k_edit.cjs","default":"./encoders/p50k_edit.js"},"./encoders/cl100k_base.json":"./encoders/cl100k_base.json","./encoders/cl100k_base":{"types":"./encoders/cl100k_base.d.ts","edge-light":"./encoders/cl100k_base.js","node":"./encoders/cl100k_base.cjs","default":"./encoders/cl100k_base.js"}},"readmeFilename":"README.md","gitHead":"7c6a304792c8612639bf5b79b328060656be2d64","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"homepage":"https://github.com/dqbd/tiktoken#readme","_id":"tiktoken@1.0.7","_nodeVersion":"19.8.1","_npmVersion":"9.6.2","dist":{"shasum":"09622011818248d4063d5e32d0907620235c615b","size":5823144,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.7.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.7.tgz"},"_npmUser":{"name":"davidduong","email":"david@duong.cz"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.7_1683978157824_0.08859956503741206"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2023-05-13T11:42:38.116Z","publish_time":1683978158116,"_source_registry_name":"default","_cnpm_publish_time":1683978158116},"0.0.1":{"name":"tiktoken","version":"0.0.1","description":"","main":"index.js","scripts":{"test":"echo \"Error: no test specified\" && exit 1"},"author":"","license":"ISC","_id":"tiktoken@0.0.1","_nodeVersion":"18.13.0","_npmVersion":"8.19.3","dist":{"shasum":"9004cadce30cd8eec381ab0c95f7c68ecc298c3e","size":236,"noattachment":false,"key":"/tiktoken/-/tiktoken-0.0.1.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-0.0.1.tgz"},"_npmUser":{"name":"gmpetrov","email":"georges.ptrv@gmail.com"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_0.0.1_1676320936925_0.19552183194030803"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2023-02-13T20:42:17.095Z","publish_time":1676320937095,"_cnpm_publish_time":1676320937095},"1.0.0":{"name":"tiktoken","version":"1.0.0","description":"","main":"index.js","scripts":{"test":"echo \"Error: no test specified\" && exit 1"},"author":"","license":"ISC","_id":"tiktoken@1.0.0","_nodeVersion":"18.13.0","_npmVersion":"8.19.3","dist":{"shasum":"cae00a33a01b345c440cdc910eec38218ace80b1","size":236,"noattachment":false,"key":"/tiktoken/-/tiktoken-1.0.0.tgz","tarball":"http://registry.cnpm.dingdandao.com/tiktoken/download/tiktoken-1.0.0.tgz"},"_npmUser":{"name":"gmpetrov","email":"georges.ptrv@gmail.com"},"directories":{},"maintainers":[{"name":"davidduong","email":""}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tiktoken_1.0.0_1676320840071_0.5646188401331305"},"_hasShrinkwrap":false,"_cnpmcore_publish_time":"2023-02-13T20:40:40.212Z","publish_time":1676320840212,"_cnpm_publish_time":1676320840212}},"readme":"# ⏳ tiktoken\n\ntiktoken is a [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with\nOpenAI's models, forked from the original tiktoken library to provide JS/WASM bindings for NodeJS and other JS runtimes.\n\nThis repository contains the following packages:\n\n- `tiktoken` (formally hosted at `@dqbd/tiktoken`): WASM bindings for the original Python library, providing full 1-to-1 feature parity.\n- `js-tiktoken`: Pure JavaScript port of the original library with the core functionality, suitable for environments where WASM is not well supported or not desired (such as edge runtimes). \n\nDocumentation for `js-tiktoken` can be found in [here](https://github.com/dqbd/tiktoken/blob/main/js/README.md). Documentation for the `tiktoken` can be found here below.\n\nThe WASM version of `tiktoken` can be installed from NPM:\n\n```\nnpm install tiktoken\n```\n\n## Usage\n\nBasic usage follows, which includes all the OpenAI encoders and ranks:\n\n```typescript\nimport assert from \"node:assert\";\nimport { get_encoding, encoding_for_model } from \"tiktoken\";\n\nconst enc = get_encoding(\"gpt2\");\nassert(\n  new TextDecoder().decode(enc.decode(enc.encode(\"hello world\"))) ===\n    \"hello world\"\n);\n\n// To get the tokeniser corresponding to a specific model in the OpenAI API:\nconst enc = encoding_for_model(\"text-davinci-003\");\n\n// Extend existing encoding with custom special tokens\nconst enc = encoding_for_model(\"gpt2\", {\n  \"<|im_start|>\": 100264,\n  \"<|im_end|>\": 100265,\n});\n\n// don't forget to free the encoder after it is not used\nenc.free();\n```\n\nIn constrained environments (eg. Edge Runtime, Cloudflare Workers), where you don't want to load all the encoders at once, you can use the lightweight WASM binary via `tiktoken/lite`.\n\n```typescript\nconst { Tiktoken } = require(\"tiktoken/lite\");\nconst cl100k_base = require(\"tiktoken/encoders/cl100k_base.json\");\n\nconst encoding = new Tiktoken(\n  cl100k_base.bpe_ranks,\n  cl100k_base.special_tokens,\n  cl100k_base.pat_str\n);\nconst tokens = encoding.encode(\"hello world\");\nencoding.free();\n```\n\nIf you want to fetch the latest ranks, use the `load` function:\n\n```typescript\nconst { Tiktoken } = require(\"tiktoken/lite\");\nconst { load } = require(\"tiktoken/load\");\nconst registry = require(\"tiktoken/registry.json\");\nconst models = require(\"tiktoken/model_to_encoding.json\");\n\nasync function main() {\n  const model = await load(registry[models[\"gpt-3.5-turbo\"]]);\n  const encoder = new Tiktoken(\n    model.bpe_ranks,\n    model.special_tokens,\n    model.pat_str\n  );\n  const tokens = encoder.encode(\"hello world\");\n  encoder.free();\n}\n\nmain();\n```\n\nIf desired, you can create a Tiktoken instance directly with custom ranks, special tokens and regex pattern:\n\n```typescript\nimport { Tiktoken } from \"../pkg\";\nimport { readFileSync } from \"fs\";\n\nconst encoder = new Tiktoken(\n  readFileSync(\"./ranks/gpt2.tiktoken\").toString(\"utf-8\"),\n  { \"<|endoftext|>\": 50256, \"<|im_start|>\": 100264, \"<|im_end|>\": 100265 },\n  \"'s|'t|'re|'ve|'m|'ll|'d| ?\\\\p{L}+| ?\\\\p{N}+| ?[^\\\\s\\\\p{L}\\\\p{N}]+|\\\\s+(?!\\\\S)|\\\\s+\"\n);\n```\n\nFinally, you can a custom `init` function to override the WASM initialization logic for non-Node environments. This is useful if you are using a bundler that does not support WASM ESM integration.\n\n```typescript\nimport { get_encoding, init } from \"tiktoken/init\";\n\nasync function main() {\n  const wasm = \"...\"; // fetch the WASM binary somehow\n  await init((imports) => WebAssembly.instantiate(wasm, imports));\n\n  const encoding = get_encoding(\"cl100k_base\");\n  const tokens = encoding.encode(\"hello world\");\n  encoding.free();\n}\n\nmain();\n```\n\n## Compatibility\n\nAs this is a WASM library, there might be some issues with specific runtimes. If you encounter any issues, please open an issue.\n\n| Runtime                      | Status | Notes                                                                                      |\n| ---------------------------- | ------ | ------------------------------------------------------------------------------------------ |\n| Node.js                      | ✅     |                                                                                            |\n| Bun                          | ✅     |                                                                                            |\n| Vite                         | ✅     | See [here](#vite) for notes                                                                |\n| Next.js                      | ✅     | See [here](#nextjs) for notes                                                              |\n| Create React App (via Craco) | ✅     | See [here](#create-react-app) for notes                                                    |\n| Vercel Edge Runtime          | ✅     | See [here](#vercel-edge-runtime) for notes                                                 |\n| Cloudflare Workers           | ✅     | See [here](#cloudflare-workers) for notes                                                  |\n| Electron                     | ✅     | See [here](#electron) for notes                                                            |\n| Deno                         | ❌     | Currently unsupported (see [dqbd/tiktoken#22](https://github.com/dqbd/tiktoken/issues/22)) |\n| Svelte + Cloudflare Workers  | ❌     | Currently unsupported (see [dqbd/tiktoken#37](https://github.com/dqbd/tiktoken/issues/37)) |\n\nFor unsupported runtimes, consider using [`js-tiktoken`](https://www.npmjs.com/package/js-tiktoken), which is a pure JS implementation of the tokeniser.\n\n### [Vite](#vite)\n\nIf you are using Vite, you will need to add both the `vite-plugin-wasm` and `vite-plugin-top-level-await`. Add the following to your `vite.config.js`:\n\n```js\nimport wasm from \"vite-plugin-wasm\";\nimport topLevelAwait from \"vite-plugin-top-level-await\";\nimport { defineConfig } from \"vite\";\n\nexport default defineConfig({\n  plugins: [wasm(), topLevelAwait()],\n});\n```\n\n### [Next.js](#nextjs)\n\nBoth API routes and `/pages` are supported with the following `next.config.js` configuration.\n\n```typescript\n// next.config.json\nconst config = {\n  webpack(config, { isServer, dev }) {\n    config.experiments = {\n      asyncWebAssembly: true,\n      layers: true,\n    };\n\n    return config;\n  },\n};\n```\n\nUsage in pages:\n\n```tsx\nimport { get_encoding } from \"tiktoken\";\nimport { useState } from \"react\";\n\nconst encoding = get_encoding(\"cl100k_base\");\n\nexport default function Home() {\n  const [input, setInput] = useState(\"hello world\");\n  const tokens = encoding.encode(input);\n\n  return (\n    <div>\n      <input\n        type=\"text\"\n        value={input}\n        onChange={(e) => setInput(e.target.value)}\n      />\n      <div>{tokens.toString()}</div>\n    </div>\n  );\n}\n```\n\nUsage in API routes:\n\n```typescript\nimport { get_encoding } from \"tiktoken\";\nimport { NextApiRequest, NextApiResponse } from \"next\";\n\nexport default function handler(req: NextApiRequest, res: NextApiResponse) {\n  const encoding = get_encoding(\"cl100k_base\");\n  const tokens = encoding.encode(\"hello world\");\n  encoding.free();\n  return res.status(200).json({ tokens });\n}\n```\n\n### [Create React App](#create-react-app)\n\nBy default, the Webpack configugration found in Create React App does not support WASM ESM modules. To add support, please do the following:\n\n1. Swap `react-scripts` with `craco`, using the guide found here: https://craco.js.org/docs/getting-started/.\n2. Add the following to `craco.config.js`:\n\n```js\nmodule.exports = {\n  webpack: {\n    configure: (config) => {\n      config.experiments = {\n        asyncWebAssembly: true,\n        layers: true,\n      };\n\n      // turn off static file serving of WASM files\n      // we need to let Webpack handle WASM import\n      config.module.rules\n        .find((i) => \"oneOf\" in i)\n        .oneOf.find((i) => i.type === \"asset/resource\")\n        .exclude.push(/\\.wasm$/);\n\n      return config;\n    },\n  },\n};\n```\n\n### [Vercel Edge Runtime](#vercel-edge-runtime)\n\nVercel Edge Runtime does support WASM modules by adding a `?module` suffix. Initialize the encoder with the following snippet:\n\n```typescript\n// @ts-expect-error\nimport wasm from \"tiktoken/lite/tiktoken_bg.wasm?module\";\nimport model from \"tiktoken/encoders/cl100k_base.json\";\nimport { init, Tiktoken } from \"tiktoken/lite/init\";\n\nexport const config = { runtime: \"edge\" };\n\nexport default async function (req: Request) {\n  await init((imports) => WebAssembly.instantiate(wasm, imports));\n\n  const encoding = new Tiktoken(\n    model.bpe_ranks,\n    model.special_tokens,\n    model.pat_str\n  );\n\n  const tokens = encoding.encode(\"hello world\");\n  encoding.free();\n\n  return new Response(`${tokens}`);\n}\n```\n\n### [Cloudflare Workers](#cloudflare-workers)\n\nSimilar to Vercel Edge Runtime, Cloudflare Workers must import the WASM binary file manually and use the `tiktoken/lite` version to fit the 1 MB limit. However, users need to point directly at the WASM binary via a relative path (including `./node_modules/`).\n\nAdd the following rule to the `wrangler.toml` to upload WASM during build:\n\n```toml\n[[rules]]\nglobs = [\"**/*.wasm\"]\ntype = \"CompiledWasm\"\n```\n\nInitialize the encoder with the following snippet:\n\n```javascript\nimport { init, Tiktoken } from \"tiktoken/lite/init\";\nimport wasm from \"./node_modules/tiktoken/lite/tiktoken_bg.wasm\";\nimport model from \"tiktoken/encoders/cl100k_base.json\";\n\nexport default {\n  async fetch() {\n    await init((imports) => WebAssembly.instantiate(wasm, imports));\n    const encoder = new Tiktoken(\n      model.bpe_ranks,\n      model.special_tokens,\n      model.pat_str\n    );\n    const tokens = encoder.encode(\"test\");\n    encoder.free();\n    return new Response(`${tokens}`);\n  },\n};\n```\n\n### [Electron](#electron)\n\nTo use tiktoken in your Electron main process, you need to make sure the WASM binary gets copied into your application package.\n\nAssuming a setup with [Electron Forge](https://www.electronforge.io) and [`@electron-forge/plugin-webpack`](https://www.npmjs.com/package/@electron-forge/plugin-webpack), add the following to your `webpack.main.config.js`:\n\n```javascript\nconst CopyPlugin = require(\"copy-webpack-plugin\");\n\nmodule.exports = {\n  // ...\n  plugins: [\n    new CopyPlugin({\n      patterns: [\n        { from: \"./node_modules/tiktoken/tiktoken_bg.wasm\" },\n      ],\n    }),\n  ],\n};\n```\n\n## Development\n\nTo build the `tiktoken` library, make sure to have:\n- Rust and [`wasm-pack`](https://github.com/rustwasm/wasm-pack) installed.\n- Node.js 18+ is required to build the JS bindings and fetch the latest encoder ranks via `fetch`.\n\nInstall all the dev-dependencies with `yarn install` and build both WASM binary and JS bindings with `yarn build`.\n\n## Acknowledgements\n\n- https://github.com/zurawiki/tiktoken-rs\n","_attachments":{},"homepage":"https://github.com/dqbd/tiktoken#readme","bugs":{"url":"https://github.com/dqbd/tiktoken/issues"},"license":"MIT"}