web 3d图形渲染器
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

669 lines
20 KiB

  1. # cacache [![npm version](https://img.shields.io/npm/v/cacache.svg)](https://npm.im/cacache) [![license](https://img.shields.io/npm/l/cacache.svg)](https://npm.im/cacache) [![Travis](https://img.shields.io/travis/npm/cacache.svg)](https://travis-ci.org/npm/cacache) [![AppVeyor](https://ci.appveyor.com/api/projects/status/github/npm/cacache?svg=true)](https://ci.appveyor.com/project/npm/cacache) [![Coverage Status](https://coveralls.io/repos/github/npm/cacache/badge.svg?branch=latest)](https://coveralls.io/github/npm/cacache?branch=latest)
  2. [`cacache`](https://github.com/npm/cacache) is a Node.js library for managing
  3. local key and content address caches. It's really fast, really good at
  4. concurrency, and it will never give you corrupted data, even if cache files
  5. get corrupted or manipulated.
  6. On systems that support user and group settings on files, cacache will
  7. match the `uid` and `gid` values to the folder where the cache lives, even
  8. when running as `root`.
  9. It was written to be used as [npm](https://npm.im)'s local cache, but can
  10. just as easily be used on its own.
  11. ## Install
  12. `$ npm install --save cacache`
  13. ## Table of Contents
  14. * [Example](#example)
  15. * [Features](#features)
  16. * [Contributing](#contributing)
  17. * [API](#api)
  18. * [Using localized APIs](#localized-api)
  19. * Reading
  20. * [`ls`](#ls)
  21. * [`ls.stream`](#ls-stream)
  22. * [`get`](#get-data)
  23. * [`get.stream`](#get-stream)
  24. * [`get.info`](#get-info)
  25. * [`get.hasContent`](#get-hasContent)
  26. * Writing
  27. * [`put`](#put-data)
  28. * [`put.stream`](#put-stream)
  29. * [`rm.all`](#rm-all)
  30. * [`rm.entry`](#rm-entry)
  31. * [`rm.content`](#rm-content)
  32. * Utilities
  33. * [`clearMemoized`](#clear-memoized)
  34. * [`tmp.mkdir`](#tmp-mkdir)
  35. * [`tmp.withTmp`](#with-tmp)
  36. * Integrity
  37. * [Subresource Integrity](#integrity)
  38. * [`verify`](#verify)
  39. * [`verify.lastRun`](#verify-last-run)
  40. ### Example
  41. ```javascript
  42. const cacache = require('cacache')
  43. const fs = require('fs')
  44. const tarball = '/path/to/mytar.tgz'
  45. const cachePath = '/tmp/my-toy-cache'
  46. const key = 'my-unique-key-1234'
  47. // Cache it! Use `cachePath` as the root of the content cache
  48. cacache.put(cachePath, key, '10293801983029384').then(integrity => {
  49. console.log(`Saved content to ${cachePath}.`)
  50. })
  51. const destination = '/tmp/mytar.tgz'
  52. // Copy the contents out of the cache and into their destination!
  53. // But this time, use stream instead!
  54. cacache.get.stream(
  55. cachePath, key
  56. ).pipe(
  57. fs.createWriteStream(destination)
  58. ).on('finish', () => {
  59. console.log('done extracting!')
  60. })
  61. // The same thing, but skip the key index.
  62. cacache.get.byDigest(cachePath, integrityHash).then(data => {
  63. fs.writeFile(destination, data, err => {
  64. console.log('tarball data fetched based on its sha512sum and written out!')
  65. })
  66. })
  67. ```
  68. ### Features
  69. * Extraction by key or by content address (shasum, etc)
  70. * [Subresource Integrity](#integrity) web standard support
  71. * Multi-hash support - safely host sha1, sha512, etc, in a single cache
  72. * Automatic content deduplication
  73. * Fault tolerance (immune to corruption, partial writes, process races, etc)
  74. * Consistency guarantees on read and write (full data verification)
  75. * Lockless, high-concurrency cache access
  76. * Streaming support
  77. * Promise support
  78. * Fast -- sub-millisecond reads and writes including verification
  79. * Arbitrary metadata storage
  80. * Garbage collection and additional offline verification
  81. * Thorough test coverage
  82. * There's probably a bloom filter in there somewhere. Those are cool, right? 🤔
  83. ### Contributing
  84. The cacache team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The [Contributor Guide](CONTRIBUTING.md) has all the information you need for everything from reporting bugs to contributing entire new features. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.
  85. All participants and maintainers in this project are expected to follow [Code of Conduct](CODE_OF_CONDUCT.md), and just generally be excellent to each other.
  86. Please refer to the [Changelog](CHANGELOG.md) for project history details, too.
  87. Happy hacking!
  88. ### API
  89. #### <a name="ls"></a> `> cacache.ls(cache) -> Promise<Object>`
  90. Lists info for all entries currently in the cache as a single large object. Each
  91. entry in the object will be keyed by the unique index key, with corresponding
  92. [`get.info`](#get-info) objects as the values.
  93. ##### Example
  94. ```javascript
  95. cacache.ls(cachePath).then(console.log)
  96. // Output
  97. {
  98. 'my-thing': {
  99. key: 'my-thing',
  100. integrity: 'sha512-BaSe64/EnCoDED+HAsh=='
  101. path: '.testcache/content/deadbeef', // joined with `cachePath`
  102. time: 12345698490,
  103. size: 4023948,
  104. metadata: {
  105. name: 'blah',
  106. version: '1.2.3',
  107. description: 'this was once a package but now it is my-thing'
  108. }
  109. },
  110. 'other-thing': {
  111. key: 'other-thing',
  112. integrity: 'sha1-ANothER+hasH=',
  113. path: '.testcache/content/bada55',
  114. time: 11992309289,
  115. size: 111112
  116. }
  117. }
  118. ```
  119. #### <a name="ls-stream"></a> `> cacache.ls.stream(cache) -> Readable`
  120. Lists info for all entries currently in the cache as a single large object.
  121. This works just like [`ls`](#ls), except [`get.info`](#get-info) entries are
  122. returned as `'data'` events on the returned stream.
  123. ##### Example
  124. ```javascript
  125. cacache.ls.stream(cachePath).on('data', console.log)
  126. // Output
  127. {
  128. key: 'my-thing',
  129. integrity: 'sha512-BaSe64HaSh',
  130. path: '.testcache/content/deadbeef', // joined with `cachePath`
  131. time: 12345698490,
  132. size: 13423,
  133. metadata: {
  134. name: 'blah',
  135. version: '1.2.3',
  136. description: 'this was once a package but now it is my-thing'
  137. }
  138. }
  139. {
  140. key: 'other-thing',
  141. integrity: 'whirlpool-WoWSoMuchSupport',
  142. path: '.testcache/content/bada55',
  143. time: 11992309289,
  144. size: 498023984029
  145. }
  146. {
  147. ...
  148. }
  149. ```
  150. #### <a name="get-data"></a> `> cacache.get(cache, key, [opts]) -> Promise({data, metadata, integrity})`
  151. Returns an object with the cached data, digest, and metadata identified by
  152. `key`. The `data` property of this object will be a `Buffer` instance that
  153. presumably holds some data that means something to you. I'm sure you know what
  154. to do with it! cacache just won't care.
  155. `integrity` is a [Subresource
  156. Integrity](#integrity)
  157. string. That is, a string that can be used to verify `data`, which looks like
  158. `<hash-algorithm>-<base64-integrity-hash>`.
  159. If there is no content identified by `key`, or if the locally-stored data does
  160. not pass the validity checksum, the promise will be rejected.
  161. A sub-function, `get.byDigest` may be used for identical behavior, except lookup
  162. will happen by integrity hash, bypassing the index entirely. This version of the
  163. function *only* returns `data` itself, without any wrapper.
  164. See: [options](#get-options)
  165. ##### Note
  166. This function loads the entire cache entry into memory before returning it. If
  167. you're dealing with Very Large data, consider using [`get.stream`](#get-stream)
  168. instead.
  169. ##### Example
  170. ```javascript
  171. // Look up by key
  172. cache.get(cachePath, 'my-thing').then(console.log)
  173. // Output:
  174. {
  175. metadata: {
  176. thingName: 'my'
  177. },
  178. integrity: 'sha512-BaSe64HaSh',
  179. data: Buffer#<deadbeef>,
  180. size: 9320
  181. }
  182. // Look up by digest
  183. cache.get.byDigest(cachePath, 'sha512-BaSe64HaSh').then(console.log)
  184. // Output:
  185. Buffer#<deadbeef>
  186. ```
  187. #### <a name="get-stream"></a> `> cacache.get.stream(cache, key, [opts]) -> Readable`
  188. Returns a [Readable Stream](https://nodejs.org/api/stream.html#stream_readable_streams) of the cached data identified by `key`.
  189. If there is no content identified by `key`, or if the locally-stored data does
  190. not pass the validity checksum, an error will be emitted.
  191. `metadata` and `integrity` events will be emitted before the stream closes, if
  192. you need to collect that extra data about the cached entry.
  193. A sub-function, `get.stream.byDigest` may be used for identical behavior,
  194. except lookup will happen by integrity hash, bypassing the index entirely. This
  195. version does not emit the `metadata` and `integrity` events at all.
  196. See: [options](#get-options)
  197. ##### Example
  198. ```javascript
  199. // Look up by key
  200. cache.get.stream(
  201. cachePath, 'my-thing'
  202. ).on('metadata', metadata => {
  203. console.log('metadata:', metadata)
  204. }).on('integrity', integrity => {
  205. console.log('integrity:', integrity)
  206. }).pipe(
  207. fs.createWriteStream('./x.tgz')
  208. )
  209. // Outputs:
  210. metadata: { ... }
  211. integrity: 'sha512-SoMeDIGest+64=='
  212. // Look up by digest
  213. cache.get.stream.byDigest(
  214. cachePath, 'sha512-SoMeDIGest+64=='
  215. ).pipe(
  216. fs.createWriteStream('./x.tgz')
  217. )
  218. ```
  219. #### <a name="get-info"></a> `> cacache.get.info(cache, key) -> Promise`
  220. Looks up `key` in the cache index, returning information about the entry if
  221. one exists.
  222. ##### Fields
  223. * `key` - Key the entry was looked up under. Matches the `key` argument.
  224. * `integrity` - [Subresource Integrity hash](#integrity) for the content this entry refers to.
  225. * `path` - Filesystem path where content is stored, joined with `cache` argument.
  226. * `time` - Timestamp the entry was first added on.
  227. * `metadata` - User-assigned metadata associated with the entry/content.
  228. ##### Example
  229. ```javascript
  230. cacache.get.info(cachePath, 'my-thing').then(console.log)
  231. // Output
  232. {
  233. key: 'my-thing',
  234. integrity: 'sha256-MUSTVERIFY+ALL/THINGS=='
  235. path: '.testcache/content/deadbeef',
  236. time: 12345698490,
  237. size: 849234,
  238. metadata: {
  239. name: 'blah',
  240. version: '1.2.3',
  241. description: 'this was once a package but now it is my-thing'
  242. }
  243. }
  244. ```
  245. #### <a name="get-hasContent"></a> `> cacache.get.hasContent(cache, integrity) -> Promise`
  246. Looks up a [Subresource Integrity hash](#integrity) in the cache. If content
  247. exists for this `integrity`, it will return an object, with the specific single integrity hash
  248. that was found in `sri` key, and the size of the found content as `size`. If no content exists for this integrity, it will return `false`.
  249. ##### Example
  250. ```javascript
  251. cacache.get.hasContent(cachePath, 'sha256-MUSTVERIFY+ALL/THINGS==').then(console.log)
  252. // Output
  253. {
  254. sri: {
  255. source: 'sha256-MUSTVERIFY+ALL/THINGS==',
  256. algorithm: 'sha256',
  257. digest: 'MUSTVERIFY+ALL/THINGS==',
  258. options: []
  259. },
  260. size: 9001
  261. }
  262. cacache.get.hasContent(cachePath, 'sha521-NOT+IN/CACHE==').then(console.log)
  263. // Output
  264. false
  265. ```
  266. ##### <a name="get-options"></a> Options
  267. ##### `opts.integrity`
  268. If present, the pre-calculated digest for the inserted content. If this option
  269. is provided and does not match the post-insertion digest, insertion will fail
  270. with an `EINTEGRITY` error.
  271. ##### `opts.memoize`
  272. Default: null
  273. If explicitly truthy, cacache will read from memory and memoize data on bulk read. If `false`, cacache will read from disk data. Reader functions by default read from in-memory cache.
  274. ##### `opts.size`
  275. If provided, the data stream will be verified to check that enough data was
  276. passed through. If there's more or less data than expected, insertion will fail
  277. with an `EBADSIZE` error.
  278. #### <a name="put-data"></a> `> cacache.put(cache, key, data, [opts]) -> Promise`
  279. Inserts data passed to it into the cache. The returned Promise resolves with a
  280. digest (generated according to [`opts.algorithms`](#optsalgorithms)) after the
  281. cache entry has been successfully written.
  282. See: [options](#put-options)
  283. ##### Example
  284. ```javascript
  285. fetch(
  286. 'https://registry.npmjs.org/cacache/-/cacache-1.0.0.tgz'
  287. ).then(data => {
  288. return cacache.put(cachePath, 'registry.npmjs.org|cacache@1.0.0', data)
  289. }).then(integrity => {
  290. console.log('integrity hash is', integrity)
  291. })
  292. ```
  293. #### <a name="put-stream"></a> `> cacache.put.stream(cache, key, [opts]) -> Writable`
  294. Returns a [Writable
  295. Stream](https://nodejs.org/api/stream.html#stream_writable_streams) that inserts
  296. data written to it into the cache. Emits an `integrity` event with the digest of
  297. written contents when it succeeds.
  298. See: [options](#put-options)
  299. ##### Example
  300. ```javascript
  301. request.get(
  302. 'https://registry.npmjs.org/cacache/-/cacache-1.0.0.tgz'
  303. ).pipe(
  304. cacache.put.stream(
  305. cachePath, 'registry.npmjs.org|cacache@1.0.0'
  306. ).on('integrity', d => console.log(`integrity digest is ${d}`))
  307. )
  308. ```
  309. ##### <a name="put-options"></a> Options
  310. ##### `opts.metadata`
  311. Arbitrary metadata to be attached to the inserted key.
  312. ##### `opts.size`
  313. If provided, the data stream will be verified to check that enough data was
  314. passed through. If there's more or less data than expected, insertion will fail
  315. with an `EBADSIZE` error.
  316. ##### `opts.integrity`
  317. If present, the pre-calculated digest for the inserted content. If this option
  318. is provided and does not match the post-insertion digest, insertion will fail
  319. with an `EINTEGRITY` error.
  320. `algorithms` has no effect if this option is present.
  321. ##### `opts.algorithms`
  322. Default: ['sha512']
  323. Hashing algorithms to use when calculating the [subresource integrity
  324. digest](#integrity)
  325. for inserted data. Can use any algorithm listed in `crypto.getHashes()` or
  326. `'omakase'`/`'お任せします'` to pick a random hash algorithm on each insertion. You
  327. may also use any anagram of `'modnar'` to use this feature.
  328. Currently only supports one algorithm at a time (i.e., an array length of
  329. exactly `1`). Has no effect if `opts.integrity` is present.
  330. ##### `opts.memoize`
  331. Default: null
  332. If provided, cacache will memoize the given cache insertion in memory, bypassing
  333. any filesystem checks for that key or digest in future cache fetches. Nothing
  334. will be written to the in-memory cache unless this option is explicitly truthy.
  335. If `opts.memoize` is an object or a `Map`-like (that is, an object with `get`
  336. and `set` methods), it will be written to instead of the global memoization
  337. cache.
  338. Reading from disk data can be forced by explicitly passing `memoize: false` to
  339. the reader functions, but their default will be to read from memory.
  340. ##### `opts.tmpPrefix`
  341. Default: null
  342. Prefix to append on the temporary directory name inside the cache's tmp dir.
  343. #### <a name="rm-all"></a> `> cacache.rm.all(cache) -> Promise`
  344. Clears the entire cache. Mainly by blowing away the cache directory itself.
  345. ##### Example
  346. ```javascript
  347. cacache.rm.all(cachePath).then(() => {
  348. console.log('THE APOCALYPSE IS UPON US 😱')
  349. })
  350. ```
  351. #### <a name="rm-entry"></a> `> cacache.rm.entry(cache, key) -> Promise`
  352. Alias: `cacache.rm`
  353. Removes the index entry for `key`. Content will still be accessible if
  354. requested directly by content address ([`get.stream.byDigest`](#get-stream)).
  355. To remove the content itself (which might still be used by other entries), use
  356. [`rm.content`](#rm-content). Or, to safely vacuum any unused content, use
  357. [`verify`](#verify).
  358. ##### Example
  359. ```javascript
  360. cacache.rm.entry(cachePath, 'my-thing').then(() => {
  361. console.log('I did not like it anyway')
  362. })
  363. ```
  364. #### <a name="rm-content"></a> `> cacache.rm.content(cache, integrity) -> Promise`
  365. Removes the content identified by `integrity`. Any index entries referring to it
  366. will not be usable again until the content is re-added to the cache with an
  367. identical digest.
  368. ##### Example
  369. ```javascript
  370. cacache.rm.content(cachePath, 'sha512-SoMeDIGest/IN+BaSE64==').then(() => {
  371. console.log('data for my-thing is gone!')
  372. })
  373. ```
  374. #### <a name="clear-memoized"></a> `> cacache.clearMemoized()`
  375. Completely resets the in-memory entry cache.
  376. #### <a name="tmp-mkdir"></a> `> tmp.mkdir(cache, opts) -> Promise<Path>`
  377. Returns a unique temporary directory inside the cache's `tmp` dir. This
  378. directory will use the same safe user assignment that all the other stuff use.
  379. Once the directory is made, it's the user's responsibility that all files
  380. within are given the appropriate `gid`/`uid` ownership settings to match
  381. the rest of the cache. If not, you can ask cacache to do it for you by
  382. calling [`tmp.fix()`](#tmp-fix), which will fix all tmp directory
  383. permissions.
  384. If you want automatic cleanup of this directory, use
  385. [`tmp.withTmp()`](#with-tpm)
  386. See: [options](#tmp-options)
  387. ##### Example
  388. ```javascript
  389. cacache.tmp.mkdir(cache).then(dir => {
  390. fs.writeFile(path.join(dir, 'blablabla'), Buffer#<1234>, ...)
  391. })
  392. ```
  393. #### <a name="tmp-fix"></a> `> tmp.fix(cache) -> Promise`
  394. Sets the `uid` and `gid` properties on all files and folders within the tmp
  395. folder to match the rest of the cache.
  396. Use this after manually writing files into [`tmp.mkdir`](#tmp-mkdir) or
  397. [`tmp.withTmp`](#with-tmp).
  398. ##### Example
  399. ```javascript
  400. cacache.tmp.mkdir(cache).then(dir => {
  401. writeFile(path.join(dir, 'file'), someData).then(() => {
  402. // make sure we didn't just put a root-owned file in the cache
  403. cacache.tmp.fix().then(() => {
  404. // all uids and gids match now
  405. })
  406. })
  407. })
  408. ```
  409. #### <a name="with-tmp"></a> `> tmp.withTmp(cache, opts, cb) -> Promise`
  410. Creates a temporary directory with [`tmp.mkdir()`](#tmp-mkdir) and calls `cb`
  411. with it. The created temporary directory will be removed when the return value
  412. of `cb()` resolves, the tmp directory will be automatically deleted once that
  413. promise completes.
  414. The same caveats apply when it comes to managing permissions for the tmp dir's
  415. contents.
  416. See: [options](#tmp-options)
  417. ##### Example
  418. ```javascript
  419. cacache.tmp.withTmp(cache, dir => {
  420. return fs.writeFileAsync(path.join(dir, 'blablabla'), Buffer#<1234>, ...)
  421. }).then(() => {
  422. // `dir` no longer exists
  423. })
  424. ```
  425. ##### <a name="tmp-options"></a> Options
  426. ##### `opts.tmpPrefix`
  427. Default: null
  428. Prefix to append on the temporary directory name inside the cache's tmp dir.
  429. #### <a name="integrity"></a> Subresource Integrity Digests
  430. For content verification and addressing, cacache uses strings following the
  431. [Subresource
  432. Integrity spec](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity).
  433. That is, any time cacache expects an `integrity` argument or option, it
  434. should be in the format `<hashAlgorithm>-<base64-hash>`.
  435. One deviation from the current spec is that cacache will support any hash
  436. algorithms supported by the underlying Node.js process. You can use
  437. `crypto.getHashes()` to see which ones you can use.
  438. ##### Generating Digests Yourself
  439. If you have an existing content shasum, they are generally formatted as a
  440. hexadecimal string (that is, a sha1 would look like:
  441. `5f5513f8822fdbe5145af33b64d8d970dcf95c6e`). In order to be compatible with
  442. cacache, you'll need to convert this to an equivalent subresource integrity
  443. string. For this example, the corresponding hash would be:
  444. `sha1-X1UT+IIv2+UUWvM7ZNjZcNz5XG4=`.
  445. If you want to generate an integrity string yourself for existing data, you can
  446. use something like this:
  447. ```javascript
  448. const crypto = require('crypto')
  449. const hashAlgorithm = 'sha512'
  450. const data = 'foobarbaz'
  451. const integrity = (
  452. hashAlgorithm +
  453. '-' +
  454. crypto.createHash(hashAlgorithm).update(data).digest('base64')
  455. )
  456. ```
  457. You can also use [`ssri`](https://npm.im/ssri) to have a richer set of functionality
  458. around SRI strings, including generation, parsing, and translating from existing
  459. hex-formatted strings.
  460. #### <a name="verify"></a> `> cacache.verify(cache, opts) -> Promise`
  461. Checks out and fixes up your cache:
  462. * Cleans up corrupted or invalid index entries.
  463. * Custom entry filtering options.
  464. * Garbage collects any content entries not referenced by the index.
  465. * Checks integrity for all content entries and removes invalid content.
  466. * Fixes cache ownership.
  467. * Removes the `tmp` directory in the cache and all its contents.
  468. When it's done, it'll return an object with various stats about the verification
  469. process, including amount of storage reclaimed, number of valid entries, number
  470. of entries removed, etc.
  471. ##### <a name="verify-options"></a> Options
  472. ##### `opts.concurrency`
  473. Default: 20
  474. Number of concurrently read files in the filesystem while doing clean up.
  475. ##### `opts.filter`
  476. Receives a formatted entry. Return false to remove it.
  477. Note: might be called more than once on the same entry.
  478. ##### `opts.log`
  479. Custom logger function:
  480. ```
  481. log: { silly () {} }
  482. log.silly('verify', 'verifying cache at', cache)
  483. ```
  484. ##### Example
  485. ```sh
  486. echo somegarbage >> $CACHEPATH/content/deadbeef
  487. ```
  488. ```javascript
  489. cacache.verify(cachePath).then(stats => {
  490. // deadbeef collected, because of invalid checksum.
  491. console.log('cache is much nicer now! stats:', stats)
  492. })
  493. ```
  494. #### <a name="verify-last-run"></a> `> cacache.verify.lastRun(cache) -> Promise`
  495. Returns a `Date` representing the last time `cacache.verify` was run on `cache`.
  496. ##### Example
  497. ```javascript
  498. cacache.verify(cachePath).then(() => {
  499. cacache.verify.lastRun(cachePath).then(lastTime => {
  500. console.log('cacache.verify was last called on' + lastTime)
  501. })
  502. })
  503. ```