web 3d图形渲染器
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

613 lines
21 KiB

  1. # minipass
  2. A _very_ minimal implementation of a [PassThrough
  3. stream](https://nodejs.org/api/stream.html#stream_class_stream_passthrough)
  4. [It's very
  5. fast](https://docs.google.com/spreadsheets/d/1oObKSrVwLX_7Ut4Z6g3fZW-AX1j1-k6w-cDsrkaSbHM/edit#gid=0)
  6. for objects, strings, and buffers.
  7. Supports pipe()ing (including multi-pipe() and backpressure transmission),
  8. buffering data until either a `data` event handler or `pipe()` is added (so
  9. you don't lose the first chunk), and most other cases where PassThrough is
  10. a good idea.
  11. There is a `read()` method, but it's much more efficient to consume data
  12. from this stream via `'data'` events or by calling `pipe()` into some other
  13. stream. Calling `read()` requires the buffer to be flattened in some
  14. cases, which requires copying memory.
  15. There is also no `unpipe()` method. Once you start piping, there is no
  16. stopping it!
  17. If you set `objectMode: true` in the options, then whatever is written will
  18. be emitted. Otherwise, it'll do a minimal amount of Buffer copying to
  19. ensure proper Streams semantics when `read(n)` is called.
  20. `objectMode` can also be set by doing `stream.objectMode = true`, or by
  21. writing any non-string/non-buffer data. `objectMode` cannot be set to
  22. false once it is set.
  23. This is not a `through` or `through2` stream. It doesn't transform the
  24. data, it just passes it right through. If you want to transform the data,
  25. extend the class, and override the `write()` method. Once you're done
  26. transforming the data however you want, call `super.write()` with the
  27. transform output.
  28. For some examples of streams that extend Minipass in various ways, check
  29. out:
  30. - [minizlib](http://npm.im/minizlib)
  31. - [fs-minipass](http://npm.im/fs-minipass)
  32. - [tar](http://npm.im/tar)
  33. - [minipass-collect](http://npm.im/minipass-collect)
  34. - [minipass-flush](http://npm.im/minipass-flush)
  35. - [minipass-pipeline](http://npm.im/minipass-pipeline)
  36. - [tap](http://npm.im/tap)
  37. - [tap-parser](http://npm.im/tap)
  38. - [treport](http://npm.im/tap)
  39. - [minipass-fetch](http://npm.im/minipass-fetch)
  40. - [pacote](http://npm.im/pacote)
  41. - [make-fetch-happen](http://npm.im/make-fetch-happen)
  42. - [cacache](http://npm.im/cacache)
  43. - [ssri](http://npm.im/ssri)
  44. - [npm-registry-fetch](http://npm.im/npm-registry-fetch)
  45. - [minipass-json-stream](http://npm.im/minipass-json-stream)
  46. - [minipass-sized](http://npm.im/minipass-sized)
  47. ## Differences from Node.js Streams
  48. There are several things that make Minipass streams different from (and in
  49. some ways superior to) Node.js core streams.
  50. Please read these caveats if you are familiar with noode-core streams and
  51. intend to use Minipass streams in your programs.
  52. ### Timing
  53. Minipass streams are designed to support synchronous use-cases. Thus, data
  54. is emitted as soon as it is available, always. It is buffered until read,
  55. but no longer. Another way to look at it is that Minipass streams are
  56. exactly as synchronous as the logic that writes into them.
  57. This can be surprising if your code relies on `PassThrough.write()` always
  58. providing data on the next tick rather than the current one, or being able
  59. to call `resume()` and not have the entire buffer disappear immediately.
  60. However, without this synchronicity guarantee, there would be no way for
  61. Minipass to achieve the speeds it does, or support the synchronous use
  62. cases that it does. Simply put, waiting takes time.
  63. This non-deferring approach makes Minipass streams much easier to reason
  64. about, especially in the context of Promises and other flow-control
  65. mechanisms.
  66. ### No High/Low Water Marks
  67. Node.js core streams will optimistically fill up a buffer, returning `true`
  68. on all writes until the limit is hit, even if the data has nowhere to go.
  69. Then, they will not attempt to draw more data in until the buffer size dips
  70. below a minimum value.
  71. Minipass streams are much simpler. The `write()` method will return `true`
  72. if the data has somewhere to go (which is to say, given the timing
  73. guarantees, that the data is already there by the time `write()` returns).
  74. If the data has nowhere to go, then `write()` returns false, and the data
  75. sits in a buffer, to be drained out immediately as soon as anyone consumes
  76. it.
  77. ### Hazards of Buffering (or: Why Minipass Is So Fast)
  78. Since data written to a Minipass stream is immediately written all the way
  79. through the pipeline, and `write()` always returns true/false based on
  80. whether the data was fully flushed, backpressure is communicated
  81. immediately to the upstream caller. This minimizes buffering.
  82. Consider this case:
  83. ```js
  84. const {PassThrough} = require('stream')
  85. const p1 = new PassThrough({ highWaterMark: 1024 })
  86. const p2 = new PassThrough({ highWaterMark: 1024 })
  87. const p3 = new PassThrough({ highWaterMark: 1024 })
  88. const p4 = new PassThrough({ highWaterMark: 1024 })
  89. p1.pipe(p2).pipe(p3).pipe(p4)
  90. p4.on('data', () => console.log('made it through'))
  91. // this returns false and buffers, then writes to p2 on next tick (1)
  92. // p2 returns false and buffers, pausing p1, then writes to p3 on next tick (2)
  93. // p3 returns false and buffers, pausing p2, then writes to p4 on next tick (3)
  94. // p4 returns false and buffers, pausing p3, then emits 'data' and 'drain'
  95. // on next tick (4)
  96. // p3 sees p4's 'drain' event, and calls resume(), emitting 'resume' and
  97. // 'drain' on next tick (5)
  98. // p2 sees p3's 'drain', calls resume(), emits 'resume' and 'drain' on next tick (6)
  99. // p1 sees p2's 'drain', calls resume(), emits 'resume' and 'drain' on next
  100. // tick (7)
  101. p1.write(Buffer.alloc(2048)) // returns false
  102. ```
  103. Along the way, the data was buffered and deferred at each stage, and
  104. multiple event deferrals happened, for an unblocked pipeline where it was
  105. perfectly safe to write all the way through!
  106. Furthermore, setting a `highWaterMark` of `1024` might lead someone reading
  107. the code to think an advisory maximum of 1KiB is being set for the
  108. pipeline. However, the actual advisory buffering level is the _sum_ of
  109. `highWaterMark` values, since each one has its own bucket.
  110. Consider the Minipass case:
  111. ```js
  112. const m1 = new Minipass()
  113. const m2 = new Minipass()
  114. const m3 = new Minipass()
  115. const m4 = new Minipass()
  116. m1.pipe(m2).pipe(m3).pipe(m4)
  117. m4.on('data', () => console.log('made it through'))
  118. // m1 is flowing, so it writes the data to m2 immediately
  119. // m2 is flowing, so it writes the data to m3 immediately
  120. // m3 is flowing, so it writes the data to m4 immediately
  121. // m4 is flowing, so it fires the 'data' event immediately, returns true
  122. // m4's write returned true, so m3 is still flowing, returns true
  123. // m3's write returned true, so m2 is still flowing, returns true
  124. // m2's write returned true, so m1 is still flowing, returns true
  125. // No event deferrals or buffering along the way!
  126. m1.write(Buffer.alloc(2048)) // returns true
  127. ```
  128. It is extremely unlikely that you _don't_ want to buffer any data written,
  129. or _ever_ buffer data that can be flushed all the way through. Neither
  130. node-core streams nor Minipass ever fail to buffer written data, but
  131. node-core streams do a lot of unnecessary buffering and pausing.
  132. As always, the faster implementation is the one that does less stuff and
  133. waits less time to do it.
  134. ### Immediately emit `end` for empty streams (when not paused)
  135. If a stream is not paused, and `end()` is called before writing any data
  136. into it, then it will emit `end` immediately.
  137. If you have logic that occurs on the `end` event which you don't want to
  138. potentially happen immediately (for example, closing file descriptors,
  139. moving on to the next entry in an archive parse stream, etc.) then be sure
  140. to call `stream.pause()` on creation, and then `stream.resume()` once you
  141. are ready to respond to the `end` event.
  142. ### Emit `end` When Asked
  143. One hazard of immediately emitting `'end'` is that you may not yet have had
  144. a chance to add a listener. In order to avoid this hazard, Minipass
  145. streams safely re-emit the `'end'` event if a new listener is added after
  146. `'end'` has been emitted.
  147. Ie, if you do `stream.on('end', someFunction)`, and the stream has already
  148. emitted `end`, then it will call the handler right away. (You can think of
  149. this somewhat like attaching a new `.then(fn)` to a previously-resolved
  150. Promise.)
  151. To prevent calling handlers multiple times who would not expect multiple
  152. ends to occur, all listeners are removed from the `'end'` event whenever it
  153. is emitted.
  154. ### Impact of "immediate flow" on Tee-streams
  155. A "tee stream" is a stream piping to multiple destinations:
  156. ```js
  157. const tee = new Minipass()
  158. t.pipe(dest1)
  159. t.pipe(dest2)
  160. t.write('foo') // goes to both destinations
  161. ```
  162. Since Minipass streams _immediately_ process any pending data through the
  163. pipeline when a new pipe destination is added, this can have surprising
  164. effects, especially when a stream comes in from some other function and may
  165. or may not have data in its buffer.
  166. ```js
  167. // WARNING! WILL LOSE DATA!
  168. const src = new Minipass()
  169. src.write('foo')
  170. src.pipe(dest1) // 'foo' chunk flows to dest1 immediately, and is gone
  171. src.pipe(dest2) // gets nothing!
  172. ```
  173. The solution is to create a dedicated tee-stream junction that pipes to
  174. both locations, and then pipe to _that_ instead.
  175. ```js
  176. // Safe example: tee to both places
  177. const src = new Minipass()
  178. src.write('foo')
  179. const tee = new Minipass()
  180. tee.pipe(dest1)
  181. tee.pipe(dest2)
  182. src.pipe(tee) // tee gets 'foo', pipes to both locations
  183. ```
  184. The same caveat applies to `on('data')` event listeners. The first one
  185. added will _immediately_ receive all of the data, leaving nothing for the
  186. second:
  187. ```js
  188. // WARNING! WILL LOSE DATA!
  189. const src = new Minipass()
  190. src.write('foo')
  191. src.on('data', handler1) // receives 'foo' right away
  192. src.on('data', handler2) // nothing to see here!
  193. ```
  194. Using a dedicated tee-stream can be used in this case as well:
  195. ```js
  196. // Safe example: tee to both data handlers
  197. const src = new Minipass()
  198. src.write('foo')
  199. const tee = new Minipass()
  200. tee.on('data', handler1)
  201. tee.on('data', handler2)
  202. src.pipe(tee)
  203. ```
  204. ## USAGE
  205. It's a stream! Use it like a stream and it'll most likely do what you
  206. want.
  207. ```js
  208. const Minipass = require('minipass')
  209. const mp = new Minipass(options) // optional: { encoding, objectMode }
  210. mp.write('foo')
  211. mp.pipe(someOtherStream)
  212. mp.end('bar')
  213. ```
  214. ### OPTIONS
  215. * `encoding` How would you like the data coming _out_ of the stream to be
  216. encoded? Accepts any values that can be passed to `Buffer.toString()`.
  217. * `objectMode` Emit data exactly as it comes in. This will be flipped on
  218. by default if you write() something other than a string or Buffer at any
  219. point. Setting `objectMode: true` will prevent setting any encoding
  220. value.
  221. ### API
  222. Implements the user-facing portions of Node.js's `Readable` and `Writable`
  223. streams.
  224. ### Methods
  225. * `write(chunk, [encoding], [callback])` - Put data in. (Note that, in the
  226. base Minipass class, the same data will come out.) Returns `false` if
  227. the stream will buffer the next write, or true if it's still in "flowing"
  228. mode.
  229. * `end([chunk, [encoding]], [callback])` - Signal that you have no more
  230. data to write. This will queue an `end` event to be fired when all the
  231. data has been consumed.
  232. * `setEncoding(encoding)` - Set the encoding for data coming of the stream.
  233. This can only be done once.
  234. * `pause()` - No more data for a while, please. This also prevents `end`
  235. from being emitted for empty streams until the stream is resumed.
  236. * `resume()` - Resume the stream. If there's data in the buffer, it is all
  237. discarded. Any buffered events are immediately emitted.
  238. * `pipe(dest)` - Send all output to the stream provided. There is no way
  239. to unpipe. When data is emitted, it is immediately written to any and
  240. all pipe destinations.
  241. * `on(ev, fn)`, `emit(ev, fn)` - Minipass streams are EventEmitters. Some
  242. events are given special treatment, however. (See below under "events".)
  243. * `promise()` - Returns a Promise that resolves when the stream emits
  244. `end`, or rejects if the stream emits `error`.
  245. * `collect()` - Return a Promise that resolves on `end` with an array
  246. containing each chunk of data that was emitted, or rejects if the stream
  247. emits `error`. Note that this consumes the stream data.
  248. * `concat()` - Same as `collect()`, but concatenates the data into a single
  249. Buffer object. Will reject the returned promise if the stream is in
  250. objectMode, or if it goes into objectMode by the end of the data.
  251. * `read(n)` - Consume `n` bytes of data out of the buffer. If `n` is not
  252. provided, then consume all of it. If `n` bytes are not available, then
  253. it returns null. **Note** consuming streams in this way is less
  254. efficient, and can lead to unnecessary Buffer copying.
  255. * `destroy([er])` - Destroy the stream. If an error is provided, then an
  256. `'error'` event is emitted. If the stream has a `close()` method, and
  257. has not emitted a `'close'` event yet, then `stream.close()` will be
  258. called. Any Promises returned by `.promise()`, `.collect()` or
  259. `.concat()` will be rejected. After being destroyed, writing to the
  260. stream will emit an error. No more data will be emitted if the stream is
  261. destroyed, even if it was previously buffered.
  262. ### Properties
  263. * `bufferLength` Read-only. Total number of bytes buffered, or in the case
  264. of objectMode, the total number of objects.
  265. * `encoding` The encoding that has been set. (Setting this is equivalent
  266. to calling `setEncoding(enc)` and has the same prohibition against
  267. setting multiple times.)
  268. * `flowing` Read-only. Boolean indicating whether a chunk written to the
  269. stream will be immediately emitted.
  270. * `emittedEnd` Read-only. Boolean indicating whether the end-ish events
  271. (ie, `end`, `prefinish`, `finish`) have been emitted. Note that
  272. listening on any end-ish event will immediateyl re-emit it if it has
  273. already been emitted.
  274. * `writable` Whether the stream is writable. Default `true`. Set to
  275. `false` when `end()`
  276. * `readable` Whether the stream is readable. Default `true`.
  277. * `buffer` A [yallist](http://npm.im/yallist) linked list of chunks written
  278. to the stream that have not yet been emitted. (It's probably a bad idea
  279. to mess with this.)
  280. * `pipes` A [yallist](http://npm.im/yallist) linked list of streams that
  281. this stream is piping into. (It's probably a bad idea to mess with
  282. this.)
  283. * `destroyed` A getter that indicates whether the stream was destroyed.
  284. * `paused` True if the stream has been explicitly paused, otherwise false.
  285. * `objectMode` Indicates whether the stream is in `objectMode`. Once set
  286. to `true`, it cannot be set to `false`.
  287. ### Events
  288. * `data` Emitted when there's data to read. Argument is the data to read.
  289. This is never emitted while not flowing. If a listener is attached, that
  290. will resume the stream.
  291. * `end` Emitted when there's no more data to read. This will be emitted
  292. immediately for empty streams when `end()` is called. If a listener is
  293. attached, and `end` was already emitted, then it will be emitted again.
  294. All listeners are removed when `end` is emitted.
  295. * `prefinish` An end-ish event that follows the same logic as `end` and is
  296. emitted in the same conditions where `end` is emitted. Emitted after
  297. `'end'`.
  298. * `finish` An end-ish event that follows the same logic as `end` and is
  299. emitted in the same conditions where `end` is emitted. Emitted after
  300. `'prefinish'`.
  301. * `close` An indication that an underlying resource has been released.
  302. Minipass does not emit this event, but will defer it until after `end`
  303. has been emitted, since it throws off some stream libraries otherwise.
  304. * `drain` Emitted when the internal buffer empties, and it is again
  305. suitable to `write()` into the stream.
  306. * `readable` Emitted when data is buffered and ready to be read by a
  307. consumer.
  308. * `resume` Emitted when stream changes state from buffering to flowing
  309. mode. (Ie, when `resume` is called, `pipe` is called, or a `data` event
  310. listener is added.)
  311. ### Static Methods
  312. * `Minipass.isStream(stream)` Returns `true` if the argument is a stream,
  313. and false otherwise. To be considered a stream, the object must be
  314. either an instance of Minipass, or an EventEmitter that has either a
  315. `pipe()` method, or both `write()` and `end()` methods. (Pretty much any
  316. stream in node-land will return `true` for this.)
  317. ## EXAMPLES
  318. Here are some examples of things you can do with Minipass streams.
  319. ### simple "are you done yet" promise
  320. ```js
  321. mp.promise().then(() => {
  322. // stream is finished
  323. }, er => {
  324. // stream emitted an error
  325. })
  326. ```
  327. ### collecting
  328. ```js
  329. mp.collect().then(all => {
  330. // all is an array of all the data emitted
  331. // encoding is supported in this case, so
  332. // so the result will be a collection of strings if
  333. // an encoding is specified, or buffers/objects if not.
  334. //
  335. // In an async function, you may do
  336. // const data = await stream.collect()
  337. })
  338. ```
  339. ### collecting into a single blob
  340. This is a bit slower because it concatenates the data into one chunk for
  341. you, but if you're going to do it yourself anyway, it's convenient this
  342. way:
  343. ```js
  344. mp.concat().then(onebigchunk => {
  345. // onebigchunk is a string if the stream
  346. // had an encoding set, or a buffer otherwise.
  347. })
  348. ```
  349. ### iteration
  350. You can iterate over streams synchronously or asynchronously in platforms
  351. that support it.
  352. Synchronous iteration will end when the currently available data is
  353. consumed, even if the `end` event has not been reached. In string and
  354. buffer mode, the data is concatenated, so unless multiple writes are
  355. occurring in the same tick as the `read()`, sync iteration loops will
  356. generally only have a single iteration.
  357. To consume chunks in this way exactly as they have been written, with no
  358. flattening, create the stream with the `{ objectMode: true }` option.
  359. ```js
  360. const mp = new Minipass({ objectMode: true })
  361. mp.write('a')
  362. mp.write('b')
  363. for (let letter of mp) {
  364. console.log(letter) // a, b
  365. }
  366. mp.write('c')
  367. mp.write('d')
  368. for (let letter of mp) {
  369. console.log(letter) // c, d
  370. }
  371. mp.write('e')
  372. mp.end()
  373. for (let letter of mp) {
  374. console.log(letter) // e
  375. }
  376. for (let letter of mp) {
  377. console.log(letter) // nothing
  378. }
  379. ```
  380. Asynchronous iteration will continue until the end event is reached,
  381. consuming all of the data.
  382. ```js
  383. const mp = new Minipass({ encoding: 'utf8' })
  384. // some source of some data
  385. let i = 5
  386. const inter = setInterval(() => {
  387. if (i --> 0)
  388. mp.write(Buffer.from('foo\n', 'utf8'))
  389. else {
  390. mp.end()
  391. clearInterval(inter)
  392. }
  393. }, 100)
  394. // consume the data with asynchronous iteration
  395. async function consume () {
  396. for await (let chunk of mp) {
  397. console.log(chunk)
  398. }
  399. return 'ok'
  400. }
  401. consume().then(res => console.log(res))
  402. // logs `foo\n` 5 times, and then `ok`
  403. ```
  404. ### subclass that `console.log()`s everything written into it
  405. ```js
  406. class Logger extends Minipass {
  407. write (chunk, encoding, callback) {
  408. console.log('WRITE', chunk, encoding)
  409. return super.write(chunk, encoding, callback)
  410. }
  411. end (chunk, encoding, callback) {
  412. console.log('END', chunk, encoding)
  413. return super.end(chunk, encoding, callback)
  414. }
  415. }
  416. someSource.pipe(new Logger()).pipe(someDest)
  417. ```
  418. ### same thing, but using an inline anonymous class
  419. ```js
  420. // js classes are fun
  421. someSource
  422. .pipe(new (class extends Minipass {
  423. emit (ev, ...data) {
  424. // let's also log events, because debugging some weird thing
  425. console.log('EMIT', ev)
  426. return super.emit(ev, ...data)
  427. }
  428. write (chunk, encoding, callback) {
  429. console.log('WRITE', chunk, encoding)
  430. return super.write(chunk, encoding, callback)
  431. }
  432. end (chunk, encoding, callback) {
  433. console.log('END', chunk, encoding)
  434. return super.end(chunk, encoding, callback)
  435. }
  436. }))
  437. .pipe(someDest)
  438. ```
  439. ### subclass that defers 'end' for some reason
  440. ```js
  441. class SlowEnd extends Minipass {
  442. emit (ev, ...args) {
  443. if (ev === 'end') {
  444. console.log('going to end, hold on a sec')
  445. setTimeout(() => {
  446. console.log('ok, ready to end now')
  447. super.emit('end', ...args)
  448. }, 100)
  449. } else {
  450. return super.emit(ev, ...args)
  451. }
  452. }
  453. }
  454. ```
  455. ### transform that creates newline-delimited JSON
  456. ```js
  457. class NDJSONEncode extends Minipass {
  458. write (obj, cb) {
  459. try {
  460. // JSON.stringify can throw, emit an error on that
  461. return super.write(JSON.stringify(obj) + '\n', 'utf8', cb)
  462. } catch (er) {
  463. this.emit('error', er)
  464. }
  465. }
  466. end (obj, cb) {
  467. if (typeof obj === 'function') {
  468. cb = obj
  469. obj = undefined
  470. }
  471. if (obj !== undefined) {
  472. this.write(obj)
  473. }
  474. return super.end(cb)
  475. }
  476. }
  477. ```
  478. ### transform that parses newline-delimited JSON
  479. ```js
  480. class NDJSONDecode extends Minipass {
  481. constructor (options) {
  482. // always be in object mode, as far as Minipass is concerned
  483. super({ objectMode: true })
  484. this._jsonBuffer = ''
  485. }
  486. write (chunk, encoding, cb) {
  487. if (typeof chunk === 'string' &&
  488. typeof encoding === 'string' &&
  489. encoding !== 'utf8') {
  490. chunk = Buffer.from(chunk, encoding).toString()
  491. } else if (Buffer.isBuffer(chunk))
  492. chunk = chunk.toString()
  493. }
  494. if (typeof encoding === 'function') {
  495. cb = encoding
  496. }
  497. const jsonData = (this._jsonBuffer + chunk).split('\n')
  498. this._jsonBuffer = jsonData.pop()
  499. for (let i = 0; i < jsonData.length; i++) {
  500. let parsed
  501. try {
  502. super.write(parsed)
  503. } catch (er) {
  504. this.emit('error', er)
  505. continue
  506. }
  507. }
  508. if (cb)
  509. cb()
  510. }
  511. }
  512. ```