Stepping Into Node Streams Twice

CLI for transforming some JSON — Chromium’s bookmarks, specifically — into HTML, for quick (and dirty) publishing here.

And while we’re at it, let’s ponder Node streams again?

Unix pipes…

  1. I’m tired of (irked by?) newbies’ idolatry of pipes. Nu, I’ve been around too long. *Yawn*. The utility of this std{in,out,err} architecture is really in its (over)simplification. Universal, sure, because it’s so primitive. Lowest common denominator.
      1. Oversimplified as opposed to what? In Unix these were byte-streams, so the evolutionary step would be XDR (1995) — schema/IDL-based stuff… leading to Protocol Buffers, etc.
      2. From there we evolve to RPC/messaging semantics (remember IIOP?), event driven, and ultimately, dynamically (aka “duck”) typed data, which is what Node streams actually provides.
      3. (And then piping goes too far, maybe, like how Julian “streams” state changes (REST?) over Socket.IO?)

    So, what’s the value in using Node’s streams API for this trivial task? The extremely abstracted/generalized API would be useful if we wanted to “freely” mix and compose modules. Do we?

  2. Another key property of this architectural pattern is (potentially) avoiding loading all data into memory (primary storage), by instead doing input-processing-output (I/O, ie secondary storage) small pieces at a time. (But, performance is complicated…)
    My bookmarks file(s) are typically just a few mebibytes, and I’ll be needing such import-export infrequently, manually, so memory footprint won’t be an issue.

Node streams API (v2?)

Event driven parser

Ah, message passing; that’s more like it! (When did parsing become synonymous with streaming?!)

NPM: the library of Babel

As usual in our ecosystem, too much going on…

  1. dominictarr/JSONStream
  2. trentm/json (almost, but not quite…)
  3. stream-json (TMI; more opaque system would pro’ly perform better?)
  4. dscape/clarinet

ad libitum (ad nauseam)…


  1. Essentially, what I wanted is, eg:
    $ bookmarks "JSON streaming" < ~/.config/chromium/Default/Bookmarks
    <li>JSON streaming
    	<li><a href="">dominictarr/JSONStream</a></li>
  2. I want nested folders flattened, ie a breadth first traversal (BFS).
  3. Input should default to ~/.config/chromium/Default/Bookmarks, obviously.
  4. Maybe more grep-like argument/options, fully regexp, or JSONPath? Overkill.


Comments are closed.