Min Streams Interface Spec

simple-streams.md

Simple Streams Interface Spec

This spec describes a minimal stream interface meant for protocol implementors. Modules written to the interfaces in this spec can be used by a wide variety of project. The protocols will not need any extra dependencies themselves. It's just an interface to implement.

Simple Stream

A simple stream is just a function. It represents a pull-stream. It has the following signature:

// Implementing a stream.
function read(abort, callback) {
  // abort is a flag for signaling to the source that you're done with the stream.
  // if abort is truthy, and you're a filter, you usually want to pass it on the next time you call the source read.
  // if youe're a source, then stop reading and cleanup.  Call the callback with an end event when done.
    
  // To encode an item, have a falsy value for the first param and the item for the second.
  callback(null, item);
  
  // To encode the end of the stream, use the special `undefined` value for item.
  callback(null, undefined);
  // or simply
  callback();
  
  // If you wish to emit an error, pass a truthy value to the first parameter to the callback;
  callback(new Error("Oops, the monkey got out and ate all the banannas."));
});

// Consuming a stream
read(null, function (err, data) {
  // I got data!
});

The callback is allowed to be called before the read function returns (sync callback like in Array.prototype.forEach), but it won't always be sync because it might depend on some non-blocking I/O source. If you're doing any recursive loops over data, make sure to take this into account and not blow your stack.

Note: Dominic Tarr's pull-streams have a slightly different encoding for end events. Use an adapter for interop.

Filters

A protocol that translates between two stream formats is called a filter. The input and output events are a many-to-many relationship. Every input event may output zero or more output events.

Pull Filter

The easiest to understand filter is a pull filter. Modules implementing this spec export a function that accepts a read function and return a new read function.

// Implementing a pull filter
// input: strings
// output: uppercase string
function toupper(read) {
  return function (abort, callback) {
    // Forward abort through, we don't need to mess with it.
    read(abort, function (err, data) {
      // Forward errors and end of stream events throguh as well.
      if (data === undefined) {
        return callback(err);
      }
      callback(null, data.toUpperCase());
    });
  }
}

// Using the filter
var words = createWords(); // An imaginary stream source that emits strings
var uppercaseWords = toupper(words);
// Now consume the new stream instead and it will be the uppercase protocol.

Push Filter

The easiest to implement filter, especially when the protocol events are many-to-many is the push filter. With this interface you are freed from worrying about abort messages and back-pressure. Just describe the core protocol logic and generic helper functions can later convert this to a proper pull filter with properback-pressure and abort forwarding.

// Creating a push filter
// input: byte arrays
// output: individual bytes
function splitBytes(emit) {
  return function (err, item) {
    // Forward errors and END events through.
    if (item === undefined) {
      return emit(err);
    }
    // Emit the items one at a time;
    for (var i = 0, l = item.length; i < l; i++) {
      emit(null, item[i]);
    }
  }
}

If your module is best described using a push filter, don't depend on some conversion library and export a pull filter. Simply export this function and document that it implements the push filter interface. Then people using your library can decide if they want to wrap it.

If several push-filters are chained in a row, the functions are directly composable without being first converted to pull filters. This removes several layers of complexity, buffering and keeps things fast.

An example of an echo server in working code https://github.com/creationix/moonslice-node/blob/master/testserver.js#L4

thanks!