Skip to content

Instantly share code, notes, and snippets.

@juandopazo
Last active December 16, 2015 08:19
Show Gist options
  • Save juandopazo/5405383 to your computer and use it in GitHub Desktop.
Save juandopazo/5405383 to your computer and use it in GitHub Desktop.
Storage module design and API for review

Storage

Motivation

There are 3 major types of storage is browsers up to date:

  • IndexedDB. The future.
  • localStorage. Limited in size and can be observed through the storage event.
  • WebSQL. Dropped by the W3C but it's still very present in the wild, with notable platforms like iOS and PhoneGap.

Browser support Green: IndexedDB; blue: WebSQL; yellow: localStorage

In the mobile world local cache and storage is very important for various reasons:

  • faster loading times
  • unreliable connections
  • offline use

My particular use case is an app to be used "in the field" where you probably don't have a connection and you need to do data entry. My solution is to save to a local storage and sync to the server when a connection is available. The 5 MB of localStorage may not be enough in my case, but I won't be able to tell until the app is used by real users.

Open questions

  • Is there enough interest in this considering the availability of localStorage?
  • Is it too early to use these technologies?

Architecture

The goal is to use conditional loading to load one of 3 or 4 sub-modules: indexeddb, websql, localstorage and a fallback to memory.

  • IndexedDB uses a key-based storage organized in "object stores".
  • WebSQL uses SQL tables. WebSQL can be used as a fallback for object stores by matching a table to a store and using tables with two columns: a key and a value column which saves the object serialized as JSON.
  • IndexedDB uses structured cloning which is similar to JSON.parse(JSON.stringify(obj)).

Open questions

  • Is falling back to localStorage a good idea considering the limited size and observability?
  • Is the whole concept OK or is it too hackish?
  • Should this include over-the-wire databases via some protocol?

API design

The API aims to find the lowest common denominator between all implementations. A database requires:

  • A name. Required by both IndexedDB and WebSQL
  • A version. Required by both IndexedDB and WebSQL
  • A structure, which means the name of each object store, tied to the version. Required by both IndexedDB and WebSQL
  • A size in bytes. Required by WebSQL

A Storage constructor receives all these parameters and creates an instance with properties with the names of each object store:

var storage = new Y.Storage({
  name: 'my database',
  version: 1,
  size: 5242880,
  stores: ['fooStore', 'barStore']
});

Assert.isObject(storage.fooStore); // true
Assert.isObject(storage.barStore); // true

A Storage instance also has a close method that closes the database connection/session.

A Store object is created for each requested store in a property of the same name. A store can perform the following actions:

  • get. Retrieve an object by key
  • put. Insert or update an object by key
  • remove. Remove an object by key
  • clear. Remove all objects from the store
  • count. Count the objects in the store

All methods return a promise.

Example:

storage.fooStore.put('some key', { foo: 'bar' }).then(...);

Open questions

  • Should Storage inherit from Base in order to use attributes for configuration and maybe fire close or error events? A versionchange event is very important.
@juandopazo
Copy link
Author

Updated the gist: just promises, no callbacks; all actions return promises. Also added the question to discuss if this should include over-the-wire databases.

@juandopazo
Copy link
Author

Ouch! light bulb

@caridy by over-the-wire do you mean using this in a Node environment to abstract over Mongo, MySQL or whatever? If that's the case, then I'm totally interested. Sadly my side project migrated from Node to Python, so I won't be using YUI on the server to test this in real life.

@caridy
Copy link

caridy commented Apr 19, 2013

@juandopazo, yes, I'm talking about Node. And that algorithm for conflict resolution is called "latency compensation" and there are a couple of interesting implementations. The one from Meteor works pretty well, we might be able to use that one. In theory, there is not much different between a REST-like API and IndexedDB.

Aside from that, I think we should use Y.DB instead of Y.Storage. I really liked the implementation from @sdesai for the CRT project.

@juandopazo
Copy link
Author

@caridy I guess I need a bit more research then. I'll look at Meteor and the literature on latency compensation.

I'd love to see @sdesai's implementation!

@caridy
Copy link

caridy commented Apr 20, 2013

@ItsAsbreuk
Copy link

Quite interesting "latency compensation".
Shouldn't YUI support this inside the core?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment