After long, grueling months (years? or does it only feel like years?), your web application nears completion. It is tightly coded, well documented, works across all modern browsers, and is well received by your beta testers. It’s nearly time to go live, and a smile of pure relief plays upon your lips… and freezes into a rictus grin when your client turns to you, and asks, "so, hey, can we speed up the dynamic cat pic loading? Especially when I close the browser and come back to it later. I think that’s really key to the whole application."
Long, long ago we discussed our jQuery plugin that will allow you to cache responses of ajax queries in Local Storage, so long as they’re strings, or something that can be coerced to a string (objects as JSON, numbers). We also previously discussed adding an ajax transport to allow us to handle sending and receiving binary blobs and array buffers via jQuery ajax.
But what if we need to cache binary blobs or arraybuffers? Say, we need those cat pics on the double – we could convert them to and from base64, but not only is that slow, but we’re certain to run up against the 5MB limit of local storage in short order. No, what we need is some way to cache binary data in some sort of client-side database…
Want to get straight to the goods? Here’s the link to the Github repo – keep reading for the details.
Picking the appropriate tool for the job
Or any tool, really
There are a number of candidates that could (or once could have) provided our solution. If we knew exactly what resources would be needed on first visit (and, as an added bonus, wanted at least some part of the web application available offline), we could have used Application Cache. I say could have, because Application Cache is dead (long live Service Workers). Its replacement, Service Workers, isn’t widely supported yet, and this doesn’t really fit our use case that well anyways.
Well, what we really need is on-demand file storage. Hey, I remember reading about a File System API once, whatever happened to that?
Our options are looking a little thinner on the ground, now. But wait! If you’re also familiar with the server-side of web development, you’re probably thinking (a little guiltily) of the last time you couldn’t use the file system but needed to store files. If we can’t use the file system – how about the database?
IndexedDB to the rescue! (Not WebSQL, that’s also dead.) IndexedDB is fairly widely-supported by modern browsers, and we can definitely store binary data in an IndexedDB database.
If you pretend complexity and async operations aren’t there, they go away on their own
One issue is that IndexedDB has a reputation for being fairly complex. No worries, some helpful open source author will help abstract that away for you (see also the excellent localForage and Dexie projects if you intend to use IndexedDB for any purpose other than caching ajax requests in your application).
The other concern for the purpose of our ajax caching is that IndexedDB operates asynchronously (like an XHR – no, I don’t want to hear about sync XHR, that’s also dead, and good riddance). Our Jalc plugin that allows us to cache jquery ajax responses, relies on the ability to specify an ajax transport for a given request – and it needs to respond with a value synchronously if it’s going to handle the request instead of simply passing it on to the next transport factory.
Two for the price of one
So, what we need to do is store and retrieve a cached value asychronously, but know whether the value exists sychronously. Thus, the solution – use both LocalStorage and IndexedDB. So, we extend our JALC plugin – first, we add the ability to send and receive binary data in the form of blobs and array buffers, and then we set up an indexeddb database for storing our values.
I’ll skip over the code for the binary sending and receiving, since we’ve mostly seen it before (you can view it in the Github repo, however). You’ll require such a patch, as adding support for binary requests in jQuery is also dead, and $.xhr and the Fetch API isn’t ready for prime-time yet.
Let’s take a quick walk through what we’re doing to support IndexedDB, however (see the up-to-date version on Github).
(function($, window){
var idb = window.indexedDB,
dbReady = $.Deferred(),
db;
/**
* At this point, we'll only support unprefixed, non-experimental versions of IndexedDB, to simplify our
* lives - there are a number of differences we would need to account for if we were to attempt to support
* early attempts at IndexedDB implementations, some of which you can read about in the excellent MDN
* documentation at
* https://developer.mozilla.org/en/docs/Web/API/IndexedDB_API and
* https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Using_IndexedDB.
*/
if (!window.indexedDB) throw new Error("Expecting unprefixed IndexedDB support on window object.");
/**
* Create the 'jalic' IndexedDB db.
* For now, we handle errors merely by logging them and rejecting the associated Deferred object with the
* error event. This should be replaced by something more robust in the future...
*
* @param idb
* @returns {*}
*/
function createDatabase(idb){
var dbreq = idb.open("jalic", 1);
/**
* Something went wrong when opening our db. This could range from the user refusing the
* request to allow this site to store data, to the current version of the db being higher
* than the one we requested, to storage issues or lacking implementation.
* @param event
*/
dbreq.onerror = function(event){
console.log(event);
dbReady.reject(event);
};
/**
* The on upgrade needed event is called whenever we're opening a database with a version number
* higher than the currently existing version number, which includes when the database doesn't
* currently exist. Within this function we define the structure of the db.
* @param event
*/
dbreq.onupgradeneeded = function(event){
var objectStore;
db = event.target.result;
db.onerror = function(event){
console.log(event);
};
objectStore = db.createObjectStore("jalicData", {keyPath:"jdName"});
objectStore.createIndex("storedAt", "storedAt", {unique:false});
objectStore.transaction.oncomplete = function(event){
dbReady.resolve();
};
};
/**
* DB was opened successfully, with no upgrade needed.
* @param event
*/
dbreq.onsuccess = function(event){
db = event.target.result;
dbReady.resolve();
};
return dbReady.promise();
}
createDatabase(idb);
/**
* Define a simple interface mimicking the Storage interface on the jQuery object, with the major difference being
* everything executes asynchronously, and returns a $.Deferred object to account for that.
*/
$.jidb = {
/**
* Set an item within the jalicData objectStore, using the given jdName and data, with optionally
* a dataType parameter to store alongside the data. If dataType is not provided, it is the result of
* typeof data.
* Notice that we 'put' data, rather than add it - that means we will always overwrite data with an
* identical key (jdName), if it already exists.
* @param jdName
* @param data
* @param dataType
* @returns {$.Deferred} Returns a jQuery Deferred object, which resolves with an empty body on success,
* or else resolves with the transaction or request error on failure.
*/
setItem:function(jdName, data, dataType){
var defer = $.Deferred(),
transaction = db.transaction(["jalicData"], "readwrite"),
objectStore = transaction.objectStore("jalicData"),
request;
dataType = dataType || typeof data;
transaction.oncomplete = function(){
return defer.resolve();
};
transaction.onerror = function(event){
console.log(event);
return defer.reject(event);
};
request = objectStore.put({jdName:jdName, storedAt:+new Date(), dataType:dataType, data:data});
request.onerror = function(event){
console.log(event);
defer.reject(event);
};
return defer.promise();
},
/**
* Retrieve an item from the jalicData objectStore, using the given jdName as the key.
* @param jdName
* @returns {$.Deferred} Returns a jQuery Deferred object, which resolves with the request result as an object
* on success, or else resolves with the transaction or request error on failure.
*/
getItem:function(jdName){
var defer = $.Deferred(),
transaction = db.transaction(["jalicData"], "readonly"),
objectStore = transaction.objectStore("jalicData"),
request = objectStore.get(jdName);
request.onerror = function(event){
console.log(event);
defer.reject(event);
};
request.onsuccess = function(event){
defer.resolve(request.result);
};
return defer.promise();
},
/**
* Remove an item from the jalicData objectStore, using the given jdName as the key.
* @param jdName
* @returns {$.Deferred} Returns a jQuery Deferred object, which resolves with an empty body on success,
* or else resolves with the transaction or request error on failure.
*/
removeItem:function(jdName){
var defer = $.Deferred(),
transaction = db.transaction(["jalicData"], "readwrite"),
objectStore = transaction.objectStore("jalicData"),
request = objectStore.delete(jdName);
request.onerror = function(event){
console.log(event);
defer.reject(event);
};
request.onsuccess = function(){
defer.resolve();
};
return defer.promise();
},
/**
* Delete the jalic database and recreate it.
* @returns {$.Deferred} Returns a jQuery Deferred object, which resolves with an empty body on success,
* or else resolves with the transaction or request error on failure.
*/
clear:function(){
var defer = $.Deferred(),
request = idb.deleteDatabase("jalic");
dbReady = $.Deferred();
request.onerror = function(event){
console.log(event);
defer.reject(event);
};
request.onsuccess = function(){
dbReady.done(function(){
defer.resolve();
});
createDatabase(idb);
};
return defer.promise();
}
};
})(jQuery, window);
The code and comments should give you a fair idea of what we’re doing here – unlike the previously mentioned localForage or Dexie, we don’t attempt to account for older, experimental versions of IndexedDB, or fallback to something like WebSQL. IndexedDB is the future! Maybe, probably, unless it also gets killed at some point. Knocks on wood
We also aren’t attempting to wrap everything IndexedDB can do – instead, we have a single database and objectStore into which we’ll be shoving everything we intend to cache, and providing a simple wrapper around it that superficially resembles the Storage interface, with the exception that everything is asynchronous and returns a $.Deferred object.
As a result of this setup, our actual caching logic can remain mostly the same – let’s take a look and discuss what’s changed (see the up-to-date code on Github):
/**
* https://github.com/SaneMethod/jalic
*/
(function($, window){
/**
* Generate the cache key under which to store the local data - either the cache key supplied,
* or one generated from the url, the type and, if present, the data.
*/
var genCacheKey = function (options) {
var url = options.url.replace(/jQuery.*/, '');
// Strip _={timestamp}, if cache is set to false
if (options.cache === false) {
url = url.replace(/([?&])_=[^&]*/, '');
}
return options.cacheKey || url + options.type + (options.data || '');
};
/**
* Determine whether we're using localStorage or, if the user has specified something other than a boolean
* value for options.localCache, whether the value appears to satisfy the plugin's requirements.
* Otherwise, throw a new TypeError indicating what type of value we expect.
* @param {boolean|object} storage
* @returns {boolean|object}
*/
var getStorage = function(storage){
if (!storage) return false;
if (storage === true) return window.localStorage;
if (typeof storage === "object" && 'getItem' in storage &&
'removeItem' in storage && 'setItem' in storage)
{
return storage;
}
throw new TypeError("localCache must either be a boolean value, " +
"or an object which implements the Storage interface.");
};
/**
* Remove the item specified by cacheKey from local storage (but not from the IndexedDB, as in all usages
* of this function we expect to overwrite the value with addToStorage shortly).
* @param {Storage|object} storage
* @param {string} cacheKey
*/
var removeFromStorage = function(storage, cacheKey){
storage.removeItem(cacheKey);
storage.removeItem(cacheKey + 'cachettl');
};
/**
* Add an item to local storage and IndexedDB storage. We use local storage to satisfy our
* need to synchronously determine whether we have a value stored for a given key (and whether it is
* within the cachettl), and use IndexedDB to store the actual value associated with the key.
* @param {Storage|object} storage
* @param {string} cacheKey
* @param {number} ttl
* @param {*} data
* @param {string} dataType
* @returns {$.Deferred}
*/
var addToStorage = function(storage, cacheKey, ttl, data, dataType){
var defer = $.Deferred();
try{
storage.setItem(cacheKey, 1);
storage.setItem(cacheKey + 'cachettl', ttl);
}catch(e){
removeFromStorage(storage, cacheKey);
defer.reject(e);
return defer.promise();
}
return $.jidb.setItem(cacheKey, data, dataType);
};
/**
* Prefilter for caching ajax calls.
* See also $.ajaxTransport for the elements that make this compatible with jQuery Deferred.
* New parameters available on the ajax call:
* localCache : true - required - either a boolean (in which case localStorage is used),
* or an object implementing the Storage interface, in which case that object is used instead.
* cacheTTL : 5, - optional - cache time in hours, default is 5.
* cacheKey : 'myCacheKey', - optional - key under which cached string will be stored
* isCacheValid : function - optional - return true for valid, false for invalid
* @method $.ajaxPrefilter
* @param options {Object} Options for the ajax call, modified with ajax standard settings
*/
$.ajaxPrefilter(function(options){
var storage = getStorage(options.localCache),
hourstl = +new Date() + 1000 * 60 * 60 * (options.cacheTTL || 5),
cacheKey = genCacheKey(options),
cacheValid = options.isCacheValid,
ttl,
value;
if (!storage) return;
ttl = storage.getItem(cacheKey + 'cachettl');
if (cacheValid && typeof cacheValid === 'function' && !cacheValid()){
removeFromStorage(storage, cacheKey);
ttl = 0;
}
if (ttl && ttl < +new Date()){
removeFromStorage(storage, cacheKey);
ttl = 0;
}
value = storage.getItem(cacheKey);
if (!value){
// If it not in the cache, we store the data, add success callback - normal callback will proceed
if (options.success) {
options.realsuccess = options.success;
}
options.success = function(data, status, jqXHR) {
var dataType = this.dataType || jqXHR.getResponseHeader('Content-Type');
// Save the data to storage, catching Storage exception and IndexedDB exceptions alike
// and reject the returned promise as a result.
addToStorage(storage, cacheKey, hourstl, data, dataType).done(function(){
if (options.realsuccess) options.realsuccess(data, status, jqXHR);
}).fail(function(event){
console.log(event);
});
};
}
});
/**
* This function performs the fetch from cache portion of the functionality needed to cache ajax
* calls and still fulfill the jqXHR Deferred Promise interface.
* See also $.ajaxPrefilter
* @method $.ajaxTransport
* @params options {Object} Options for the ajax call, modified with ajax standard settings
*/
$.ajaxTransport("+*", function(options){
if (options.localCache)
{
var cacheKey = genCacheKey(options),
storage = getStorage(options.localCache),
value = (storage) ? storage.getItem(cacheKey) : false;
if (value){
/**
* If the key is in the Storage-based cache, indicate that we want to handle this ajax request
* (by returning a value), and use the cache key to retrieve the value from the IndexedDB. Then,
* call the completeCallback with the fetched value.
*/
return {
send:function(headers, completeCallback) {
$.jidb.getItem(cacheKey).done(function(result){
var response = {};
response[result.dataType] = result.data;
completeCallback(200, 'success', response, '');
}).fail(function(){
completeCallback(500, 'cache failure', void 0, '');
});
},
abort:function() {
console.log("Aborted ajax transport for caching.");
}
};
}
}
});
})(jQuery, window);
If you’re acquainted with the Jalc plugin, this should look familiar. The main changes lie in the addToStorage function, which we call to actual store the values – notice that we need to handle both the case where we’ve hit an error in local storage (such as exceeding the storage limit), and potential errors responses from our IndexedDB wrapper – and the ajaxTransport we use to deliver the cached response which, as promised, uses our dirty little hack of relying on the synchronous nature of LocalStorage to tell us whether a key exists and, once we’ve asserted that we’ll be delivering the transport for this request, taking our time with the asynchronous operation to actually fetch and return the requested data.
That’s it, folks
This plugin should be considered experimental! It works as a proof of concept, but I wouldn’t place it into production if I were you – it doesn’t attempt to handle cases like Privacy mode in Firefox (where IndexedDB says it’s available, but fails to function), no fallbacks, and weak-to-nonexistant error handling. Want to use it somewhere serious? Pull requests are welcome. 🙂
Take a look at the new Jalic plugin on Github.
