blog

html5 logo

Ajax Upload Part II: XHR2 (and FileReader)

by

So, the client has told you their users should be able to upload their drunken party pictures for all the internet to see. "We want the very best experience possible," they tell you. "Simple, seamless – maybe using that new html5 thing I’ve heard so much about."

You choke back a little bit of hope. You’ve been disappointed so often. Trying hard to sound non-chalant, you say "only modern browsers, right?" You hold your breath. They nod, and on the outside, you nod back. On the inside, you giggle. No flash, no iframe hacks – just ajax, as God (well, the W3C) intended. Ah, beautiful.

As promised in part I, we’re going take a quick walk through what it takes to upload a file with ajax. We’re also going to take a look at reading a file into memory to play with in your app.

Inputs and Drop Targets

So, let’s say we have two images of some hip twenty-somethings having a good time, drunk1.png and drunk2.jpg. Our job is to get these files from the user’s computer to our server as simply as possible. The standard input is a familiar way of doing this, but we might also want to take advantage of the new drag and drop capabilities in modern browsers. So, we could have something like:

<form>
<input id="hupinput" type="file" />
</form>

or:

&lt;div id=&quot;hupdiv&quot;&gt;&lt;/div&gt;

and then just define some styles that will give us a properly sized drop target.

Now, the drag and drop api in html5 is a little strange. We need to cancel the default action and stop propagation for dragover and drop in order to allow something to be drag and dropped. The idea seems to be that the default action is to not allow anything to be dragged and dropped. You’ll just have to swing with it.

$('inputname').off('dragover').on('dragover', function(event){
    event = event.originalEvent;
    event.preventDefault();
    event.stopPropagation();
});
var files;
$('inputname').off('drop change').on('drop change', function(event){
    event = event.originalEvent;
    event.preventDefault();
    event.stopPropagation();
    // if drop and drag target
    files = event.dataTransfer.files;
    // or if an input
    files = event.target.files;
});

End result of this is that we’ll end up with a FileList object that acts like an array, with each element being a reference to a file. We can then iterate through it to upload or read our files.

Uploading

XHR2 introduces some new features, with the ones of interest being the new upload attribute on xhr, which returns an XMLHttpRequestUpload object. This exposes the progress of an upload to us. Otherwise, we’ll be using the standard send() call with an ArrayBuffer of the file to upload it.
Here’s what part of the setup and upload looks like in the plugin we’ll be discussing shortly:

    /**
     * Deferred wrapper for xhr upload.
     * @param options
     * @param file
     * @returns {Object} promise The deferred promise object.
     * @constructor
     */
    function DeferXhr(options, file){
        var that = this;
    this.defer = $.Deferred();
    this.file = file;
    this.options = options;
    this.paused = false;
    this.progress = 0;
    this.xhr = new '<a class="zem_slink" title="XMLHttpRequest" href="http://en.wikipedia.org/wiki/XMLHttpRequest" target="_blank" rel="wikipedia">XMLHttpRequest</a>();'
    if (this.options.chunked)
    {
        this.start = 0;
        this.end = Math.min(this.start+this.options.chunk_size, this.file.size);
    }
    this.xhr.addEventListener('load', function(){that.complete();}, false);
    this.xhr.upload.addEventListener('progress', function(event){that.uploadProgress(event);}, false);
    this.xhr.upload.addEventListener('error', function(event){that.uploadError(event);}, false);
    this.upload();
    return this.defer.promise();
}
/**
 * Carry out the xhr upload, optionally chunked.
 */
DeferXhr.prototype.upload = function(){
    this.xhr.open(this.options.type, this.options.url, this.options.async);
    this.xhr.setRequestHeader('Accept', 'application/json');
    this.xhr.setRequestHeader('X-File-Name', this.file.name);
    this.xhr.setRequestHeader('X-File-Type', this.file.type);
    if (this.options.chunked)
    {
        this.xhr.overrideMimeType('application/octet-stream');
        this.xhr.setRequestHeader('Content-Range', 'bytes '+this.start+"-"+this.end+'/'+this.file.size);
        this.xhr.send(this.file.slice(this.start, this.end));
    }
    else
    {
        this.xhr.overrideMimeType((this.file.type || 'application/octet-stream'));
        this.xhr.send(this.file);
    }
}

The main things to notice are that if we’re sending a file ‘chunked’ (that is, in pieces of a pre-determined size – this is a good way to get around, say, php upload file size limits or allow for pausing and resuming a file upload, by the way) we’re using the slice() method on our file to get an ArrayBuffer of the appropriate size out of the buffer representing the whole file. This also saves us from needing to read in the whole file before sending it.

In order to know what’s happening with our file upload, we just need to listen (as noted above) on the load, progress and errors events of xhr and xhr.upload.

Reading

This is probably one of my favourite new things to play with in javascript – the FileReader api (which is a subset of the File api). Get a file from the local user filesystem and work with it in the browser. Yay!

Its exciting, because coming from a C and Java background, getting a file from the local system wasn’t ever a problem, and there’s been numerous times in my web dev endeavours when it has been. Ever needed to use that ugly hack where you upload a file to a server and bounce it immediately back to the client, just so you can have access to some image or text file from the user? Yeah, good riddance to that.

Anyways, our plugin is going to do that to, because the two concepts are fairly closely related – we might want the user to drag and drop those drunken party photos into the browser for manipulation in a photo booth type app (airbrush me out of there… or maybe just add some clothes) before uploading, for instance.

Here’s what that looks like:

    /**
     * Deferred wrapper for file reader.
     * @param read_method
     * @param file
     * @returns {Object} promise The Deferred promise object
     * @constructor
     */
    function DeferReader(read_method, file){
        this.defer = $.Deferred();
        this.reader = new FileReader();
        this.file = file;
        this.read_method = read_method;
    this.listen();
    this.reader[read_method](file);
    return this.defer.promise();
}
/**
 * Listen for the various events of interest on the file reader, and return notification or resolution
 * to deferred as appropriate.
 */
DeferReader.prototype.listen = function(){
    var that = this;
    this.reader.addEventListener('error', function(event){
        var err = event.target.error,
            errCode = event.target.error.code,
            errMsg = 'Error attempting to read file "'+this.file.name+'": ';
        switch(errCode)
        {
            case err.NOT_FOUND_ERR:
                errMsg += "File could not be found.";
                break;
            case err.NOT_READABLE_ERR:
                errMsg += "File is not readable.";
                break;
            case err.ABORT_ERR:
                errMsg += "File read was aborted.";
                break;
            default:
                errMsg += "An unexpected error occurred.";
                break;
        }
        that.defer.reject({state:Hup.state.FILE_READ_ERROR, error:errMsg});
    }, false);
    this.reader.addEventListener('progress', function(event){
        if (event.lengthComputable)
        {
            that.defer.notify({state:Hup.state.FILE_READ_PROGRESS, file_name:that.file.name,
                progress:(event.loaded/event.total)});
        }
    });
    this.reader.addEventListener('loadend', function(event){
        if (event.target.readyState == FileReader.DONE)
        {
            that.defer.resolve({state:Hup.state.FILE_READ_FINISHED,
                file_name:that.file.name, file_size:that.file.size, file_type:that.file.type,
                read_method:that.read_method, read_result:event.target.result});
        }
    }, false);
};

Pretty simple (you’ll notice I’m using the promises here to wrap around the callbacks that FileReader offer us – we’re doing the same with the xhr upload). We set up our reader with a file read method (which will determine what kind of result we’re given.

We can readAsText (text files, obviously), readAsDataURL (images are a good candidate for being read this way – we’ll be returned a data url with the contents encoded as base64), readAsBinaryString (which will return us a string with the binary contents encoded – we can get the bytes values by getting the char codes of each character in the string) or readAsArrayBuffer (see Array Buffer spec).

The reader will call back to use on progress (useful for a progress bar, for instance) on the file read, and when finished loading, with the result of our read (which will be encoded as we’ve specified with our read method).

All Together Now

So, been teasing about that plugin, and pulling it apart should be a good way to get started on learning how this all works; and if you want to forego that, just use the plugin! Keep in mind, the plugin doesn’t offer any UI interaction – it just covers reading a file or uploading it, and returns the results of these operations to the element(s) the plugin is called on – you’ll need/want to build your UI on top of that.

Usage, the plugin, a test page and an example php script to receive and assemble chunked uploads can be found at the github repo.

NOTE: The code below is OUT OF DATE – see the GITHUB REPO for the up-to-date code.

Meanwhile, here’s just the plugin:

/**
 * Copyright (c) 2013 Christopher Keefer. All Rights Reserved.
 *
 * jQuery plugin for reading in files or uploading them with the HTML5 file api and xhr2.
 */
(function($){
    /**
     * Construct html5 reader/uploader.
     * @param {Object} options
     * @constructor
     */
    function Hup(options){
        this.init(options);
    }
/**
 * Set options, listen for events on input element that indicate we should read/upload selected file(s).
 * @param options
 */
Hup.prototype.init = function(options)
{
    this.options = $.extend({
        async:true, // Whether to send this file asynchronously
        chunked:true, // Whether to send the file in chunks
        chunk_size:1048576, // Size of each chunk (default 1024*1024)
        input:'', // Input element
        make_dnd:false, // Whether to make the input element handle drag and drop - auto-true if not file input
        read_method:'readAsDataURL', // the read method to use for reading in the file - one of
        // readAsText, readAsBinaryString, readAsDataURL or readAsArrayBuffer
        type:'PUT', // Type of request to use for uploading
        url:false // Url endpoint to send file to - if not specified or false, we read the file and return it
    }, options);
    this.input = $(this.options.input);
    var that = this;
    if (this.options.make_dnd || !this.isFileInput(this.input))
    {
        this.options.make_dnd = true;
        this.input.off('dragover').on('dragover', function(event){
            event = event.originalEvent;
            that.handleDragover(event);
        });
    }
    this.input.off('drop change').on('drop change', function(event){
        event = event.originalEvent;
        that.handleSelect(event);
    });
}
/**
 * Return whether the passed element is an input of type file.
 * @param input Element to check.
 * @returns {boolean}
 */
Hup.prototype.isFileInput = function(input){
    return (input[0].tagName === 'INPUT' &amp;amp;&amp;amp; input[0].getAttribute('type').indexOf('file') !== -1);
};
/**
 * Handle the dragging of file(s) to a target, preventing the rejection of the dragover.
 * @param event
 */
Hup.prototype.handleDragover = function(event){
    event.preventDefault();
    event.stopPropagation();
    event.dataTransfer.dropEffect = 'copy';
};
/**
 * Handle the selection of files to upload via an input or drag and drop to a target.
 * @param event
 */
Hup.prototype.handleSelect = function(event){
    var files;
    if (this.options.make_dnd)
    {
        event.preventDefault();
        event.stopPropagation();
        files = event.dataTransfer.files;
    }
    else
    {
        files = event.target.files;
    }
    if (!files.length)
    {
        this.input.trigger(Hup.state.FILE_LIST_ERROR, {state:Hup.state.FILE_LIST_ERROR,
            error:'No files found in file list; no files were selected.'});
        return;
    }
    this.input.trigger(Hup.state.FILE_LIST_LOADED, {state:Hup.state.FILE_LIST_LOADED, files:files});
    this.processFiles(files, this.options.url);
};
/**
 * Process the files in the fileList, uploading them if a url is specified, otherwise reading them into
 * memory and passing them on to be used in the browser.
 * @param files
 * @param upload
 */
Hup.prototype.processFiles = function(files, upload){
    var that = this,
        processed = 0;
    for (var i=0, len = files.length; i &amp;lt; len; i++)
    {
        var fprocess = (upload) ? new DeferXhr(this.options, files[i]) :
            new DeferReader(this.options.read_method, files[i]);
        fprocess.progress(function(progress){
            that.input.trigger(progress.state, progress);
        }).done(function(res){
            that.input.trigger(res.state, res);
            processed++;
            if (processed == len)
                that.input.trigger((upload) ? Hup.state.FILE_UPLOAD_ALL : Hup.state.FILE_READ_ALL ,
                    {state:(upload) ? Hup.state.FILE_UPLOAD_ALL : Hup.state.FILE_READ_ALL, files:len});
        }).fail(function(res)
        {
            that.input.trigger(res.state, res);
        });
    }
};
/**
 * Custom events we'll trigger on our input element at the appropriate times.
 * @type {{FILE_LIST_ERROR: string, FILE_LIST_LOADED: string, FILE_READ_ERROR: string,
 * FILE_READ_PROGRESS: string, FILE_READ_FINISHED: string, FILE_READ_ALL: string,
 * FILE_UPLOAD_ERROR: string, FILE_UPLOAD_PROGRESS: string, FILE_UPLOAD_PAUSE: string,
 * FILE_UPLOAD_RESUME: string, FILE_UPLOAD_FINISHED: string, FILE_UPLOAD_ALL: string}}
 */
Hup.state = {
    FILE_LIST_ERROR:'fileListError',
    FILE_LIST_LOADED:'fileListLoaded',
    FILE_READ_ERROR:'fileReadError',
    FILE_READ_PROGRESS:'fileReadProgress',
    FILE_READ_FINISHED:'fileReadFinished',
    FILE_READ_ALL:'fileReadAll',
    FILE_UPLOAD_ERROR:'fileUploadError',
    FILE_UPLOAD_PROGRESS:'fileUploadProgress',
    FILE_UPLOAD_PAUSE:'fileUploadPause',
    FILE_UPLOAD_RESUME:'fileUploadResume',
    FILE_UPLOAD_FINISHED:'fileUploadFinished',
    FILE_UPLOAD_ALL:'fileUploadAll'
};
/**
 * Deferred wrapper for xhr upload.
 * @param options
 * @param file
 * @returns {Object} promise The deferred promise object.
 * @constructor
 */
function DeferXhr(options, file){
    var that = this;
    this.defer = $.Deferred();
    this.file = file;
    this.options = options;
    this.paused = false;
    this.progress = 0;
    this.time = {start:0, end:0, speed:0}; // Speed is measured in bytes per second
    this.xhr = new XMLHttpRequest();
    if (this.options.chunked)
    {
        this.start = 0;
        this.end = Math.min(this.start+this.options.chunk_size, this.file.size);
    }
    this.xhr.addEventListener('load', function(){that.complete();}, false);
    this.xhr.upload.addEventListener('progress', function(event){that.uploadProgress(event);}, false);
    this.xhr.upload.addEventListener('error', function(event){that.uploadError(event);}, false);
    this.upload();
    return this.defer.promise();
}
/**
 * Carry out the xhr upload, optionally chunked.
 */
DeferXhr.prototype.upload = function(){
    this.time.start = +new Date();
    this.xhr.open(this.options.type, this.options.url, this.options.async);
    this.xhr.setRequestHeader('Accept', 'application/json');
    this.xhr.setRequestHeader('X-File-Name', this.file.name);
    this.xhr.setRequestHeader('X-File-Type', this.file.type);
    if (this.options.chunked)
    {
        this.xhr.overrideMimeType('application/octet-stream');
        this.xhr.setRequestHeader('Content-Range', 'bytes '+this.start+"-"+this.end+"/"+this.file.size);
        this.xhr.send(this.file.slice(this.start, this.end));
    }
    else
    {
        this.xhr.overrideMimeType((this.file.type || 'application/octet-stream'));
        this.xhr.send(this.file);
    }
}
/**
 * Report on the upload progress, as a number between 0 and 1, modifying the progress if we're uploading a
 * file in chunks to report on the progress as a percentage of file upload and total chunks uploaded.
 * @param event
 */
DeferXhr.prototype.uploadProgress = function(event){
    if (event.lengthComputable)
    {
        this.progress = (event.loaded/event.total);
        if (this.options.chunked)
        {
            this.progress *= (this.end/this.file.size);
        }
        this.time.end = +new Date();
        this.time.speed = (this.file.size*this.progress)/(this.time.end-this.time.start)*1000;
        console.log('time:', this.time.end-this.time.start, 'speed:', this.time.speed);
        this.defer.notify({state:Hup.state.FILE_UPLOAD_PROGRESS, file_name:this.file.name, speed:this.time.speed,
            progress:this.progress});
    }
};
DeferXhr.prototype.uploadError = function(event){
    this.defer.reject({state:Hup.state.FILE_UPLOAD_ERROR, error:event});
}
/**
 * Called when we've completed an upload (full file or chunk). If full file, or we've reached the last chunk,
 * the upload is complete. Otherwise, we calculate the next chunk offsets and, if the upload isn't paused,
 * upload it.
 */
DeferXhr.prototype.complete = function(){
    this.time.end = +new Date();
    if (!this.options.chunked || this.end == this.file.size)
    {
        this.uploadComplete();
        return;
    }
    this.defer.notify({state:Hup.state.FILE_UPLOAD_PROGRESS, file_name:this.file.name,
        response:this.parseResponse(this.xhr.responseText), progress:this.progress});
    this.start = this.end;
    this.end = Math.min(this.start+this.options.chunk_size, this.file.size);
    if (!this.paused)
    {
        this.upload();
    }
};
/**
 * Called when the full file has been uploaded.
 */
DeferXhr.prototype.uploadComplete = function(){
    this.defer.resolve({state:Hup.state.FILE_UPLOAD_FINISHED, file_name:this.file.name,
        file_size:this.file.size, file_type:this.file.type,
        response:this.parseResponse(this.xhr.responseText)});
};
/**
 * Try to parse the response as a JSON, and on failure return the error and the plaintext.
 * @param response
 * @returns {Object}
 */
DeferXhr.prototype.parseResponse = function(response)
{
    var response;
    try{
        response = JSON.parse(this.xhr.responseText);
    }catch(e){
        response = {error:e, text:this.xhr.responseText};
    }
    return response;
}
/**
 * Pause the upload (works for chunked uploads only).
 */
DeferXhr.prototype.pause = function(){
    this.paused = true;
    this.defer.notify({state:Hup.state.FILE_UPLOAD_PAUSE, current_range:{start:this.start, end:this.end,
        total:this.file.size}});
}
/**
 * Resume the upload (works for chunked uploads only).
 */
DeferXhr.prototype.resume = function(){
    if (this.paused)
    {
        this.paused = false;
        this.defer.notify({state:Hup.state.FILE_UPLOAD_RESUME, current_range:{start:this.start, end:this.end,
            total:this.file.size}});
        this.upload();
    }
}
/**
 * Deferred wrapper for file reader.
 * @param read_method
 * @param file
 * @returns {Object} promise The Deferred promise object
 * @constructor
 */
function DeferReader(read_method, file){
    this.defer = $.Deferred();
    this.reader = new FileReader();
    this.file = file;
    this.read_method = read_method;
    this.listen();
    this.reader[read_method](file);
    return this.defer.promise();
}
/**
 * Listen for the various events of interest on the file reader, and return notification or resolution
 * to deferred as appropriate.
 */
DeferReader.prototype.listen = function(){
    var that = this;
    this.reader.addEventListener('error', function(event){
        var err = event.target.error,
            errCode = event.target.error.code,
            errMsg = 'Error attempting to read file "'+this.file.name+'": ';
        switch(errCode)
        {
            case err.NOT_FOUND_ERR:
                errMsg += "File could not be found.";
                break;
            case err.NOT_READABLE_ERR:
                errMsg += "File is not readable.";
                break;
            case err.ABORT_ERR:
                errMsg += "File read was aborted.";
                break;
            default:
                errMsg += "An unexpected error occurred.";
                break;
        }
        that.defer.reject({state:Hup.state.FILE_READ_ERROR, error:errMsg});
    }, false);
    this.reader.addEventListener('progress', function(event){
        if (event.lengthComputable)
        {
            that.defer.notify({state:Hup.state.FILE_READ_PROGRESS, file_name:that.file.name,
                progress:(event.loaded/event.total)});
        }
    });
    this.reader.addEventListener('loadend', function(event){
        if (event.target.readyState == FileReader.DONE)
        {
            that.defer.resolve({state:Hup.state.FILE_READ_FINISHED,
                file_name:that.file.name, file_size:that.file.size, file_type:that.file.type,
                read_method:that.read_method, read_result:event.target.result});
        }
    }, false);
};
/**
 * Entry point for calling the reader/uploader, with the element to be used as input specified.
 * Usage:
 * $('#input').hup({options}).on('events') --OR--
 * $('.inputs').hup({options}).on('events')
 * @param options
 * @returns {Hup} Promise deferred from Hup.
 */
$.fn.hup = function(options){
    var options = (options || {});
    return this.each(function(){
        options.input = this;
        var $this = $(this),
            hup = $this.data('hup');
        if (!hup)
        {
            $this.data('hup', new Hup(options));
        }
        else if (hup instanceof Hup)
        {
            hup.init(options);
        }
    });
};
})(jQuery);

+ more

Accurate Timing

Accurate Timing

In many tasks we need to do something at given intervals of time. The most obvious ways may not give you the best results. Time? Meh. The most basic tasks that don't have what you might call CPU-scale time requirements can be handled with the usual language and...

read more