Ajax Upload Part II: XHR2 (and FileReader)


So, the client has told you their users should be able to upload their drunken party pictures for all the internet to see. “We want the very best experience possible,” they tell you. “Simple, seamless – maybe using that new html5 thing I’ve heard so much about.”

You choke back a little bit of hope. You’ve been disappointed so often. Trying hard to sound non-chalant, you say “only modern browsers, right?” You hold your breath. They nod, and on the outside, you nod back. On the inside, you giggle. No flash, no iframe hacks – just ajax, as God (well, the W3C) intended. Ah, beautiful.

As promised in part I, we’re going take a quick walk through what it takes to upload a file with ajax. We’re also going to take a look at reading a file into memory to play with in your app.

Inputs and Drop Targets

So, let’s say we have two images of some hip twenty-somethings having a good time, drunk1.png and drunk2.jpg. Our job is to get these files from the user’s computer to our server as simply as possible. The standard input is a familiar way of doing this, but we might also want to take advantage of the new drag and drop capabilities in modern browsers. So, we could have something like:

<form>
<input id="hupinput" type="file" />
</form>

or:

<div id="hupdiv"></div>

and then just define some styles that will give us a properly sized drop target.

Now, the drag and drop api in html5 is a little strange. We need to cancel the default action and stop propagation for dragover and drop in order to allow something to be drag and dropped. The idea seems to be that the default action is to not allow anything to be dragged and dropped. You’ll just have to swing with it.

$('inputname').off('dragover').on('dragover', function(event){
    event = event.originalEvent;
    event.preventDefault();
    event.stopPropagation();
});

var files;
$('inputname').off('drop change').on('drop change', function(event){
    event = event.originalEvent;
    event.preventDefault();
    event.stopPropagation();
    // if drop and drag target
    files = event.dataTransfer.files;
    // or if an input
    files = event.target.files;
});

End result of this is that we’ll end up with a FileList object that acts like an array, with each element being a reference to a file. We can then iterate through it to upload or read our files.

Uploading

XHR2 introduces some new features, with the ones of interest being the new upload attribute on xhr, which returns an XMLHttpRequestUpload object. This exposes the progress of an upload to us. Otherwise, we’ll be using the standard send() call with an ArrayBuffer of the file to upload it.
Here’s what part of the setup and upload looks like in the plugin we’ll be discussing shortly:

    /**
     * Deferred wrapper for xhr upload.
     * @param options
     * @param file
     * @returns {Object} promise The deferred promise object.
     * @constructor
     */
    function DeferXhr(options, file){
        var that = this;

        this.defer = $.Deferred();
        this.file = file;
        this.options = options;
        this.paused = false;
        this.progress = 0;
        this.xhr = new <a class="zem_slink" title="XMLHttpRequest" href="http://en.wikipedia.org/wiki/XMLHttpRequest" target="_blank" rel="wikipedia">XMLHttpRequest</a>();

        if (this.options.chunked)
        {
            this.start = 0;
            this.end = Math.min(this.start+this.options.chunk_size, this.file.size);
        }

        this.xhr.addEventListener('load', function(){that.complete();}, false);
        this.xhr.upload.addEventListener('progress', function(event){that.uploadProgress(event);}, false);
        this.xhr.upload.addEventListener('error', function(event){that.uploadError(event);}, false);

        this.upload();

        return this.defer.promise();
    }

    /**
     * Carry out the xhr upload, optionally chunked.
     */
    DeferXhr.prototype.upload = function(){

        this.xhr.open(this.options.type, this.options.url, this.options.async);
        this.xhr.setRequestHeader('Accept', 'application/json');
        this.xhr.setRequestHeader('X-File-Name', this.file.name);
        this.xhr.setRequestHeader('X-File-Type', this.file.type);

        if (this.options.chunked)
        {
            this.xhr.overrideMimeType('application/octet-stream');
            this.xhr.setRequestHeader('Content-Range', 'bytes '+this.start+"-"+this.end+"/"+this.file.size);
            this.xhr.send(this.file.slice(this.start, this.end));
        }
        else
        {
            this.xhr.overrideMimeType((this.file.type || 'application/octet-stream'));
            this.xhr.send(this.file);
        }
    }

The main things to notice are that if we’re sending a file ‘chunked’ (that is, in pieces of a pre-determined size – this is a good way to get around, say, php upload file size limits or allow for pausing and resuming a file upload, by the way) we’re using the slice() method on our file to get an ArrayBuffer of the appropriate size out of the buffer representing the whole file. This also saves us from needing to read in the whole file before sending it.

In order to know what’s happening with our file upload, we just need to listen (as noted above) on the load, progress and errors events of xhr and xhr.upload.

Reading

This is probably one of my favourite new things to play with in javascript – the FileReader api (which is a subset of the File api). Get a file from the local user filesystem and work with it in the browser. Yay!

Its exciting, because coming from a C and Java background, getting a file from the local system wasn’t ever a problem, and there’s been numerous times in my web dev endeavours when it has been. Ever needed to use that ugly hack where you upload a file to a server and bounce it immediately back to the client, just so you can have access to some image or text file from the user? Yeah, good riddance to that.

Anyways, our plugin is going to do that to, because the two concepts are fairly closely related – we might want the user to drag and drop those drunken party photos into the browser for manipulation in a photo booth type app (airbrush me out of there… or maybe just add some clothes) before uploading, for instance.

Here’s what that looks like:

    /**
     * Deferred wrapper for file reader.
     * @param read_method
     * @param file
     * @returns {Object} promise The Deferred promise object
     * @constructor
     */
    function DeferReader(read_method, file){
        this.defer = $.Deferred();
        this.reader = new FileReader();
        this.file = file;
        this.read_method = read_method;

        this.listen();
        this.reader[read_method](file);

        return this.defer.promise();
    }

    /**
     * Listen for the various events of interest on the file reader, and return notification or resolution
     * to deferred as appropriate.
     */
    DeferReader.prototype.listen = function(){
        var that = this;

        this.reader.addEventListener('error', function(event){
            var err = event.target.error,
                errCode = event.target.error.code,
                errMsg = 'Error attempting to read file "'+this.file.name+'": ';
            switch(errCode)
            {
                case err.NOT_FOUND_ERR:
                    errMsg += "File could not be found.";
                    break;
                case err.NOT_READABLE_ERR:
                    errMsg += "File is not readable.";
                    break;
                case err.ABORT_ERR:
                    errMsg += "File read was aborted.";
                    break;
                default:
                    errMsg += "An unexpected error occurred.";
                    break;
            }
            that.defer.reject({state:Hup.state.FILE_READ_ERROR, error:errMsg});
        }, false);

        this.reader.addEventListener('progress', function(event){
            if (event.lengthComputable)
            {
                that.defer.notify({state:Hup.state.FILE_READ_PROGRESS, file_name:that.file.name,
                    progress:(event.loaded/event.total)});
            }
        });

        this.reader.addEventListener('loadend', function(event){
            if (event.target.readyState == FileReader.DONE)
            {
                that.defer.resolve({state:Hup.state.FILE_READ_FINISHED,
                    file_name:that.file.name, file_size:that.file.size, file_type:that.file.type,
                    read_method:that.read_method, read_result:event.target.result});
            }
        }, false);
    };

Pretty simple (you’ll notice I’m using the promises here to wrap around the callbacks that FileReader offer us – we’re doing the same with the xhr upload). We set up our reader with a file read method (which will determine what kind of result we’re given.

We can readAsText (text files, obviously), readAsDataURL (images are a good candidate for being read this way – we’ll be returned a data url with the contents encoded as base64), readAsBinaryString (which will return us a string with the binary contents encoded – we can get the bytes values by getting the char codes of each character in the string) or readAsArrayBuffer (see Array Buffer spec).

The reader will call back to use on progress (useful for a progress bar, for instance) on the file read, and when finished loading, with the result of our read (which will be encoded as we’ve specified with our read method).

All Together Now

So, been teasing about that plugin, and pulling it apart should be a good way to get started on learning how this all works; and if you want to forego that, just use the plugin! Keep in mind, the plugin doesn’t offer any UI interaction – it just covers reading a file or uploading it, and returns the results of these operations to the element(s) the plugin is called on – you’ll need/want to build your UI on top of that.

Usage, the plugin, a test page and an example php script to receive and assemble chunked uploads can be found at the github repo.

NOTE: The code below is OUT OF DATE – see the GITHUB REPO for the up-to-date code.

Meanwhile, here’s just the plugin:

/**
 * Copyright (c) 2013 Christopher Keefer. All Rights Reserved.
 *
 * jQuery plugin for reading in files or uploading them with the HTML5 file api and xhr2.
 */
(function($){
    /**
     * Construct html5 reader/uploader.
     * @param {Object} options
     * @constructor
     */
    function Hup(options){
        this.init(options);
    }

    /**
     * Set options, listen for events on input element that indicate we should read/upload selected file(s).
     * @param options
     */
    Hup.prototype.init = function(options)
    {
        this.options = $.extend({
            async:true, // Whether to send this file asynchronously
            chunked:true, // Whether to send the file in chunks
            chunk_size:1048576, // Size of each chunk (default 1024*1024)
            input:'', // Input element
            make_dnd:false, // Whether to make the input element handle drag and drop - auto-true if not file input
            read_method:'readAsDataURL', // the read method to use for reading in the file - one of
            // readAsText, readAsBinaryString, readAsDataURL or readAsArrayBuffer
            type:'PUT', // Type of request to use for uploading
            url:false // Url endpoint to send file to - if not specified or false, we read the file and return it
        }, options);

        this.input = $(this.options.input);

        var that = this;
        if (this.options.make_dnd || !this.isFileInput(this.input))
        {
            this.options.make_dnd = true;
            this.input.off('dragover').on('dragover', function(event){
                event = event.originalEvent;
                that.handleDragover(event);
            });
        }
        this.input.off('drop change').on('drop change', function(event){
            event = event.originalEvent;
            that.handleSelect(event);
        });
    }

    /**
     * Return whether the passed element is an input of type file.
     * @param input Element to check.
     * @returns {boolean}
     */
    Hup.prototype.isFileInput = function(input){
        return (input[0].tagName === 'INPUT' && input[0].getAttribute('type').indexOf('file') !== -1);
    };

    /**
     * Handle the dragging of file(s) to a target, preventing the rejection of the dragover.
     * @param event
     */
    Hup.prototype.handleDragover = function(event){
        event.preventDefault();
        event.stopPropagation();
        event.dataTransfer.dropEffect = 'copy';
    };

    /**
     * Handle the selection of files to upload via an input or drag and drop to a target.
     * @param event
     */
    Hup.prototype.handleSelect = function(event){
        var files;

        if (this.options.make_dnd)
        {
            event.preventDefault();
            event.stopPropagation();
            files = event.dataTransfer.files;
        }
        else
        {
            files = event.target.files;
        }
        if (!files.length)
        {
            this.input.trigger(Hup.state.FILE_LIST_ERROR, {state:Hup.state.FILE_LIST_ERROR,
                error:'No files found in file list; no files were selected.'});
            return;
        }
        this.input.trigger(Hup.state.FILE_LIST_LOADED, {state:Hup.state.FILE_LIST_LOADED, files:files});

        this.processFiles(files, this.options.url);
    };

    /**
     * Process the files in the fileList, uploading them if a url is specified, otherwise reading them into
     * memory and passing them on to be used in the browser.
     * @param files
     * @param upload
     */
    Hup.prototype.processFiles = function(files, upload){
        var that = this,
            processed = 0;

        for (var i=0, len = files.length; i < len; i++)
        {
            var fprocess = (upload) ? new DeferXhr(this.options, files[i]) :
                new DeferReader(this.options.read_method, files[i]);

            fprocess.progress(function(progress){
                that.input.trigger(progress.state, progress);
            }).done(function(res){
                that.input.trigger(res.state, res);
                processed++;
                if (processed == len)
                    that.input.trigger((upload) ? Hup.state.FILE_UPLOAD_ALL : Hup.state.FILE_READ_ALL ,
                        {state:(upload) ? Hup.state.FILE_UPLOAD_ALL : Hup.state.FILE_READ_ALL, files:len});
            }).fail(function(res)
            {
                that.input.trigger(res.state, res);
            });
        }
    };

    /**
     * Custom events we'll trigger on our input element at the appropriate times.
     * @type {{FILE_LIST_ERROR: string, FILE_LIST_LOADED: string, FILE_READ_ERROR: string,
     * FILE_READ_PROGRESS: string, FILE_READ_FINISHED: string, FILE_READ_ALL: string,
     * FILE_UPLOAD_ERROR: string, FILE_UPLOAD_PROGRESS: string, FILE_UPLOAD_PAUSE: string,
     * FILE_UPLOAD_RESUME: string, FILE_UPLOAD_FINISHED: string, FILE_UPLOAD_ALL: string}}
     */
    Hup.state = {
        FILE_LIST_ERROR:'fileListError',
        FILE_LIST_LOADED:'fileListLoaded',
        FILE_READ_ERROR:'fileReadError',
        FILE_READ_PROGRESS:'fileReadProgress',
        FILE_READ_FINISHED:'fileReadFinished',
        FILE_READ_ALL:'fileReadAll',
        FILE_UPLOAD_ERROR:'fileUploadError',
        FILE_UPLOAD_PROGRESS:'fileUploadProgress',
        FILE_UPLOAD_PAUSE:'fileUploadPause',
        FILE_UPLOAD_RESUME:'fileUploadResume',
        FILE_UPLOAD_FINISHED:'fileUploadFinished',
        FILE_UPLOAD_ALL:'fileUploadAll'
    };

    /**
     * Deferred wrapper for xhr upload.
     * @param options
     * @param file
     * @returns {Object} promise The deferred promise object.
     * @constructor
     */
    function DeferXhr(options, file){
        var that = this;

        this.defer = $.Deferred();
        this.file = file;
        this.options = options;
        this.paused = false;
        this.progress = 0;
        this.time = {start:0, end:0, speed:0}; // Speed is measured in bytes per second
        this.xhr = new XMLHttpRequest();

        if (this.options.chunked)
        {
            this.start = 0;
            this.end = Math.min(this.start+this.options.chunk_size, this.file.size);
        }

        this.xhr.addEventListener('load', function(){that.complete();}, false);
        this.xhr.upload.addEventListener('progress', function(event){that.uploadProgress(event);}, false);
        this.xhr.upload.addEventListener('error', function(event){that.uploadError(event);}, false);

        this.upload();

        return this.defer.promise();
    }

    /**
     * Carry out the xhr upload, optionally chunked.
     */
    DeferXhr.prototype.upload = function(){
        this.time.start = +new Date();

        this.xhr.open(this.options.type, this.options.url, this.options.async);
        this.xhr.setRequestHeader('Accept', 'application/json');
        this.xhr.setRequestHeader('X-File-Name', this.file.name);
        this.xhr.setRequestHeader('X-File-Type', this.file.type);

        if (this.options.chunked)
        {
            this.xhr.overrideMimeType('application/octet-stream');
            this.xhr.setRequestHeader('Content-Range', 'bytes '+this.start+"-"+this.end+"/"+this.file.size);
            this.xhr.send(this.file.slice(this.start, this.end));
        }
        else
        {
            this.xhr.overrideMimeType((this.file.type || 'application/octet-stream'));
            this.xhr.send(this.file);
        }
    }

    /**
     * Report on the upload progress, as a number between 0 and 1, modifying the progress if we're uploading a
     * file in chunks to report on the progress as a percentage of file upload and total chunks uploaded.
     * @param event
     */
    DeferXhr.prototype.uploadProgress = function(event){
        if (event.lengthComputable)
        {
            this.progress = (event.loaded/event.total);
            if (this.options.chunked)
            {
                this.progress *= (this.end/this.file.size);
            }
            this.time.end = +new Date();
            this.time.speed = (this.file.size*this.progress)/(this.time.end-this.time.start)*1000;
            console.log('time:', this.time.end-this.time.start, 'speed:', this.time.speed);
            this.defer.notify({state:Hup.state.FILE_UPLOAD_PROGRESS, file_name:this.file.name, speed:this.time.speed,
                progress:this.progress});
        }
    };

    DeferXhr.prototype.uploadError = function(event){
        this.defer.reject({state:Hup.state.FILE_UPLOAD_ERROR, error:event});
    }

    /**
     * Called when we've completed an upload (full file or chunk). If full file, or we've reached the last chunk,
     * the upload is complete. Otherwise, we calculate the next chunk offsets and, if the upload isn't paused,
     * upload it.
     */
    DeferXhr.prototype.complete = function(){
        this.time.end = +new Date();
        if (!this.options.chunked || this.end == this.file.size)
        {
            this.uploadComplete();
            return;
        }

        this.defer.notify({state:Hup.state.FILE_UPLOAD_PROGRESS, file_name:this.file.name,
            response:this.parseResponse(this.xhr.responseText), progress:this.progress});

        this.start = this.end;
        this.end = Math.min(this.start+this.options.chunk_size, this.file.size);

        if (!this.paused)
        {
            this.upload();
        }
    };

    /**
     * Called when the full file has been uploaded.
     */
    DeferXhr.prototype.uploadComplete = function(){
        this.defer.resolve({state:Hup.state.FILE_UPLOAD_FINISHED, file_name:this.file.name,
            file_size:this.file.size, file_type:this.file.type,
            response:this.parseResponse(this.xhr.responseText)});
    };

    /**
     * Try to parse the response as a JSON, and on failure return the error and the plaintext.
     * @param response
     * @returns {Object}
     */
    DeferXhr.prototype.parseResponse = function(response)
    {
        var response;
        try{
            response = JSON.parse(this.xhr.responseText);
        }catch(e){
            response = {error:e, text:this.xhr.responseText};
        }
        return response;
    }

    /**
     * Pause the upload (works for chunked uploads only).
     */
    DeferXhr.prototype.pause = function(){
        this.paused = true;
        this.defer.notify({state:Hup.state.FILE_UPLOAD_PAUSE, current_range:{start:this.start, end:this.end,
            total:this.file.size}});
    }

    /**
     * Resume the upload (works for chunked uploads only).
     */
    DeferXhr.prototype.resume = function(){
        if (this.paused)
        {
            this.paused = false;
            this.defer.notify({state:Hup.state.FILE_UPLOAD_RESUME, current_range:{start:this.start, end:this.end,
                total:this.file.size}});
            this.upload();
        }
    }

    /**
     * Deferred wrapper for file reader.
     * @param read_method
     * @param file
     * @returns {Object} promise The Deferred promise object
     * @constructor
     */
    function DeferReader(read_method, file){
        this.defer = $.Deferred();
        this.reader = new FileReader();
        this.file = file;
        this.read_method = read_method;

        this.listen();
        this.reader[read_method](file);

        return this.defer.promise();
    }

    /**
     * Listen for the various events of interest on the file reader, and return notification or resolution
     * to deferred as appropriate.
     */
    DeferReader.prototype.listen = function(){
        var that = this;

        this.reader.addEventListener('error', function(event){
            var err = event.target.error,
                errCode = event.target.error.code,
                errMsg = 'Error attempting to read file "'+this.file.name+'": ';
            switch(errCode)
            {
                case err.NOT_FOUND_ERR:
                    errMsg += "File could not be found.";
                    break;
                case err.NOT_READABLE_ERR:
                    errMsg += "File is not readable.";
                    break;
                case err.ABORT_ERR:
                    errMsg += "File read was aborted.";
                    break;
                default:
                    errMsg += "An unexpected error occurred.";
                    break;
            }
            that.defer.reject({state:Hup.state.FILE_READ_ERROR, error:errMsg});
        }, false);

        this.reader.addEventListener('progress', function(event){
            if (event.lengthComputable)
            {
                that.defer.notify({state:Hup.state.FILE_READ_PROGRESS, file_name:that.file.name,
                    progress:(event.loaded/event.total)});
            }
        });

        this.reader.addEventListener('loadend', function(event){
            if (event.target.readyState == FileReader.DONE)
            {
                that.defer.resolve({state:Hup.state.FILE_READ_FINISHED,
                    file_name:that.file.name, file_size:that.file.size, file_type:that.file.type,
                    read_method:that.read_method, read_result:event.target.result});
            }
        }, false);
    };

    /**
     * Entry point for calling the reader/uploader, with the element to be used as input specified.
     * Usage:
     * $('#input').hup({options}).on('events') --OR--
     * $('.inputs').hup({options}).on('events')
     * @param options
     * @returns {Hup} Promise deferred from Hup.
     */
    $.fn.hup = function(options){
        var options = (options || {});
        return this.each(function(){
            options.input = this;
            var $this = $(this),
                hup = $this.data('hup');
            if (!hup)
            {
                $this.data('hup', new Hup(options));
            }
            else if (hup instanceof Hup)
            {
                hup.init(options);
            }
        });
    };
})(jQuery);
Christopher Keefer

Christopher Keefer

Christopher Keefer is a Senior Software Engineer at Art & Logic. He generally spends his spare time on the computer too, so there isn't much hope for him.
Christopher Keefer

Latest posts by Christopher Keefer (see all)

Tags:

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

11 Comments

  1. Neil

    While setting the options, what’s the use of ‘input’?

    • sanemethod

      If you want to use HUp directly, instead of via the jquery plugin, the input parameter is where you would plugin what element you want to use as your input. If you’re using the jquery plugin, you can safely ignore that parameter, as it will be set by the plugin based on the elements you’ve selected.

      • Neil

        Cool thanks. Got it! Great script. 😉

  2. Neil

    I’m finding it hard to understand why you used `that = this;` Tried to understand from elsewhere, but wasn’t similar to this.
    Also, if I want to send the file name or any other data along with each file upload, how do I do it, and what’s the php code to read the same?

    • sanemethod

      The local variable ‘that’ is being used to reference ‘this’ – that is, the enclosing function. When inside of a function within a function (such as a callback like the .done block of an $.ajax call), ‘this’ will refer to the innermost function. jQuery also sets the context (‘this’) to the DOM element being referenced inside of event callbacks, $.each blocks, etc. If we want to reference the containing function, we need a reference to it other than ‘this’ – thus, assigning this to ‘that’ in the enclosing function. We could set the context as we desire using .call or .apply, but this is simpler and adequate for most situations.

      Take a look at the github repo (https://github.com/SaneMethod/HUp) for some example php code indicating how to handle non-standard file headers, how to reassemble a chunked upload, etc. Be aware that this sample code is meant for demonstration purposes – you’ll want to do some further work testing and securing it.

  3. John

    How can I detect filesize of all selected files to count overall progress? For example, if I have ideal speed for upload, choose two files, one with 10MB size, second with 2 MB size, start upload, and if both files are paralelly uploaded with the same speed, so when the second file is fully uploaded (2 of 2 MB = 100%), the first file have 2 of 10 MB also uploaded (20%) the final progress should be (2 + 2 = 4MB of 10+2 MB = 4 of 12 MB = 33%). I’m able to detect per file progress, but I can’t find how to check this overall progress.

    • sanemethod

      I’m able to detect per file progress, but I can’t find how to check this overall progress.

      The simplest answer: add the per-file progress and divide by the number of files to be uploaded.
      That is:
      file1 progress: 100%
      file2 progress: 55%
      file3 progress: 33%
      Overall progress = (file1progress+file2progress+file3progress)/3

  4. Marty Goldberg

    What about multiple file support?

  5. Alexander

    Thanks for a nice post! I’d prefer something without jQuery, but it’s a good reading anyway. The explanation on why to disable the default drag&drop action is somewhat obscure for a beginner imho. For those who didn’t understand, the reason behind disabling the default action is we are preventing the browser to try to open your file, which you are dropping in. It is it’s default behavior. Instead, we want to prepare that file for uploading and thus we need to disable the default action.

  6. Kiran

    How can have a custom upload button.

  7. RLK

    How do I fire the Pause and Resume events? I get the Hup.state object, but I can’t figure out how to send one of those states to the plugin.

Trackbacks/Pingbacks

  1. Ajax Upload XHR2, Take 2 | - […] Last time we touched this subject, I shared an (admittedly rough) jQuery plugin that allowed you to enjoy HTML5…