Recording Audio & Video with HTML5 (co-starring Meteor)

Filming in ProgressA few weeks ago I got a germ of an idea in my head for a personal web-application that required recording and playing video, something with which I have had very little experience. I have seen how effortless is it to play video with HTML5 so I thought this would be simple. After searching countless sites looking for the HTML5 magic bullet for recording both audio & video, I had pretty much given up.

If you have stumbled upon this article also looking for a way to record audio & video together, you can stop searching now. I can say with fairly strong confidence that such a mechanism does not yet exist (as of the publish date of this article). However, I believe I have a workable solution for the time being.

The full source for the example application is on github.

The most frustrating part is that the MediaStream API provides a nearly effortless way to route video and audio but doesn’t yet support recording the entire stream. The MediaRecorder API  would be perfect but it isn’t currently supported by a major browser. (It is in development for Firefox and Chrome but doesn’t appear to be on the Safari radar. Status for IE is unknown).

So where do we go from here? We have the audio and the video…just not as a single stream. Let’s record and upload them separately and, later, retrieve and play them together. Obviously this isn’t a perfect solution but it should be sufficient for some applications.

I put together a simple example that uses the Whammy WebM encoder (to record & encode the video) and Recorder.js (to record & encode the audio). I used Meteor.js for the framework since it is great for building quick prototypes, especially if you are primarily focused on HTML and JavaScript (as this example is). Since both of the recording classes provide output blob data, a class wrapper around the FileReader class was used to convert the blob data to binary data, which can then be inserted into the MongoDB instance.

The Meteor Project

If you don’t care about understanding the project and just want to see how the html5 audio & video recording works, skip to the “Recording Audio and Video” section.

Even if you haven’t used Meteor before, the code is quite simple and easy to follow. Named mustache templates are used for the front-facing html and JavaScript is used for both the server and client code. Meteor gives you freedom to organize the files and folders of a project how you want so I want to provide an overview of how this particular project is structured.

/lib/router.js

This file contains the configuration and url mapping for the iron-router package. There are three routes defined. The ‘home’ route matches the root path, the ‘record’ route matches /record/, which shows the ‘record’ template, and the ‘showVideo’ route, which matches /video/:_id where “:_id” matches the id variable for a specific user.

The most important code to note here is in the ‘showVideo’ route:

this.route('showVideo',
{
    path: '/video/:_id',
    template: 'showVideo',
    data: function () {
        var id = this.params._id;
        console.log("route-showVideo: " + id);
        var user = Meteor.users.findOne({ _id: id });
        if (!user) {
            console.log("! no user found for route");
            return null;
        }

        console.log("Finding audio/video for user: " + id);
        var vid = UserVideos.findOne({ userId: id },
            { sort: { save_date: -1 }});
        var aud = UserAudios.findOne({ userId: id },
            { sort: { save_date: -1 }});

        return {
            audio: aud,
            video: vid,
            userId: id
        }
    }
});

Note that the ‘data’ section finds the video and audio for the user id passed through the url and returns an object containing all of that information, the audio data, the video data, and the userId. This audio and video data will be generated and added to the database from the ‘record’ template, discussed in more detail later.

/lib/models.js

This file defines the models used in this app: UserAudios and UserVideos, both collections that are used to store the audio & video data, respectively, for each user.

/server/publishing.js

Since “auto-publish” is enabled for this Meteor project, this file specifies the data that should be published to the client. Since this is a sample application, it is intentionally liberal.

/client/layout.html

All of the templates of this project are in the client folder. The ‘layout’ and ‘header’ templates are both in layout.html and provide an application layout template and login functionality.

/client/templates.html

This view file contains all of the functional templates for this application. The helper scripts for each template are in matching ‘.js’ files. For example, the ‘home’ template has a matching ‘home.js’ file with template helpers and associated script methods.

home, userVideo: together these templates display a list of all recorded user videos. Scripts for these templates are in home.js.

<template name="home">
    <div id="video-list">
        {{#each userList}}
            {{> userVideo}}
        {{/each}}
    </div>
</template>

<template name="userVideo">
    <div id="{{_id}}" class="video-item">
        {{email}}: <a href="{{pathFor 'showVideo'}}">Video</a>
    </div>
</template>

record: this template displays an html5 video and controls for interacting with the video. The important part of this template is in the script handlers, which are in record.js. See “Recording Audio & Video” below for a description of the actual interactions going on here (the important part).

<template name="home">
    <div id="video-list">
        {{#each userList}}
            {{> userVideo}}
        {{/each}}
    </div>
</template>

<template name="userVideo">
    <div id="{{_id}}" class="video-item">
        {{email}}: <a href="{{pathFor 'showVideo'}}">Video</a>
    </div>
</template>

– showVideo: this template shows a user-recorded video using ‘audio’ and ‘video’ elements.

<template name="showVideo">
    {{onLoad}}
    {{userEmail}}
    {{#if hasVideo}}
        <button id="play-recording">Play</button>
        <video id="review_video" width="320" height="240" src="{{userVideo}}">
        </video>
        <audio hidden="true" id="review_audio" src="{{userAudio}}">
        </audio>
    {{else}}
        {{#if hasUser}}
            Please wait while the video is retrieved.
        {{else}}
            No user found.
        {{/if}}
    {{/if}}
</template>

Recording Audio and Video

The meat of this application, the actual recording and storing of the audio and video, is performed in the ‘record’ template and its associated scripts (in record.js). When the user clicks the ‘start-recording’ button, the event handler calls startRecording which uses the Whammy WebM encoder to record video and Recorder.js audio recorder to record the audio.

Setup (utilizing navigator.getUserMedia)

First let’s take a look at the code that sets up the media. Notice in the ‘record’ template code above, there is a call to the {{onLoad}} template parameter. I’m doing this as a way to get some code to be called as the page is being processed. That template code just calls the setupMedia function, which is the real center of the functionality.

Template.record.onLoad = function () { setupMedia(); };

Let’s example the setupMedia function:

function setupMedia() {
    if (supportsMedia()) {
        audioContext = new AudioContext();

        navigator.getUserMedia(
            {
                video: true,
                audio: true
            },
            function (localMediaStream) {
                // map the camera
                var video = document.getElementById('live_video');
                video.src = window.URL.createObjectURL(localMediaStream);

                // create the canvas & get a 2d context
                videoCanvas = document.createElement('canvas');
                videoContext = videoCanvas.getContext('2d');

                // setup audio recorder
                var audioInput =
                    audioContext.createMediaStreamSource(localMediaStream);
                //audioInput.connect(audioContext.destination);
                // had to replace the above with the following to
                // mute playback (so you don't get feedback)
                var audioGain = audioContext.createGain();
                audioGain.gain.value = 0;
                audioInput.connect(audioGain);
                audioGain.connect(audioContext.destination);

                audioRecorder = new Recorder(audioInput);
                mediaStream = localMediaStream;
                mediaInitialized = true;

                document.getElementById('uploading').hidden = true;
                document.getElementById('media-error').hidden = true;
                document.getElementById('record').hidden = false;
            },
            function (e) {
                console.log('web-cam & microphone not initialized: ', e);
                document.getElementById('media-error').hidden = false;
            }
        );
    }
};

To sum up the important steps of this function:

  • Call navigator.getUserMedia to get the audio and video from the user’s webcam and microphone. Note that the user must accept this or the page will not work.
  • Send the webCam video to the <video id=’live_video’> element’s src so the user can see what is being recorded.
  • Create a canvas element and 2d context, which will be used later to pull & record video frames.
  • Create an AudioContext object (http://www.w3.org/TR/webaudio/#AudioContext-section) and capture the local audio as a stream.
  • Create a Recorder.js object to receive the audio stream. This will be used to record the audio.

A note on feedback: To ensure that we don’t get feedback while recording audio, we create a gain object from the audioContext, set the gain to 0, and put that in the audio stream.

Starting the Recording

Now that we have the video and audio streams routed to some objects that we can access, we need to figure out what to do when the user clicks the ‘start-recording’ button. The handler for that button simply ensures there is a valid user and calls startRecording.

function startRecording() {
    console.log("Begin Recording");

    videoElement = document.getElementById('live_video');
    videoCanvas.width = videoElement.width;
    videoCanvas.height = videoElement.height;

    imageArray = [];

    // do request frames until the user stops recording
    recording = true;
    frameTime = new Date().getTime();
    requestAnimationFrame(recordFrame);

    // begin recording audio
    audioRecorder.record();
}

Simply enough, this just resets the information we have about the video, creates an empty array that will be used to store video frames, calls requestAnimationFrame to begin recording frames of video with the ‘recordFrame’ method, and calls ‘record’ on the Recorder.js audioRecorder created in setupMedia above.

Recording Video

As mentioned, the recordFrame method performs the actual frame saving during recording.

function recordFrame() {
//    console.log("-frame");

    if (recording) {
        var image;
        // draw the video to the context, then get the image data
        var video = document.getElementById('live_video');
        var width = video.width;
        var height = video.height;
        videoContext.drawImage(video, 0, 0, width, height);

        // optionally get the image, do some filtering on it, then
        // put it back to the context
        imageData = videoContext.getImageData(0, 0, width, height);
        // - do some optional image manipulations on imageData
        videoContext.putImageData(imageData, 0, 0);

        var frameDuration = new Date().getTime() - frameTime;

        console.log("duration: " + frameDuration);
        // NOTE: the following attempted to add the frames to the
        // encoder as it was recording but ended up getting saving
        // the frame duration incorrectly.
        //whammyEncoder.add(videoContext, frameDuration);
        imageArray.push(
            {
                duration: frameDuration,
                image: imageData
            });
        frameTime = new Date().getTime();

        // request another frame
        requestAnimationFrame(recordFrame);
    }
    else {
        completeRecording();
    }
}

To sum up this code, we are drawing the current video image to our video context (created in setupMedia), then pulling the image data, then doing some image effects (left out here for simplicity) and putting it back to the context. Lastly, we store off the image data and the amount of time since the last frame was recorded. These two items of data will be used by the encoder when recording is complete.

Note that pushing the image data back to the context could be skipped if no effects are performed. This sample code just shows where you would do it if you wanted to.

The ‘stop-recording’ button handler simply sets the ‘recording’ flag to false, so that the recordFrame method here calls completeRecording, which does the processing and uploading for both audio and video.

Saving and Uploading the Data

We saw above in the startRecording method that we call audioRecorder.record() (on the Recorder.js object) to begin audio recording. Unlike with the video we don’t need to take any action while recording. The handler for the ‘stop-recording’ button sets the ‘recording’ flag to false, which results in a call to the completeRecording function.

function completeRecording() {
    // stop & export the recorder audio
    audioRecorder.stop();

    var user = Meteor.user();
    if (!user) {
        // must be the logged in user
        console.log("completeRecording - NO USER LOGGED IN");
        return;
    }
    console.log("completeRecording: " + user._id);

    document.getElementById('uploading').hidden = false;

    audioRecorder.exportWAV(function (audioBlob) {
        // save to the db
        BinaryFileReader.read(audioBlob, function (err, fileInfo) {
            UserAudios.insert({
                userId: user._id,
                audio: fileInfo,
                save_date: Date.now()
            });
        });
        console.log("Audio uploaded");
    });

    // do the video encoding
    // note: tried doing this in real-time as the frames were requested but
    // the result didn't handle durations correctly.
    var whammyEncoder = new Whammy.Video();
    for (i in imageArray) {
        videoContext.putImageData(imageArray[i].image, 0, 0);
        whammyEncoder.add(videoContext, imageArray[i].duration);
        delete imageArray[i];
    }
    var videoBlob = whammyEncoder.compile();

    BinaryFileReader.read(videoBlob, function (err, fileInfo) {
        UserVideos.insert({
            userId: user._id,
            video: fileInfo,
            save_date: Date.now()
        });
    });
    console.log("Video uploaded");

    // stop the stream & redirect to show the video
    mediaStream.stop();
    Router.go('showVideo', { _id: user._id });
}

Audio: First, we stop the audio recording. Since we aren’t recording any more matching video frames, we don’t want any more audio to be recorded. After some administrative stuff (enabled/disabling buttons and such), we save the audio. The Recorder.js object has an exportWAV method, which saves the audio data to a blob. Then we convert the blob to a data file and add it to our UserAudios collection.

You’ll notice that we use the BinaryFileReader object (shown below) to convert the blob, which is basically a simple wrapper around the FileReader class. You could just as easily use the FileReader class directly.

Video:  For the video we create a new Video instance using the Whammy encoder class. We go through each image that was put into the array during recording, send it to the video context, and tell whammy to add the image from the context (passing in the saved duration as well). Lastly, we do the video encoding to get a blob for the video and, like the audio, convert the blob to file data.

To clean up, we stop the local media stream so we aren’t holding onto the client’s webcam and microphone and then reroute to show the recorded video.

var BinaryFileReader = {
    read: function (file, callback) {
        var reader = new FileReader;

        var fileInfo = {
            name: file.name,
            type: file.type,
            size: file.size,
            file: null
        }

        reader.onload = function () {
            fileInfo.file = new Uint8Array(reader.result);
            callback(null, fileInfo);
        }
        reader.onerror = function () {
            callback(reader.error);
        }

        reader.readAsArrayBuffer(file);
    }
}

Playing the Video

The ‘showVideo’ template performs the task of displaying the video. If you’ll remember from much earlier, the route for ‘showVideo’ actually pulls the data that we saved and sends it to the ‘showVideo’ template. At that point, the template has and object with this info, which was added to the database during the recording process:

{
   audio: [the file data for the audio],
   video: [the file data for the video],
   userId: [the id for the user]
}

The ‘showVideo’ template then takes that information in the handlers for ‘userVideo’ and ‘userAudio’ and converts it to something useful:

Template.showVideo.userVideo = function () {
    console.log("Template.record.userVideo: " + this.userId);
    if (!this.video) {
        return "";
    }

    var blob = new Blob([this.video.video.file],
        {type: this.video.video.type});
    return window.URL.createObjectURL(blob);
};

Template.showVideo.userAudio = function () {
    console.log("Template.showVideo.userAudio: " + this.userId);
    if (!this.audio) {
        return "";
    }

    var blob = new Blob([this.audio.audio.file],
        {type: this.audio.audio.type});
    return window.URL.createObjectURL(blob);
};

The template uses the above data as the src for the video and audio:

<button id="play-recording">Play</button>
<video id="review_video" width="320" height="240" src="{{userVideo}}"></video>
<audio hidden="true" id="review_audio" src="{{userAudio}}"></audio>

The handler for ‘play-recording’ simply calls play for the video and audio elements:

document.getElementById("review_video").play();
document.getElementById("review_audio").play();

Conclusion

I am under no delusions that this is the most perfect way to record both audio and video but, after an extensive search, do not think there is a simple method for recording both audio and video using html5 at the moment. I fully welcome someone proving me wrong. This approach gets the job done in a fairly straight-forward manner.

Obviously this isn’t a production application. I have only tested in Chrome. To use this in a production environment you would probably need a development phase focusing on the idiosyncrasies of all modern browsers. There are a number of things that could be improved upon and fixed. Some of the things I had considered or noticed but never go around to addressing:

  • Use Meteor-CollectionFS  for cleanly uploading files
  • Error in Firefox saving video when recording.
  • Safari won’t play video at all.
  • Fix the occasional stuttering that occurs.
Brian Poteat

Brian Poteat

Brian Poteat

Latest posts by Brian Poteat (see all)

Tags:

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

7 Comments

  1. Paul Fernandez (@paul4nandez)

    Great writeup! I’ve been using almost exactly the same setup in a pet audio recording project of my own: https://github.com/pfernandez/songnotes and it’s good to see that someone else has wound up at the same place I have. Browser support is frustrating but improving every day on this stuff, but I can’t help but think that this is where it’s going to be @ in the near future.

    My next step is to try storing the audio and video files in a directory on the server rather than as binary in a Mongo database, to avoid Mongo’s db size cap (which I believe is about 15MB), to make downloads easier, and hopefully to improve performance. I also found that trying to play back audio from blobs (rather than files) is problematic on mobile browsers. It looks like you might be thinking along the same lines?

    • Paul Fernandez (@paul4nandez)

      Actually, following your link to CollectionFS, I see that you have indeed been thinking about this, Thanks for the lead!

  2. Henry1502inc

    Do you have a demo page?

  3. Christopher Gray

    Hi – Do you have a demo page? Would be great to see this in action on FF.

  4. Stefan Smiljkovic (@shtefcs)

    Hey, glad i found your post. It would be really really nice to have online demo.

  5. alex

    Hi! Awesome Article! Really helped a lot! But where is supportsMedia() initialized? Couldn’t find it anywhere

  6. naicuoctavian

    For anyone looking for a production ready video recorder don’t forget that video recording with Flash and a media server has worked very well since 2002. HDFVR is a good example of such a video recording tool that uses Flash.

    On mobile you also have the option to use the new HTML Media Capture tags which are well supported by practically all mobile browsers. HTML Media Capture works by bringing up the native video recording app to capture the video and then uploading the recorded or selected video to the web server http://hdfvr.com/recording-video-on-mobile-devices-with-HTML-Media-Capture .