RSS

Tag Archives: HTML5

Taking a [passport] photo using your camera and HTML 5 and uploading the result with ASP.NET Ajax

If you just want the sample, right click this link, save as, rename to zip and extract.

7

You can use your HTML 5 browser to capture video and photos. That is if your browser supports this feature (at the time of this writing, this example works well on Firefox and Chrome but not IE11).
I have followed some good references on the internet on how to do that, but I also needed some implementation on how to take a “passport” photo and upload the result. This is the intention of this post.

Steps:

  1. Capture video and take snapshot.
  2. Display target area.
  3. Crop the photo to a desired size.
  4. Upload the result to the server.

Step 1: Capture video and take snapshot.
This step relies mainly on the Eric Bidelman’s excellent article. After consideration I decided not to repeat the necessary steps for taking a snapshot using HTML 5, so if you require detailed explanation please read his good article. However the minimum code for this is pretty much straight forward so consider reading on. What you basically need is a browser that supports the video element and getUserMedia(). Also required is a canvas element for showing a snapshot of the video source.

<!DOCTYPE html>
<html>
<head>
    <title></title>
</head>
<body>
    <video autoplay width="320" height="240"></video>
    <canvas width='320' height='240' style="border:1px solid #d3d3d3;"></canvas>
    <div>
        <input type="button" value="start capture" onclick="startCapture()" />
        <input type="button" value="take snapshot" onclick="takePhoto()" />
        <input type="button" value="stop capture" onclick="stopCapture()" />
    </div>
    <script type="text/javascript">
        var localMediaStream = null;
        var video = document.querySelector('video');
        var canvas = document.querySelector('canvas');

        function takePhoto() {
            if (localMediaStream) {
                var ctx = canvas.getContext('2d');
                ctx.drawImage(video, 0, 0, 320, 240);
            }
        }

        navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia;
        window.URL = window.URL || window.webkitURL;

        function startCapture() {
            navigator.getUserMedia({ video: true }, function (stream) {
                video.src = window.URL.createObjectURL(stream);
                localMediaStream = stream;
            }, function (e) {
                console.log(e);
            });
        }

        function stopCapture() {
            video.pause();
            localMediaStream.stop();
        }
    </script>
</body>
</html>

Several points of interest:

  • Line 7: Video element for showing the captured stream. My camera seems to show a default of 640×480 but here this is set to 320×240 so it will take less space on the browser. Bear this in mind, it’ll be important for later.
  • Line 8: Canvas element for the snapshots. Upon clicking ‘take photo’, the captured stream is rendered to this canvas. Note the canvas size.
  • Line 22: Drawing the snapshot image onto the canvas.
  • Line 26: Consider testing support for getUserMedia.
  • Line 30: Capture video.

The result, after starting a capture and taking a snapshot (video stream is on the left, canvas with snapshot is to the right):
2

Step 2: Display target area.
As the camera takes pictures in “landscape”, we will attempt to crop the image to the desired portrait dimensions. Therefore the idea is to place a div on top of the video element to mark the target area, where the head is to be placed.

4

The code:

<!DOCTYPE html>
<html>
<head>
    <title></title>
    <style type="text/css">
        .container {
            width: 320px;
            height: 240px;
            position: relative;
            border: 1px solid #d3d3d3;
            float: left;
        }

        .container video {
            width: 100%;
            height: 100%;
            position: absolute;
        }

        .container .photoArea {
            border: 2px dashed white;
            width: 140px;
            height: 190px;
            position: relative;
            margin: 0 auto;
            top: 40px;
        }

        canvas {
            float: left;
        }

        .controls {
            clear: both;
        }
    </style>
</head>
<body>
    <div class="container">
        <video autoplay></video>
        <div class="photoArea"></div>
    </div>
    <canvas width='320' height='240' style="border: 1px solid #d3d3d3;"></canvas>
    <div class="controls">
        <input type="button" value="start capture" onclick="startCapture()" />
        <input type="button" value="take snapshot" onclick="takePhoto()" />
        <input type="button" value="stop capture" onclick="stopCapture()" />
    </div>
    <script type="text/javascript">
        var localMediaStream = null;
        var video = document.querySelector('video');
        var canvas = document.querySelector('canvas');

        function takePhoto() {
            if (localMediaStream) {
                var ctx = canvas.getContext('2d');
                ctx.drawImage(video, 0, 0, 320, 240);
            }
        }

        navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia;
        window.URL = window.URL || window.webkitURL;

        function startCapture() {
            navigator.getUserMedia({ video: true }, function (stream) {
                video.src = window.URL.createObjectURL(stream);
                localMediaStream = stream;
            }, function (e) {
                console.log(e);
            });
        }

        function stopCapture() {
            video.pause();
            localMediaStream.stop();
        }
    </script>
</body>
</html>

As you can see, the code was modified to place the dashed area on top of the video. Points of interest:

  • Lines 20-27: note the dimensions of the target area. Also note that the target area is positioned horizontally automatically using ‘margin’.
  • Line 41: The dashed area.

Step 3: Crop picture to desired size.
Luckily the drawImage() method can not only resize a picture but also crop it. A good reference on drawImage is here, and the very good example is here. Still, this is tricky as this isn’t an existing image as shown in the example, but a captured video source which is originally not 320×240 but 640×480. It took me some time to understand that and figure out that it means that the x,y,width and height of the source arguments should be doubled (and if this understanding is incorrect I would appreciate if someone can comment and provide the correct explanation).

As cropping might be a confusing business, my suggestion is to first “crop without cropping”. This means invoking drawImage() to crop, but ensuring that the target is identical to the source in dimensions.

function takePhoto() {
    if (localMediaStream) {
        var ctx = canvas.getContext('2d');
        // original draw image
        //ctx.drawImage(video, 0, 0, 320, 240); 

        // crop without cropping: source args are doubled; 
        // target args are the expected dimensions
        // the result is identical to the previous drawImage
        ctx.drawImage(video, 0, 0, 640, 480, 0, 0, 320, 240);
    }
}

The result:
5

Let’s review the arguments (skipping the first ‘video’ argument):

  • The first pair are the x,y of the starting points of the source.
  • The second pair are the width and height of the source.
  • The third pair are the x,y of the starting points of the target canvas (these can be greater than zero, for example if you would like to have some padding).
  • The fourth pair are the width and height of the target canvas, effectively allowing you also to resize the picture.

Now let’s review the dimensions in our case:
6

In this example the target area is 140×190 and starts at y=40. As the width of the capture area is 320 and the target area is 140, each margin is 90. So basically we should start cropping at x=90.

But since in the source picture everything is doubled as explained before, the drawImage looks different as the first four arguments are doubled:

function takePhoto() {
    if (localMediaStream) {
        var ctx = canvas.getContext('2d');
        //ctx.drawImage(video, 0, 0, 320, 240); // original draw image
        //ctx.drawImage(video, 0, 0, 640, 480, 0, 0, 320, 240); // entire image

        //instead of using the requested dimensions "as is"
        //ctx.drawImage(video, 90, 40, 140, 190, 0, 0, 140, 190);

        // we double the source args but not the target args
        ctx.drawImage(video, 180, 80, 280, 380, 0, 0, 140, 190);
    }
}

The result:
7

Step 4: Upload the result to the server.
Finally we would like to upload the cropped result to the server. For this purpose we will take the image from the canvas and set it as a source of an img tag.

<!DOCTYPE html>
<html>
<head>
    <title></title>
    <style type="text/css">
        .container {
            width: 320px;
            height: 240px;
            position: relative;
            border: 1px solid #d3d3d3;
            float: left;
        }

        .container video {
            width: 100%;
            height: 100%;
            position: absolute;
        }

        .container .photoArea {
            border: 2px dashed white;
            width: 140px;
            height: 190px;
            position: relative;
            margin: 0 auto;
            top: 40px;
        }

        canvas, img {
            float: left;
        }

        .controls {
            clear: both;
        }
    </style>
</head>
<body>
    <div class="container">
        <video autoplay></video>
        <div class="photoArea"></div>
    </div>
    <canvas width='140' height='190' style="border: 1px solid #d3d3d3;"></canvas>
    <img width="140" height="190" />
    <div class="controls">
        <input type="button" value="start capture" onclick="startCapture()" />
        <input type="button" value="take snapshot" onclick="takePhoto()" />
        <input type="button" value="stop capture" onclick="stopCapture()" />
    </div>
    <script type="text/javascript">
        var localMediaStream = null;
        var video = document.querySelector('video');
        var canvas = document.querySelector('canvas');

        function takePhoto() {
            if (localMediaStream) {
                var ctx = canvas.getContext('2d');
                //ctx.drawImage(video, 0, 0, 320, 240); // original draw image
                //ctx.drawImage(video, 0, 0, 640, 480, 0, 0, 320, 240); // entire image

                //instead of
                //ctx.drawImage(video, 90, 40, 140, 190, 0, 0, 140, 190);

                // we double the source coordinates
                ctx.drawImage(video, 180, 80, 280, 380, 0, 0, 140, 190);
                document.querySelector('img').src = canvas.toDataURL('image/jpeg');
            }
        }

        navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia;
        window.URL = window.URL || window.webkitURL;

        function startCapture() {
            navigator.getUserMedia({ video: true }, function (stream) {
                video.src = window.URL.createObjectURL(stream);
                localMediaStream = stream;
            }, function (e) {
                console.log(e);
            });
        }

        function stopCapture() {
            video.pause();
            localMediaStream.stop();
        }
    </script>
</body>
</html>
  • Lines 43-44: Note that the canvas has been resized to the desired image size, and the new img element is also resized to those dimensions. If we don’t match them we might see the cropped image stretched or resized not according to the desired dimensions.
  • Line 66: We instruct the canvas to return a jpeg as a source for the image (other image formats are also possible, but this is off topic).

This is how it looks like. The video is on the left, the canvas is in the middle and the new img is to the right (it is masked with blue because of the debugger inspection). It is important to notice the debugger, which shows that the source image is a base64 string.
8

Now we can add a button to upload the base64 string to the server. The example uses ASP.NET PageMethods but obviously you can pick whatever is convenient for yourself. The client code:

<!DOCTYPE html>
<html>
<head>
    <title></title>
    <style type="text/css">
        .container {
            width: 320px;
            height: 240px;
            position: relative;
            border: 1px solid #d3d3d3;
            float: left;
        }

        .container video {
            width: 100%;
            height: 100%;
            position: absolute;
        }

        .container .photoArea {
            border: 2px dashed white;
            width: 140px;
            height: 190px;
            position: relative;
            margin: 0 auto;
            top: 40px;
        }

        canvas, img {
            float: left;
        }

        .controls {
            clear: both;
        }
    </style>
</head>
<body>
    <form runat="server">
        <asp:ScriptManager runat="server" EnablePageMethods="true"></asp:ScriptManager>
    </form>
    <div class="container">
        <video autoplay></video>
        <div class="photoArea"></div>
    </div>
    <canvas width='140' height='190' style="border: 1px solid #d3d3d3;"></canvas>
    <img width="140" height="190" />
    <div class="controls">
        <input type="button" value="start capture" onclick="startCapture()" />
        <input type="button" value="take snapshot" onclick="takePhoto()" />
        <input type="button" value="stop capture" onclick="stopCapture()" />
        <input type="button" value="upload" onclick="upload()" />
    </div>
    <script type="text/javascript">
        var localMediaStream = null;
        var video = document.querySelector('video');
        var canvas = document.querySelector('canvas');

        function upload() {
            var base64 = document.querySelector('img').src;
            PageMethods.Upload(base64,
                function () { /* TODO: do something for success */ },
                function (e) { console.log(e); }
            );
        }

        function takePhoto() {
            if (localMediaStream) {
                var ctx = canvas.getContext('2d');
                //ctx.drawImage(video, 0, 0, 320, 240); // original draw image
                //ctx.drawImage(video, 0, 0, 640, 480, 0, 0, 320, 240); // entire image

                //instead of
                //ctx.drawImage(video, 90, 40, 140, 190, 0, 0, 140, 190);

                // we double the source coordinates
                ctx.drawImage(video, 180, 80, 280, 380, 0, 0, 140, 190);
                document.querySelector('img').src = canvas.toDataURL('image/jpeg');
            }
        }

        navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia;
        window.URL = window.URL || window.webkitURL;

        function startCapture() {
            navigator.getUserMedia({ video: true }, function (stream) {
                video.src = window.URL.createObjectURL(stream);
                localMediaStream = stream;
            }, function (e) {
                console.log(e);
            });
        }

        function stopCapture() {
            video.pause();
            localMediaStream.stop();
        }
    </script>
</body>
</html>
  • Line 40: PageMethods support.
  • Line 60-61: Get the base64 string from the image and call the proxy Upload method.

The server side:

public partial class _Default : System.Web.UI.Page
{
    [WebMethod]
    public static void Upload(string base64)
    {
        var parts = base64.Split(new char[] { ',' }, 2);
        var bytes = Convert.FromBase64String(parts[1]);
        var path = HttpContext.Current.Server.MapPath(string.Format("~/{0}.jpg", DateTime.Now.Ticks));
        System.IO.File.WriteAllBytes(path, bytes);
    }
}
  • Line 6: As can be seen in the client debugger above, the base64 has a prefix. So we parse the string on the server side into two sections, separating the prefix metadata from the image data.
  • Line 7: Into bytes.
  • Lines 8-9: Save to a local photo. Replace with whatever you need, such as storing in the DB.

Addendum
There are several considerations you should think of:

  • What happens if the camera provides a source of different dimensions?
  • Browsers that do not support these capabilities.
  • The quality of the image. You can use other formats and get a better photo quality (at the price of a larger byte size).
  • You might be required to clear the ‘src’ attribute of the video and or img elements, if you need to reset them towards taking a new photo and ensuring a “fresh state” of these elements.
 
3 Comments

Posted by on 01/06/2014 in Software Development

 

Tags: , , , , , ,

HTML5 Drag and Drop Ajax file upload using jQuery File Upload plugin

If you just want the sample, right click this link, save as, rename to zip and extract.

This post is in a way a continuation of an earlier post, which explains how you can achieve an Ajax File Upload with ASP.NET. Only in this post I’ll focus on how to achieve this using HTML5 Drag and Drop features from a Windows Explorer.

Note: IE9 or below have limited Drag/Drop support so don’t expect this solution to work on those browsers.
Note 2: The FileUpload and jQuery used in this post are not the latest, so some changes maybe required to make this work well with newer versions.

In order to implement drag and drop to an existing File Upload solution, all you have to do is bind three events, two of them toggle CSS classes and have nothing to do with the upload itself, and one of them handles the actual ‘drop’. It’s not that the CSS classes are mandatory in order to get upload to work, it’s simply one of the ways to provide a feedback to the end-user that a drag operation takes place.

Before dragging, the page looks like so:
beforedrag

During the drag operation CSS classes provide a feedback to the user:
drag

The JavaScript used here:

$(function () {
    // file upload
    $('#fileupload').fileupload({
        replaceFileInput: false,
        formData: function (form) {
            return [{ name: "name1", value: "value1" }, { name: "name2", value: "value2"}];
        },
        dataType: 'json',
        url: '<%= ResolveUrl("AjaxFileHandler.ashx") %>',
        done: function (e, data) {
            $.each(data.result, function (index, file) {
                $('<p/>').text(file).appendTo('.divUpload');
            });
        }
    });

    // handle drag/drop
    $('body').bind("dragover", function (e) {
        $('.divUpload').addClass('drag');
        e.originalEvent.dataTransfer.dropEffect = 'copy';
        return false;
    }).bind("dragleave", function (e) {
        $('.divUpload').removeClass('drag');
        return false;
    }).bind("drop", function (e) {
        e.preventDefault();
        $('.divUpload').removeClass('drag');
        var list = $.makeArray(e.originalEvent.dataTransfer.files);
        $('#fileupload').fileupload('add', { files: list });
        return false;
    });
});

The code it quite self-explanatory, but nevertheless here’s a short explanation:

  • Lines 3-15: this is the same code from the previous post. It uses the jQuery FileUpload plugin for Ajax file upload.
  • Lines 18-21: handle a “drag start” event. This simply adds a CSS class to provide a feedback to the user.
  • Lines 22-24: handle a “drag leave” event, and removes the CSS class.
  • Lines 25-30: handles the “drop” event. This is the important stuff. We remove the CSS class that provides a feedback on the dragging, take the select files and add them to the File Upload plugin, which starts the upload. Note that the original files object from the event is converted to an array (“list”), and only then the information is passed on to the File Upload plugin. If you skip this the plugin might not work.
  • Lines 11-13: When done, the file names will be displayed as HTML elements. Change this to whatever you require.

The HTML looks like this:

<div class='divUpload'>
    <div class='file'><input id="fileupload" type="file" name="file" multiple="multiple" /></div>
    <div class='dropzone'><div>drag files here</div></div>
</div>

Not much here either. Mainly different DIVs which are displayed according to the drag/drop events.

  • Line 2: The original file input field.
  • Line 3: The DIV which appears upon drag.

Finally there’s the CSS. I have implemented it one way but naturally this can be done completely differently. The “trick” is to add the “drag” class on the top DIV, which causes the browser to use different CSS rules on the nested DIV elements. The file input field becomes hidden and the drag feedback is shown.

body, html, .divUpload
{
    height: 100%;
    margin: 0;
    overflow: hidden;
    background-color: beige;
}

.divUpload
{
    margin: 8px;
}

.divUpload.drag .file
{
    display: none;
}

.divUpload.drag .dropzone
{
    display: table;
}

.dropzone
{
    border: 3px dashed black;
    display: none;
    font-size: 20px;
    font-weight: bold;
    height: 95%;
    position: relative;
    text-align: center;
    vertical-align: middle;
    width: 99%;
    background-color: rgba(37, 255, 78, 0.33);
}

.dropzone div
{
    display: table-cell;
    vertical-align: middle;
}
  • Line 27: The dropzone is not hidden by default.
  • Line 16: When dragging takes place, the default file field is being hidden.

The result looks like this:
afterdrag

 
4 Comments

Posted by on 24/02/2013 in Software Development

 

Tags: , , ,

Web Storage: localStorage and sessionStorage

Every now and then I come across tech stuff I never even heard of. What’s even more frustrating is that when you come across such a thing, you are amazed that it already exists for quite some time. Then you end up cursing because had you known of it’s existence you would have put it to use months ago instead of what you used back then. This is exactly what I felt when I learned of localStorage, and saw that it is working since IE8, FF3.5, Safari 4, and early versions of Chrome. Web Storage (localStorage and sessionStorage) is related to HTML5, and is one of those things that were probably missing for years – the option to store stuff in the client for client-only purposes. I think that cookies were once the only way to store persistent client-only information, but this solution has many down sides, such as having the cookie information sent back and forth to and from the server for no reason. This is the point where I wish I would have heard about localStorage earlier. localStorage is quite simply – Local Storage. It allows you to store key-value information on the client browser. sessionStorage is a similar implementation, but one which will expire when the session is gone (e.g. the browser window/tab closes).

In case you’re wondering which information you would like to store only in the client, just think of web sites which either do not store a user profile on the server at all but still need to store some information, or simply pages that need to maintain a certain page state. Naturally, sessionStorage might be considered for storing sensitive or temporary information.

Sample:

This sample code easily demonstrates Web Storage, and the difference between a localStorage and sessionStorage.

<!DOCTYPE html>
<html>
<head>
    <script src="http://ajax.aspnetcdn.com/ajax/jQuery/jquery-1.7.1.min.js"></script>
</head>
<body>
    <script type="text/javascript">
        function storeItem() {
            var item = $('#item').val();
            localStorage.setItem('myItem', item);
            sessionStorage.setItem('myItem', item);

            refresh();
        }

        function refresh() {
            $('#localItem').text(localStorage.getItem('myItem'));
            $('#sessionItem').text(sessionStorage.getItem('myItem'));
        }

        $(function () { refresh(); });
    </script>
    Enter item:
    <input type="text" id='item' />
    <input type='button' value='store' onclick='storeItem()' />
    <br />
    Stored Local item:<span id='localItem'></span>
    <br />
    Stored Session item:<span id='sessionItem'></span>
</body>
</html>
  • Lines 8-14 define a storeItem function, which stores the input from line 24 in both a localStorage and a sessionStorage.
  • Lines 16-19 simply displays the stored items in the spans in lines 27 and 29.
  • Line 21 shows the stored items when the page loads.

Running this code demonstrates a bunch of stuff:

  1. A localStorage persists the items across sessions, page reloads etc.
  2. A sessionStorage does not persist, not even across tabs.
  3. This code is compatible as expected with all the major browsers (including IE8).
  4. The behavior seems to be identical in all the major browsers.

Additional methods that exist in Web Storage are removeItem(), clear(), length property and key() which allows iteration using an index.

An additional advantage to using Web Storage is the allocated storage space. According to this, FF, Chrome and Opera allow a 5MB storage space per domain (!), and IE allows 10MB per storage area (I assume this means 10 for localStorage and 10 for sessionStorage). This is a vast amount of storage space in the client, especially compared to what cookies offer.

setItem() uses a string key and a string value. That’s no problem of course, because we can easily use JSON for storing complex objects. Consider this example:

<!DOCTYPE html>
<html>
<head>
    <script src="http://ajax.aspnetcdn.com/ajax/jQuery/jquery-1.7.1.min.js"></script>
</head>
<body>
    <script type="text/javascript">
        function storeItem() {
            var item = $('#item').val();

            var items = localStorage.getItem('myItems');
            if (items != null) {
                items = JSON.parse(items);
            } else {
                items = new Array();
            }

            items.push(item);

            localStorage.setItem('myItems', JSON.stringify(items));

            refresh();
        }

        function refresh() {
            var items = localStorage.getItem('myItems');
            var ul = $('ul');
            ul.html('');
            if (items != null) {
                items = JSON.parse(items);
                $(items).each(function (index, data) {
                    ul.append('<li>' + data + '</li>');
                });
            }
        }

        $(function () { refresh(); });
    </script>
    Enter item:
    <input type="text" id='item' />
    <input type='button' value='store' onclick='storeItem()' />
    <br />
    <ul></ul>
</body>
</html>

As JSON parse() and stringify() are now built-in functions in the same browsers that support Web Storage, they can be easily used for storing complex objects.

  • Lines 11-16 get an existing Array from the storage or creates one for the first time.
  • Line 20 uses JSON.stringify() to take a JavaScript object, serialize it into a JSON string and store it in the localStorage.
  • Line 30 parses the JSON string to a JavaScript object again.
  • Lines 31-33 displays the stored items.

Web Storage can be viewed in Google Chrome under the Resources tab:

In Firefox you can either Firebug or FireStorage add-on, although I found both of them uncomfortable at this stage.

Storage event

This is quite a cool feature. You can bind an event handler so that every time you make a change to the storage, you are notified. What’s cool about this feature, is that it’s working “cross tabs”, so that changes to the storage in one tab raise events on the other tabs. This can be used to sync between the tabs either on storage or even on non-storage events (i.e. use this feature to signal other browsers about state changes, even if they are not necessarily storage related). Note that this applies to the same browser only: don’t expect the storage event to be raised on a Chrome browser if FF or IE change something in the storage (and vice versa).


<!DOCTYPE html>
<html>
<head>
    <script src="http://ajax.aspnetcdn.com/ajax/jQuery/jquery-1.7.1.min.js"></script>
</head>
<body>
    <script type="text/javascript">
        function storeItem() {
            var item = $('#item').val();

            var items = localStorage.getItem('myItems');
            if (items != null) {
                items = JSON.parse(items);
            } else {
                items = new Array();
            }

            items.push(item);

            localStorage.setItem('myItems', JSON.stringify(items));

            refresh();
        }

        function refresh() {
            var items = localStorage.getItem('myItems');
            var ul = $('ul');
            ul.html('');
            if (items != null) {
                items = JSON.parse(items);
                $(items).each(function (index, data) {
                    ul.append('<li>' + data + '</li>');
                });
            }
        }

        $(function () {
            refresh(); // show the current storage when the page first loads

            // refresh the storage and show alert, if the 'storage' event is raised
            $(window).bind('storage', function (e) {
                refresh();
                alert('storage changed for key: ' + e.originalEvent.key);
            });
        });
    </script>
    Enter item:
    <input type="text" id='item' />
    <input type='button' value='store' onclick='storeItem()' />
    <br />
    <ul></ul>
</body>
</html>

Note the highlighted rows 41-44: they employ jQuery to bind the Storage event. In line 42 the items are refreshed; In line 43 you can see an example on how to retrieve the event arg’s specific properties (‘key’ in this example). There are several properties which provide information regarding the change, so you may differ between item creation, removal etc. (they are listed here).

There is a catch here though, as there seems to be some sort of a conflict in the specification (“W3C Candidate Recommendation 08 December 2011”) with regards to when the Storage event is raised. In one location, the spec states that (the bold is my emphasis):

“When the setItem(), removeItem(), and clear() methods are invoked, events are fired on other Document objects that can access the newly stored or removed data, as defined in the sections on the sessionStorage and localStorage attributes.”

This seems to be how Google Chrome and FF interpreted the triggering of the Storage event: It is raised only on other Document objects (i.e. other tabs/windows). So if you give it a try on these browsers, you’ll have to open more than one instance to see this working.

However, under the Storage event section, the spec reads (bold is my emphasis):

“This includes Document objects that are not fully active, but events fired on those are ignored by the event loop until the Document becomes fully active again.

This is a somewhat unclear statement, which I understand as:

  1. Tabs/Windows that are not in focus (i.e. “not fully active”) will have this event fired only when they become active/focused (“fully active”).
  2. If the previous understanding is correct, then this can be understood implicitly as if this event is fired on the currently active Document, which is the current tab.

At first I thought that this interpretation is the one implemented in IE9, or at least partially: if you run the code above in 2 IE tabs, same window, and change the content of the storage, the event is fired on the current tab, and the other tab “blinks” in the task bar (but no alert message is displayed). The alert of the second tab is only shown when you put the other tab in focus. When I first saw this, I was excited: finally MS did something according to the specs! However, when I opened another IE9 window with two tabs (i.e. 4 tabs in total), I witnessed a change in behavior – I got 3 alerts all at once! Two alerts would have been understandable as the two windows potentially represent two “fully active” Documents, but I got three alerts and the forth showed when I switched to that tab… so this led me to assume that my initial thoughts were mistaken and that the “blink” in the taskbar was only caused by the pending alert message, and therefore I now assume that IE raises the events on all the tabs regardless of the active or non-active tabs. To test this new theory I simply commented out the alert() message and behold: no blinking occurred, which strengthens the idea that IE does indeed raise the event everywhere – active or non-active tabs. Anyhow, regardless of the implementation, I think that this feature has lots of potential.

Summary

Web Storage is not the only new implementation of client storage. There’s the (now obsolete) Web SQL Database, and there’s IndexedDB, which I have yet to explore. However, Web Storage seems like a good solution for storing simple or JSON values; It also looks like a good solution for session-only [sensitive or temp] Storage; and it provides an option to sync multiple tabs or Windows of the same origin. This sync doesn’t have to be storage related, so unless there’s another option to sync between multiple tab’s states, this could be a workaround on how to achieve this.

One final word here on cookies. I’m not a fan of cookies and I do expect new client storages such as Web Storage to replace them completely with regards to client-only data. However, it is arguable whether cookies should still be used as a method to persist data on the client, which is transferred back and forth to the server. Basically, I think that there’s no real reason why such data can’t be read when a person first interacts with a server, sent once to the server and will be persisted occasionally as required. Perhaps session cookies are the sole true leftover role of the cookies, although I assume some would claim that this too can be replaced by some other alternatives.

 
2 Comments

Posted by on 27/04/2012 in Software Development

 

Tags: , , ,

HTML5 WebSockets Revolution (?)

This post is mainly combined of two parts: a somewhat philosophical approach to WebSockets, and a technical example for pushing data from the server using WebSockets. If you’re just looking for the technicals, you can click here and skip the philosophy.

Every now and then comes a technology that revolutionizes how we program. You can argue what Revolution in software development means, but I just like to think of it as something that either really changed our everyday programming (e.g. Object Oriented, LINQ, Ajax) or something that is somewhat an architectural change (e.g. SOA, Cloud programming). Microsoft has issued plenty of technologies in the past decade, many of which could have been considered as revolutionary but have either failed in competition (e.g. Silverlight), or have improved the existing but weren’t really revolutionary in concept by themselves (e.g. WCF, WPF). Don’t get me wrong – I use Silverlight for all those things that cannot be achieved using plain HTML & JavaScript, but the tables have turned in favor of HTML5, also as far as MS is concerned. But, Silverlight for the browser will probably continue to fill-in the gaps where HTML5 is still lacking in functionality or lacking implementation. For example, IE9 doesn’t implement multiple input file upload yet, and even if it did, you can’t run a code that will compress the files prior to uploading them – this is solvable using Silverlight.

Looking for the “what’s new” in .NET 4.5 got me reading about “Support for WebSockets Protocol”. When I read it I was truly shocked and amazed, as I wasn’t aware of HTML5 WebSockets at all up to that point, and here before me lay the foundations to a possible revolution in web development architecture, no less. At first, one can read about WebSockets and think that it’s just another feature among many others, but I believe that it is revolutionary in concept. It revolutionizes the concept of request/response. Unlike regular client-server non-web applications which are free to “push” messages from servers to clients, the web architecture is built of the client initiating a request and the server responding with a response. The server is basically unable to “push” messages to the client (excluding plugins which may be doing so). WebSockets allow you to do exactly that. In the MS article referenced above, their sample uses WebSockets to respond to an initiating client request. However, WebSockets could be used to perform the opposite, and upon opening a socket from the client – have the server initiate from then on and push messages to the client without having the client to initiate a request as before.

So what’s so revolutionary? I can think of two main things: 1) No more polling and 2) content can be controlled and pushed from the server without having the client initiate a request.

Polling
This will surely be a winner here. Web applications today are more and more “live”. Our clients browse to our website and receive “live” updates about events or messages. Just open up a Firebug’s Net panel in Facebook, Gmail or your favorite “real-time” stock related website and see polling in action. There are a couple of good pics that describe polling and “long polling” here. As you can easily see, polling is a periodical event which is triggered occasionally by a client “just to check” if there are updates from the server. This architecture is problematic in several ways, some of which are listed below:

  1. Polling isn’t “real time”. I believe that this isn’t really an issue for the majority of websites. After all, you can wait several seconds till you’re notified about that email that has already arrived on the server. But you might want to be notified immediately about a change in the stock market. So, if you have to have “real time” updates, you have to poll very frequently, which leads us to the next issue.
  2. If you have gazillions of users, polling may have a serious performance impact on your servers and as a result on your clients. And the more frequent the polling is, the more frustrating your clients will be.
  3. Lengthy operations: if you poll just to “keep alive” or to receive a minor update, polling may work quite smoothly. But what if your polling triggers a somewhat lengthy operation on the server? For example,  if your polling request results retrieved using a lengthy DB query, or a call to a Web Service that just happens to be down or busy? Multiply that by the number of your clients, and expect possible client timeouts or thread-short servers.

WebSockets clearly solve most of these issues. If a server maintains a list of client WebSockets, the client no-longer have to perform polling, as the server may notify them of changes. The server may either notify changes to specific clients (e.g. notify a client email application that an email has arrived to the user); or it can notify all the users (or groups of users) about a certain update. If you have gazillions of users, this update can be performed smartly, split across groups of users or different servers according to a certain logic which fits the nature of your application.

Program Flow
This is where the revolution can really occur, or die…. I’ll use ASP.NET WebForms and MVC as an example to this. The difference between WebForms and MVC is not so much in technology, as in concept. The general idea behind MVC is that requests arrive at a certain Controller, which retrieves a Model and decides which View should be displayed to the user, whereas in WebForms the request usually starts and ends at a referenced Page and rarely redirects the client to a different page (you can also use Server.Transfer, but this has it’s own challenges). So, in MVC, a Controller is responsible for deciding what to do, right? OK, so, this may be used as a concept with WebSockets too. Imagine a client application that loads an html page with the basic required layout, that registers a WebSocket with a server. Then imagine a Controller-like class on the server which decides what the user should see next, and pushes that to the client browser. Similar to MVC, a Controller class decides what the user should read next, but unlike MVC, that Controller class pushes the content to the client, without having the client request for it. The server has the initiative. Now, you may argue if this is desired, but this really depends on your web application. The web today is mostly interactive – the user has to request information and then the server responds. But think of how T.V. works (that is, non-interactive T.V.) – content is constantly pushed from the server onto the client. This includes media, commercials, real-time tickers etc. While interactivity is important to our users, some users don’t even know about stuff that exists in our website in the first place, so they don’t even know to request that content. How many times have you learned of something you were sorry that you didn’t know about sooner? How many times did you open your favorite news website and was disappointed that there was nothing new and you started to actively look for something new to read? You had to request for that information and look for it, instead of having smart servers push that information to you.

Example: You program a web application that focuses on entertainment, news or social networking. These usually thrive on “live” events: “just let us notify you of something that just happened, so you won’t leave our website.” This doesn’t have to end with polling for the latest news and displaying it in a ticker; You can change the entire html layout of the page according to the content you wish to display. A sports website can notify the client that an anticipated game you have registered to see is starting (“Let’s take you live to the game”); an entertainment website can notify a client when a show starts, or when the Oscar is being given to the Best Movie; A social network can push messages, events, pics and more; Forums can update their threads live. Another example would be “live profile-specific” commercials. You can “push” commercials to your clients upon certain events and profiles. Yet another example I can think of is that your website may offer assistance to your users based on their exposure level to certain features or content that they were never (or hardly ever) exposed to. I don’t know if these given examples convince you of the possible revolution here, but I’m sure that more examples can be thought of. The point is, that the web has a chance to become more content and logic driven and less request-based, where required. Even if the examples above can all be resolved using polling, I’m referring more to the architectural change that can be made due to the technology.

IIS

OK – this is a really frustrating thing now. Turns out that MS is currently supporting WebSockets only in IIS 8. When I read about WebSockets in ASP.NET 4.5, I downloaded Visual Studio 11 Developer Preview and installed it, only to find out that IIS 7 does not support it, and is not going to in the future according to this. I guess that MS is strategically not releasing new technologies for older O/S platforms in order to force us to update to their latest O/S. The same goes for .NET 4.5; as you can understand from the System Requirements of the Developer Preview, .NET 4.5 has a minimum request of Windows 7 (that’s right, not even Windows Vista). I believe that this really cripples us ASP.NET developers from taking advantage of WebSockets soon. Many of the web applications still use Windows 2003 with IIS6. It will be a real shame that browsers will support WebSockets a lot sooner than we can actually employ them, unless an open source alternative plugin will come up or MS comes to their senses.

So, I had to download and install Windows 8 Developer Preview, and install it on a VirtualBox (read here if you have to have custom resolutions to fit your screen). After that, I had to install VS11 Developer Preview because the VS11 that comes with the Windows 8 Developer Preview with development tools is a VS Express version that doesn’t support ASP.NET – and I wanted to debug stuff and not only run it. Finally, I had to install IIS with the WebSockets support:

After that, I was able to test WebSockets.

“Push, push!”

The first thing that I did was to try out MS’ WebSockets sample. This was working as expected, and sending content from the client using a WebSocket resulted in the server sending back that content.

Based on the MS sample, I proceeded with testing the theory described earlier, and maintain a list of WebSockets in order to push data from the server willingly (demonstrated by a simple timer which every 10 seconds transmitted the time-stamp from the server). Yes! it worked. And there was much rejoicing.

Here’s the screenshot (note, that neither Firebug nor IE10 Developer Tools fail to notify at this time that such interactivity took place):

Here’s the client code:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
    <script type="text/javascript">
        var socket;
        if (typeof(WebSocket) !== 'undefined') {
            socket = new WebSocket("ws://localhost/WebSockets/MyDispatcher.ashx");
        } else {
            socket = new MozWebSocket("ws://localhost/WebSockets/MyDispatcher.ashx");
        }

        // Receive a string message from the server.
        socket.onmessage = function (msg) {
            document.getElementById("serverData").innerHTML = msg.data;
        };
</script>
    <form id="form1" runat="server">
    <div id='serverData'></div>
    </form>
</body>
</html>
  • Lines 9-13 instantiate a WebSocket. For FF, I had to add ‘Moz’ prefix as it is still considered experimental. IE10 uses WebSocket as the standard specifies it.
  • Lines 16-18 simply update the div on line 21 with whatever is being received from the server.

Here’s the server code. As it’s quite lengthy, I have embedded comments within (highlighted lines are explained below):

<%@ WebHandler Language="C#" Class="MyDispatcher" %>

using System;
using System.Text;
using System.Web;
using System.Threading.Tasks;
using System.Threading;
using System.Timers;
using System.Web.WebSockets;
using System.Net.WebSockets;
using System.Collections.Generic;

public class MyDispatcher : IHttpHandler
{
    // list of client WebSockets that are open
    private static HashSet<WebSocket> list = new HashSet<WebSocket>();

    // ensure thread-safety of the WebSocket clients
    private static ReaderWriterLockSlim locker = new ReaderWriterLockSlim();

    // timer to simulate server actions which initiate notifications to clients
    private static System.Timers.Timer timer = new System.Timers.Timer();

    // initiate timer
    static MyDispatcher()
    {
        timer.AutoReset = false;
        timer.Elapsed += new System.Timers.ElapsedEventHandler(timer_Elapsed);
        timer.Interval = 10000; // notify every 10 seconds
        timer.Start();
    }

    // when ellapsed, notify all clients what the date/time is
    private async static void timer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
    {
        timer.Stop();

        locker.EnterReadLock();
        try
        {
            if (list.Count == 0)
                return;

            string userMessage = string.Format("server sent: {0}", DateTime.Now);
            var buffer = new ArraySegment<byte>(Encoding.UTF8.GetBytes(userMessage));

            foreach (var socket in list)
            {
                if (socket.CloseStatus == null && socket.State == WebSocketState.Open)
                {
                    // asynchronously send a message to the client
                    await socket.SendAsync(buffer, WebSocketMessageType.Text,
                        true, CancellationToken.None);
                }
            }
        }
        finally
        {
            locker.ExitReadLock();
            timer.Start();
        }
    }

    // entry point to the handler, when a client requests to open a WebSocket
    public void ProcessRequest(HttpContext context)
    {
        if (context.IsWebSocketRequest)
            HttpContext.Current.AcceptWebSocketRequest(MyWebSocket);
    }

    // web socket async handler
    private async Task MyWebSocket(AspNetWebSocketContext context)
    {
        WebSocket socket = context.WebSocket;

        // add socket to socket list
        locker.EnterWriteLock();
        try
        {
            list.Add(socket);
        }
        finally
        {
            locker.ExitWriteLock();
        }

        // maintain socket
        while (true)
        {
            ArraySegment<byte> buffer = new ArraySegment<byte>(new byte[1024]);

            // async wait for a change in the socket
            WebSocketReceiveResult result =
                await socket.ReceiveAsync(buffer, CancellationToken.None);

            if (socket.State == WebSocketState.Open)
            {
                // TODO: handle what the client sent and send a message back from here
            }
            else
            {
                // client is no longer available - delete from list
                locker.EnterWriteLock();
                try
                {
                    list.Remove(socket);
                }
                finally
                {
                    locker.ExitWriteLock();
                }

                break;
            }
        }
    }

    public bool IsReusable { get { return true; } }
}

I’d like to emphasize several things, despite the embedded comments:

  • Line 49: Although the list is supposed to be cleared of non-open WebSockets, it is a safety. This sample did not originally remove closed sockets, and I noticed that accessing sockets in the list just to check if their state was Open, sometimes results in a “cannot access Disposed object” exception. Currently there’s no catch clause in that location as I have not experience this ever since I got rid of them closed sockets, but you may want to consider doing so. BTW: that’s why I also query the nullable CloseStatus prior to querying the Open state: I’m hoping that it won’t result in that exception if there is a CloseStatus – but I added this later in development and I can’t tell for sure if it is really helping.
  • Line 80: That’s where the new opened WebSocket is stored in the list (or rather HashSet).
  • Lines 93-94: This is important: what these lines basically do, is to asynchronously wait for a message from the client. But, it seems like this method returns whenever the socket status changes. So, when await returns, either the WebSocket is open, and it means that the client sent something which can be handled, or the WebSocket is not open, which probably means that it is closing and should be removed from the list of sockets (Line 106).

So, that’s a possible template for having the server send data to the client without being requested to do so by a client request.

Summary:
To make a long story short, WebSockets seem like something that will definitely will be used in web applications in the future, even if just to replace polling. Whether this will lead to an actual architectural change – I haven’t the slightest idea. All I can say is that there are many examples of things that started as something “small” but evolved to a huge success. Who knows – maybe in the future the web will change from a world of a request/response to a more client-server approach?

One thing that does bother me is that IIS7 does not support this feature. For us ASP.NET developers this could really pose a limitation. It’s not that easy to upgrade existing O/S to the newest servers, especially if those servers are customers’ servers, and WebSockets aren’t a good enough reason to do so, at least not yet. I hope that someone at MS will realize that this is a mistake and will help change this decision from within.

Note:
It’s important to note that WebSockets are not a Microsoft-only feature and are an HTML5 concept. This means that it will work on all of the major browsers one day, and as you can see here, it is already supported on Chrome, FF and IE 10.

UPDATE: I urge you to go and read this post too, in which I came to a conclusion that unfortunately WebSockets may not be a full solution as it will be supported from IE10 only.

 
4 Comments

Posted by on 17/02/2012 in Software Development

 

Tags: , , , , , , , ,