Speedier upload using Nodejs and Resumable.js

[updated] View source code at https://github.com/karthik20522/MultiPortUpload

Resumable.js is by far one of the best file uploading plugin that I have used followed by Plupload. Resumable.js provides offline mode features where if a user gets disconnected while uploading it would automatically resume when online. Similar to Plupload it has chunking options. Nodejs on other hand provides a non- blocking functionality which is perfect for uploading purposes.

There is no upload speed difference between upload plugins (resumablejs, plupload etc) except for few features hear and there. Recently I developed a proof of concept for speedier upload using existing plugins and systems. As part of the research was to emulate other file accelerator where multiple ports are used to upload files, thus making uploading quicker.

Using the same above concept, I modified the resumable.js to accept multiple urls as an array and upload individual chunks to different urls in a round-robin style. On the backend I spawned nodejs in multiple ports. But resumable.js only uploads multiple chunks in parallel but not multiple files. This limitation was overcome with some simple code change and following is a test result with various scenarios.

Note: in resumable.js, simultaneous sends option was set to 3

	Single Server single file upload	Multiple Server single file upload	Multiple server + multiple file upload
1 file (109MB)	54secs	56 secs	56 secs
59 file (109MB)	152secs	156 secs	17 secs

Single Server single file upload – default configuration on resumable.js and single Node.js server to accept files/chunks
Multiple Server single file upload – modified resumable.js to take multiple urls and Node.js was configured to listen to different ports (3000, 3001, 3002). Resumable.js when uploading chunks would upload to different ports in parallel.
Multiple Server + multiple file upload – modified resumable.js to upload multiple files and multiple chunks in parallel instead of one file at a time.

But the above test results are for only 3 simultaneous connections. Modern browsers can handle more than 3 connections, following is the number of connections per server supported by current browsers. The theory is that browsers make parallel connections when different domains are used and uploading parallel would make use of full user bandwidth for faster upload.

Browser	Connections
IE 6,7	2
IE8	6
Firefox 2	2
Firefox 3	6
Firefox 4	6 (12?)
Safari 4	6
Opera	4
Chrome 6	7

Let’s test the above scenario with 10 simultaneous connections:

	Single Server single file upload	Multiple Server single file upload	Multiple server + multiple file upload
1 file (109MB)	27 secs	18 secs	18 secs
59 files (109MB)	156 secs	158 secs	14 secs

Server was using almost entire Bandwidth on a multi-file upload! ~1Gbps; 986Mbps!

As you can clearly see from the above results having different upload endpoints (ports/host-names) would allow browser to make parallel connections as it would treat as a new host.

Advantages:

• Customizable. In house development
• As Fast as user bandwidth
• Use Resumable.js plugin for offline support! Win-Win for everyone!

Disadvantages:

• Html5 only i.e. No IE 9 and below support!
• Server s/w needs to be reliable enough to handle huge data & IO operations

Note: Maximum chunk size for the above test were set to 1MB. There is a bit of code which determines the user Internet speed and determines the chunksize; I am doing this basically by downloading a JPEG file and calculating the time token to download. This chunkSize calculation is just a POC

Teacup UI - Fun, simple to use jQuery plugin

Download the code at https://github.com/karthik20522/TeacupUI

A simple representation of percentage data as a layer of ingredients in a tea cup. Yes, a tea cup!! Why? Well just for fun I built it. For whom? Possibly for some one who has a coffee or a tea site and want to present their mix in a teacup ui or possibly for a online info-graphics. Usage is limited by imagination only!

Anyways, how to get this setup and working. Just follow these steps:

- Add the css
- Add the js file and include jQuery
- Add your data Note that your values should add up to 100 to fill up the teacup. Your value can be larger but it would look funny and if your value is lesser than 100 that's OK as the plugin would fill the rest up with emptiness. Also note that each of the href's are clickable within the teacup.

- Bind the plugin

What else:

- How about custom colors - How about multiple teacups on same page

Download the code and example at https://github.com/karthik20522/TeacupUI

jQTagCloud - Simple Tag Cloud in jQuery

A simple tag cloud plugin that I built in JQuery (1.5kb in size uncompressed). https://github.com/karthik20522/jqTagCloud

An example usage:
maxSize: Maximum font size to use
minSize: Minimum font size to use

Multiple tag clouds can be used on same page. Checkout the example file for demo:




Screenshot usage of this plugin:

Javascript - My programming guidelines

JavaScript in general becomes hard to maintain for large projects when the code gets fragmented. So as a general rule of thumb following guidelines are what I follow:

Javsacript files

• All javascript files must be placed in /content/JS folder
• All plugins must be placed in /content/JS/plugins folder
  
  • Plugins are files that are extensions to default UI or binding behavior.
  • Example: Autocomplete, Watermarking, Lazyloading, livequery etc.
• When building a custom UI behavior that potentially could be used in other pages should be placed in /contents/JS/Shared folder.

Shared UI functionality
In situations where functionality has subtle differences, option based extension model is preferred to avoid duplication of code. For example:



Javascript Classes

• This would help in avoiding function name collisions.
• Each view probably should have its own javascript class using {view name} as the naming convention. This class must hold all local variables and functions associated to that view.

Two ways of defining classes in javascript:


Inline Scripting / UI Binding

• No inline scripting must be used
• No html based inline property scripting must be used either. For example: "input type=”test” value=”hello world” class=”cs1” id=”test1” onClick=”callSomeFunction();” "
• Instead all binding must be declared in the class’s init() function. The init() function could/should be called in document.ready() method.

Face.com Post image - Local File - Face Detect

Face.com has two ways of accessing it’s API. One way is a simple Http GET, for example:

http://api.face.com/faces/detect.json?api_key={YOURAPIKEY}&api_secret={YOURSECRETKEY}&urls=http://farm3.static.flickr.com/2566/3896283279_0209be7a67.jpg

and the second way is to use Http POST to url [http://api.face.com/faces/detect.json]. In the GET request, you have to provide the URL of the image and in the POST request you can post image from your local machine. Unfortunately the examples on Face.com is quite vague. The example that they provided is like the following:

Content-Type: multipart/form-data; boundary=nonRelevantString
Content-Length: 104687
-----------------------------nonRelevantString
Content-Disposition: form-data; name="api_key"
4b4b4c6d54c37
-----------------------------nonRelevantString
Content-Disposition: form-data; name="api_secret"
sfhs8fj948fj29
-----------------------------nonRelevantString
Content-Disposition: form-data; filename="image.jpg"
Content-Type: image/jpeg
<>

Looking at the above example, it provides no information on how the post parameters should be constructed. Some of the examples that I came across was building the above text and post this as part of request header like the following example:

http://developers.face.com/forums/post2315.html#p2315

Bummer!

The basic idea of Face.com POST request is to post a multipart form with fields like “api_key”, “api_Secret” and “filename”. Following is a much cleaner code to post an image to face.com for face detection.



that “fileToUpload” is a input of type “File” like the following:



For those who are not aware of FormData, it’s a HTML5 element to build form object.