Speedier upload using Nodejs and Resumable.js

[updated] View source code at https://github.com/karthik20522/MultiPortUpload

Resumable.js is by far one of the best file uploading plugin that I have used followed by Plupload. Resumable.js provides offline mode features where if a user gets disconnected while uploading it would automatically resume when online. Similar to Plupload it has chunking options. Nodejs on other hand provides a non- blocking functionality which is perfect for uploading purposes.

There is no upload speed difference between upload plugins (resumablejs, plupload etc) except for few features hear and there. Recently I developed a proof of concept for speedier upload using existing plugins and systems. As part of the research was to emulate other file accelerator where multiple ports are used to upload files, thus making uploading quicker.

Using the same above concept, I modified the resumable.js to accept multiple urls as an array and upload individual chunks to different urls in a round-robin style. On the backend I spawned nodejs in multiple ports. But resumable.js only uploads multiple chunks in parallel but not multiple files. This limitation was overcome with some simple code change and following is a test result with various scenarios.

Note: in resumable.js, simultaneous sends option was set to 3

	Single Server single file upload	Multiple Server single file upload	Multiple server + multiple file upload
1 file (109MB)	54secs	56 secs	56 secs
59 file (109MB)	152secs	156 secs	17 secs

Single Server single file upload – default configuration on resumable.js and single Node.js server to accept files/chunks
Multiple Server single file upload – modified resumable.js to take multiple urls and Node.js was configured to listen to different ports (3000, 3001, 3002). Resumable.js when uploading chunks would upload to different ports in parallel.
Multiple Server + multiple file upload – modified resumable.js to upload multiple files and multiple chunks in parallel instead of one file at a time.

But the above test results are for only 3 simultaneous connections. Modern browsers can handle more than 3 connections, following is the number of connections per server supported by current browsers. The theory is that browsers make parallel connections when different domains are used and uploading parallel would make use of full user bandwidth for faster upload.

Browser	Connections
IE 6,7	2
IE8	6
Firefox 2	2
Firefox 3	6
Firefox 4	6 (12?)
Safari 4	6
Opera	4
Chrome 6	7

Let’s test the above scenario with 10 simultaneous connections:

	Single Server single file upload	Multiple Server single file upload	Multiple server + multiple file upload
1 file (109MB)	27 secs	18 secs	18 secs
59 files (109MB)	156 secs	158 secs	14 secs

Server was using almost entire Bandwidth on a multi-file upload! ~1Gbps; 986Mbps!

As you can clearly see from the above results having different upload endpoints (ports/host-names) would allow browser to make parallel connections as it would treat as a new host.

Advantages:

• Customizable. In house development
• As Fast as user bandwidth
• Use Resumable.js plugin for offline support! Win-Win for everyone!

Disadvantages:

• Html5 only i.e. No IE 9 and below support!
• Server s/w needs to be reliable enough to handle huge data & IO operations

Note: Maximum chunk size for the above test were set to 1MB. There is a bit of code which determines the user Internet speed and determines the chunksize; I am doing this basically by downloading a JPEG file and calculating the time token to download. This chunkSize calculation is just a POC