Ror - Large File Uploads In Rails

I have a rails webapp that allows users to upload videos, where they are stored in an NFS-mounted directory. The current setup is fine for smaller files, but I need to support larg

Solution 1:

You might consider using MiniProfiler to get a better sense of where the time is being spent.

Large file uploading needs to be handled in the background. Any controllers or database access should simply mark that the file was uploaded, and then queue a background processing job to move it around, and any other operations that may need to happen.

http://mattgrande.com/2009/08/11/delayedjob/

That article has the gist of it, every implementation is going to be different.

Solution 2:

I finally found the answer to my main question: What is happening during this 3 minute wait after the upload completes and before my action is called?

It's all explained very clearly in this post: The Rails Way - Uploading Files

"When a browser uploads a file, it encodes the contents in a format called ‘multipart mime’ (it’s the same format that gets used when you send an email attachment). In order for your application to do something with that file, rails has to undo this encoding. To do this requires reading the huge request body, and matching each line against a few regular expressions. This can be incredibly slow and use a huge amount of CPU and memory."

I tried the modporter Apache module mentioned in the post. The only problem is that the module and its corresponding plugin were written 4 years ago, and with their website no longer in operation, there's almost no documentation on either one.

With modporter, I wanted to specify my NFS-mounted directory as the PorterDir, in the hopes that it would pass the file right along to the NAS without any extra copying from a temp directory. However, I was not able to get this far since the module seemed to be ignoring my specified PorterDir, and was returning a completely different path to my actions. On top of that, the path it was returning didn't even exist, so I had no idea what was actually happening to my uploads.

My Workaround

I had to get the problem solved quickly, so I went with a somewhat hacky solution for now which consisted of writing corresponding JavaScript/Ruby code in order to handle chunked file uploads.

JS Example:

varMAX_CHUNK_SIZE = 20000000; // in byteswindow.FileUploader = function (opts) {
    var file = opts.file;
    var url = opts.url;
    var current_byte = 0;
    var success_callback = opts.success;
    var progress_callback = opts.progress;
    var percent_complete = 0;

    this.start = this.resume = function () {
        paused = false;
        upload();
    };

    this.pause = function () {
        paused = true;
    };

    functionupload() {
        var chunk = file.slice(current_byte, current_byte + MAX_CHUNK_SIZE);
        var fd = newFormData();
        fd.append('chunk', chunk);
        fd.append('filename', file.name);
        fd.append('total_size', file.size);
        fd.append('start_byte', current_byte);

        $.ajax(url, {
          type: 'post',
          data: fd,
          success: function (data) {
              current_byte = data.next_byte;
              upload_id = data.upload_id;

              if (data.path) {
                  success_callback(data.path);
              }
              else {
                  percent_complete= Math.round(current_byte / file.size * 100);
                  if (percent_complete> 100) percent_complete = 100;
                  progress_callback(percent_complete); // update some UI element to provide feedback to userupload();
              }
          }
        });
    }
};

(forgive any syntax errors, just typing this off the top of my head)

Server-side, I created a new route to accept the file chunks. On first chunk submission, I generate an upload_id based on filename/size, and determine if I already have a partial file from an interrupted upload. If so, I pass back the next starting byte I need along with the id. If not, I store the first chunk and pass back the id.

The process with additional chunk uploads appending the partial file until the file size matches the original file size. At this point, the server responds with the temporary path to the file.

The javascript then removes the file input from the form, and replaces it with a hidden input whose value is the file path returned from the server, and then posts the form.

Then finally server-side, I handle moving/renaming the file and saving its final path to my model.

Phew.