Upload and Resume large files with TusDotNet

I’m using Kestrel on Ubuntu 20.04 as an internet facing edge web server, using ASP.NET 5 with Razor Pages.

Update - am using nginx as a reverse proxy now so can handle: redirect to apex, ssl cert, no iptables direct to port 5000. Detailed blog here

I need to be able to handle large file uploads, which initially are zip files up to a few GB

https://github.com/djhmateer/tus-test is a sample project

https://github.com/osr4rightstools/osr4rights-tools bigger project

Tus

https://tus.io/ Open Protocol for Resumable File Uploads.

https://tus.io/demo.html

Give user feedback on upload progress

and then can resume even after the browser has been closed.

https://github.com/transloadit/uppy is build by the team at https://transloadit.com/ who wrote Tus.

TusDotNet

https://github.com/tusdotnet/tusdotnet .NET server implementation of the Tus protocol for resumable file uploads.

Tus-js-client

https://github.com/tus/tus-js-client A pure JavaScript client for the tus resumable upload protocol

There is a nice html sample included in the project:

git clone https://github.com/tus/tus-js-client.git

# gets all the node_modules
npm install

# creates dist folder (essentially tus.js)
npm run build

# run the demos\browser\index.html file

Resume works fine with the sample server implementation

I had some issues with the server side implementation and resuming.

It uses local storage to remember previous uploads

Razor Pages Source for this demo is https://github.com/djhmateer/tus-test

Remember to run npm.

Test files

To test the validity of the files, we need something to verify:

https://github.com/NVlabs/ffhq-dataset/ - Flickr-Faces-HQ Dataset. Has 70k images on Google Drive totally 2.56TB.

I ended up making sample image zips up to

100MB was a limit (imposed by cloudflare)
2GB was a limit (imposed by some filesystems)
4GB
6GB

Manually Create large test files

https://stackoverflow.com/questions/257844/quickly-create-a-large-file-on-a-linux-system

dd if=/dev/zero of=1GB.test bs=1G count=1

Can be useful, but I prefer to have files I can sanity check.

A Simple Upload form

The simplest possible thing is always a great place to start, but having no user feedback is not a good experience.

<form method="post" enctype="multipart/form-data">
    <input type="file" asp-for="Upload" />
    <input type="submit" />
</form>

then on the code behind

[BindProperty]
public IFormFile? Upload { get; set; }

public async Task<IActionResult> OnPostAsync(CancellationToken cancellationToken)
{
    var connectionString = AppConfiguration.LoadFromEnvironment().ConnectionString;

    // check that Upload is something
    if (Upload is { } && Upload.Length > 0)
    {
        Log.Information($"Uploaded file {Upload.FileName}");
        // eg C:\Users\djhma\AppData\Local\Temp on Windows
        var path = Path.GetTempPath();

        // get next jobId from Db
        var jobId = await Db.InsertJobWithOrigFileNameAndReturnJobId(connectionString, Upload.FileName);

        var fileName = Path.Combine(path, $"job-{jobId}.tmp");

        // copy uploaded file to tempPath location as job-{jobId}.tmp
        using (var fileStream = new FileStream(fileName, FileMode.Create, FileAccess.Write))
        {
            await Upload.CopyToAsync(fileStream, cancellationToken);
        }

        // The Result page will kick off the job
        return Redirect($"Results/{jobId}");
    }

    return RedirectToPage("UploadFailed");
}

Startup.cs

// to allow large form uploads
services.Configure<KestrelServerOptions>(options =>
{
    options.Limits.MaxRequestBodySize = int.MaxValue; // if don't set default value is: 30 MB
});

// to allow large form uploads
services.Configure<FormOptions>(x =>
{
    x.ValueLengthLimit = int.MaxValue;
    x.MultipartBodyLengthLimit = int.MaxValue; // if don't set default value is: 128 MB
    x.MultipartHeadersLengthLimit = int.MaxValue;
});

Cloudflare and Tus

After switching on cloudflare for my project to do:

SSL Certificates automatically
301 Redirects to Apex domain from www

I noticed that my large file uploads were failing (on a simple pre Tus upload)

https://gridpane.com/kb/cloudflares-cdn-and-upload-limitations/ explained why - 100mb file upload limit

Uploading a 92Mb file worked fine, and 121Mb gave (sometimes)

413 Request Entity too large cloudflare

A 200Mb actually gave a blank screen in chrome and

ERR_CONNECTION_CLOSED

Interestingly I found that chunking which Tus supports gave strange errors through Cloudflare. So I turned off Cloudflare.

Tus Bugs

When a successful file upload, why does it prompt if I upload that file again? I found that it was the file permissions on the tusstore on my production server. OSR4Rights-Tools has it implemented properly with the permissions set on VM build stage.

don’t have javascript popup as Chrome wont work if that window isn’t in focus