Shrine::Tus

Provides tools for integrating Shrine with tus-ruby-server.

Installation

gem "shrine-tus", "~> 2.0"

Usage

When the file is uploaded to the tus server, you'll probably want to attach it to a database record and move it to permanent storage. The storage that tus-ruby-server uses should be considered temporary, as some uploads might never be finished, and tus-ruby-server needs to clear expired uploads.

Shrine-tus provides three ways of moving the file uploaded into the tus server to permanent storage, which differ in performance and level of decoupling. But regardless of the approach you choose, the code for attaching the uploaded file works the same:

class VideoUploader < Shrine
  # ...
end

class Movie < Sequel::Model
  include VideoUploader::Attachment.new(:video)
end

file_data #=> {"id":"http://tus-server.org/68db42638388ae645ab747b36a837a79", "storage":"cache", "metadata":{...}}
Movie.create(video: file_data)

See shrine-tus-demo for an example application that uses shrine-tus.

Note that by default Shrine won't extract metadata from directly upload files, instead it will just copy metadata that was extracted on the client side. See this section for the rationale and instructions on how to opt in.

Approach A: Downloading through tus server

Conceptionally the simplest setup is to have Shrine download the uploaded file through your tus server app. This is also the most decoupled option, because your main app can remain completely oblivious to what storage the tus server app uses internally, or in which programming language is it written (e.g. you could swap it for tusd and everything should continue to work the same).

To use this approach, you need to assign Shrine::Storage::Tus as your temporary storage:

require "shrine/storage/tus"

Shrine.storages = {
  cache: Shrine::Storage::YourTemporaryStorage.new(...),
  store: Shrine::Storage::YourPermanentStorage.new(...),
  tus:   Shrine::Storage::Tus.new
}

class VideoUploader < Shrine
  storages[:cache] = storages[:tus] # set Shrine::Storage::Tus as temporary storage
end

Shrine::Storage::Tus is a subclass of Shrine::Storage::Url, which uses Down for downloading. By default, the Down::Http backend is used, which is implemented using HTTP.rb.

If you're experiencing a lot of network hiccups while downloading, you might want to consider switching to the Down::Wget backend, as wget automatically resumes the download in case of network hiccups.

Shrine::Storage::Tus.new(downloader: :wget)

Approach B: Downloading directly from storage

While the appoach A is decoupled from the tus server implementation, it might not be the most performant depending on the overhead between your main app and the tus server app, how well the web server that you use for the tus server app handles streaming downloads (whether it blocks the web worker, thus affecting the app's request throughput), and whether you have hard request timeout limits like on Heroku. Down will also temporarily cache downloaded content to disk while copying to the permanent storage, so that adds I/O and disk usage.

Instead of downloading through the tus server app, you can download directly from the underlying storage that it uses. This requires that you have the tus storage configured in your main app (which might already by the case if you're running tus server in the same process as your main app), and you can pass that storage to Shrine::Storage::Tus:

Shrine::Storage::Tus.new(tus_storage: Tus::Server.opts[:storage])

Approach C: Tus storage equals Shrine storage

Approach B internally utilizes the common interface of each tus storage object for streaming the file uploaded to that storage, via #each. However, certain Shrine storage classes have optimizations when copying/moving a file between two storages of the same kind.

So if you want to use the same kind of permanent storage as your tus server uses, you can reap those performance benefits. In order to do this, instead of using Shrine::Storage::Tus as your temporary Shrine storage as we did in approaches A and B, we will be using a regular Shrine storage which will match the storage that the tus server uses. In other words, your Shrine storage and your tus storage would reference the same files. So, if your tus server app is configured with either of the following storages:

Tus::Server.opts[:storage] = Tus::Storage::Filesystem.new("data")
Tus::Server.opts[:storage] = Tus::Storage::Gridfs.new(client: mongo, prefix: "tus")
Tus::Server.opts[:storage] = Tus::Storage::S3.new(prefix: "tus", **s3_options)

Then Shrine should be configured with the corresponding temporary storage:

Shrine.storages[:cache] = Shrine::Storage::FileSystem.new("data")
Shrine.storages[:cache] = Shrine::Storage::Gridfs.new(client: mongo, prefix: "tus")
Shrine.storages[:cache] = Shrine::Storage::S3.new(prefix: "tus", **s3_options)

In approaches A and B we didn't need to change the file data received from the client, because we were using a subclass of Shrine::Storage::Url, which accepts the id field as a URL. But with this approach the id field will need to be translated from the tus URL to the correct ID for your temporary Shrine storage, using the tus plugin that ships with shrine-tus.

Shrine.storages = {
  cache: Shrine::Storage::YourTemporaryStorage.new(...),
  store: Shrine::Storage::YourPermanentStorage.new(...),
}

class VideoUploader < Shrine
  plugin :tus
end

Note that it's not recommended to use the delete_promoted Shrine plugin with this this approach, because depending on the tus storage implementation it could cause HEAD requests to the tus server app to return a success for files that were deleted by Shrine.

These are the performance advantages for each of the official storages:

Filesystem

Shrine::Storage::FileSystem will have roughly the same performance as in option B, though it will allocate less memory. However, if you load the moving plugin, Shrine will execute a mv command between the tus storage and permanent storage, which executes instantaneously regardless of the filesize.

Shrine.plugin :moving

Mongo GridFS

Shrine::Storage::Gridfs will use more efficient copying, resulting in up to 2x speedup according to my benchmarks.

AWS S3

Shrine::Storage::S3 will issue a single S3 COPY request for files smaller than 100MB, while files 100MB or larger will be divided into multiple chunks which will be copied individually and in parallel using S3's multipart API.

Contributing

$ bundle exec rake test

License

MIT