-
Notifications
You must be signed in to change notification settings - Fork 180
CFC 1 Retargetable Cloud Files API
We desire a multi-cloud-vendor-capable means of managing files located in cloud storage. A few vendor-specific (e.g., Rackspace-specific) or technology-specific (e.g., S3 vs. OpenStack) solutions exist; but, migrating from one vendor to another may cost the developer time in retargeting their APIs. Thus, as a first draft attempt to arrive at a retargetable cloud files solution, I propose the access methods in this document. I believe that these solutions provide a nice abstraction layer covering the most common use-cases for cloud storage, while providing a mechanism for soliciting platform- or vendor-specific features.
I’m only aware of Openstack APIs. Thus, it’s influenced strongly by the Openstack way of doing things. If anyone has any AWS, Azure, or other cloud services experience, I’d love your input on how to refine this proposal before coding actually begins.
There’s some ambiguity in the Openstack and Rackspace APIs. - Openstack says an account may have custom metadata attached to it, via the Object Storage API. However, Rackspace offers no corresponding documentation for its Cloud Files API. Thus, I’m extrapolating/guessing at what the interface is likely to be. - Neither Rackspace nor Openstack document how to delete a custom attribute from the account using the Object Storage API. Thus, I’m guessing based on established Container conventions.
I’ve sorted the concepts below from most elementary to most comprehensive. For example, we can’t really talk about containers without first mentioning object names, so I list object names first.
"Interfaces" are Go constructs which lets a programmer specify, in a statically-typed way, "duck-typed" methods. They serve as both contracts (implementing an interface means every method in that interface exists in some capacity) and as handles to structures that could well implement other interfaces as well. It is important to note that interfaces are structurally typed, unlike structures, which are declaratively typed. This means you must declare which structure a variable is, but you do NOT need to explicitly declare which interfaces it supports. The compiler can divine that knowledge by statically analyzing the source code.
"Structures" are records stored in RAM, having one or more fields containing data of some type. Structures often provide the data backing an interface. Together, a structure and its set of interfaces more or less mimic objects in the object-oriented programming sense. However, it’s important to note that they are not objects in full-standing (no concept of inheritance exists in Go, for example); what’s more, OpenStack assigns a different concept to the term "object," so this motivates me to avoid the term object. Instead, I prefer to use structure or entity, depending upon specific context.
"Readers" are interfaces that implement the Read() function, used for grabbing data from a storage device, file, or I/O channel.
"Writers" are the opposite of readers — they implement Write(), and used for sending data to a storage device, file, or I/O channel.
"Seekers" are interfaces which implements the Seek() method, used to reposition where a file or buffer will be read from or written to next. Not all readers or writers need implement Seek(), and it’s not always appropriate to do so. For example, chunked object uploads are inherently sequential by their nature, due to limitations with the HTTP protocol itself. Thus, such a writer will not respond to the Seek() method, and in fact, will generate a compile-time error.
"Objects" contain arbitrary blobs of data. The storage system assigns no meaning to the data within, and takes no actions based on their contents. Except as documented by your provider of choice, or in some cases by available memory, Gophercloud imposes no limitations on object size. You can think of objects as files; however, we prefer to reserve files for entities stored on your local filesystem.
"Object names" are human-readable names for objects. Through the use of a naming convention, object names may even emulate subdirectories. Every object has a unique name.
"Containers" in the OpenStack world serve a role similar to directories. Each container, internally, maintains a table of contents which maps an object name to the data it represents. No subdirectories exist; however, they may be emulated through the consistent use of naming techniques.
A "store" contains all the containers belonging to a user’s account. In the specific case of Rackspace, for example, a user may create and manage up to 500 000 containers per account.
According to the OpenStack documentation on its Object API, five operations exist that deal specifically with containers:
-
List the objects stored in the container.
-
Create a new container.
-
Delete an empty container.
-
Get container metadata (includes the number of objects and total byte size of all objects).
-
Create or update arbitrary metadata.
Except for creating a new container, all operations operate on an existing container. Thus, we should expose a Container interface:
type Container interface {
ListObjects(opts ListOptions) ([]Object, error)
Delete() error
Metadata() (Metadata, error)
SetMetadata(map[string]string) error
DeleteMetadata([]string) error
}
I defer the definition of creating a container to the next section, as it is not a function of an existing container.
The behavior of ListObjects would depend on how the supplied options are set. The fully default behavior, with no options configured, should be to just do a basic GET against a container’s endpoint, and grab the list of filenames. The returned slice of objects will only have their names set in this case; all other fields will be set to their respective zero-values.
objs, err := myContainer.ListObjects(gophercloud.ListOptions{})
if err != nil {
panic(err)
}
for _, obj := range objects {
fmt.Println("Filename: ", obj.Name)
}
However, if more complete information is required, which necessarily consumes more bandwidth during retrieval, a "full" listing should be specified:
objs, err := myContainer.ListObjects(gophercloud.ListOptions{
Full: true, // Issue -- change the key to Complete? Formatted?
})
if err != nil {
panic(err)
}
for _, obj := range objects {
fmt.Println("Filename: ", obj.Name)
fmt.Println("Hash: ", obj.Hash)
fmt.Println("Size: ", obj.Size)
fmt.Println("Content-Type: ", obj.ContentType)
fmt.Println("Last-Modified: ", obj.LastModified)
}
This will cause the package to invoke the endpoint with the ?formatted=json option, which will provide the complete set of attributes to fill up an Object.
This implies our Object type is at least:
type Object struct {
Name string
Hash string
Size int
ContentType string
LastModified string
}
The options parameter may also be used to set pagination markers, limits, and end-markers.
objs, err := myContainer.ListObjects(gophercloud.ListOptions{
Limit: 10,
})
for {
if err != nil {
panic(err)
}
for _, obj := range objects {
fmt.Println(obj.Name)
}
if len(objs) < 10 {
break
}
objs, err = myContainer.ListObjects(gophercloud.ListOptions{
Limit: 10,
Marker: objs[10].Name,
})
}
Indeed, here’s my idea for the options type. My study of the Cloud Files API suggests these options are applicable to any endpoint capable of pagination.
type ListOptions struct {
Full bool
Limit int
Marker string
EndMarker string
}
Invoking a Delete() method would remove the container, if it’s empty. Nothing too special about this method, really. As per Openstack specifications, an error results when attempting to dispose of a non-empty container.
The Metadata() method causes a HEAD request against the container endpoint. Metadata consists of a handful of standard attributes, plus a collection of user-specified key/value pairs.
type ContainerMetadata struct {
ObjectCount int
BytesUsed int
Date string
}
The structure contains the standard fields that are well-known to be returned with a typical request. The ContainerMetadata structure implements the MetadataManager interface, granting access to custom metadata keys and values. The custom metadata will not have the "X-Container-Meta-" prefix on any of its keys; it will be automatically stripped for convenience and safety. The design of the MetadataManager interface will appear in section 4.
Regrettably, no standard list of headers exists; therefore, I had to guess what headers would be returned reliably based on Openstack and Rackspace documentation. Standard headers should receive their own Metadata structure fields for greatest type-safety.
For example:
md, err := myContainer.Metadata()
if err == nil {
if md.CustomValue("backedup") == "false" {
// perform container back-up here.
err = md.SetCustomValue("backedup", "true")
}
}
if err != nil {
panic(err)
}
A store represents an interface to a collection of containers. In the context of Gophercloud, it’d represent the top-level HTTP interface to an Openstack Cloud Files implementation. Because it’s a top-level entity, the module function GetStoreApi() would be used to gain access to it, in exactly the same way one uses GetCloudServersApi():
store, err := gophercloud.GetStoreApi(accessApi)
if err != nil {
panic(err)
}
Once a reference to a store has been acquired, several top-level administrative functions become available. According to OpenStack’s documentation, the following capabilities exist:
-
List storage containers.
-
Get account metadata and details.
-
Create or update account metadata.
I’ll append to this list:
-
Create a new container.
These basic features imply the following interface definition for a store:
type Store interface {
CreateContainer(string, map[string]string) (Container, error)
ListContainers(ListOptions) ([]ContainerInfo, error)
}
Creating a container requires only two pieces of information: - Name of the container, and, - An optional set of custom metadata.
container, err := store.CreateContainer("devNull", map[string]string{
"description": "File storage for the dev/null blog posts",
"backedup": "false",
})
if err != nil {
panic(err)
}
Openstack-specific implementation note: It should be pointed out that this call does not return anything. Rather, it synthesizes the container reference from what its URL would be based on the provided name.
The reference returned also supports the ContainerInfo interface, documented in the next section. We return Container directly with this method, presuming that since you already know the name of the container, you’re more interested in just using it rather than querying its properties.
Listing containers returns a slice of ContainerInfo interfaces. At least for OpenStack providers, the backing structures for these interfaces derive directly from the wire-format used by OpenStack:
type ContainerInfo interface {
Name() string
Count() int
Size() int
}
Two levels of container listings exist: basic and full. Basic listings fill in only the Name field, so invoking Count() or Size() on such entries will yield 0. Full listings provide the full set of details, but consumes proportionally more bandwidth to retrieve. In a full listing, a ContainerInfo’s Count() and Size() methods will yield correct values.
// Basic listing...
containers, err := store.ListContainers(gophercloud.ListOptions{})
...
// Full listing...
containers, err := store.ListContainers(gophercloud.ListOptions{
Full: true,
})
if err != nil {
panic(err)
}
for _, c := range containers {
fmt.Println("Container ", c.Name(), " has", c.Size(), "bytes used.")
}
ContainerInfo interfaces do not provide accessor methods; to acquire these, you’ll need to type-assert. For those unfamiliar with type assertions in Go, think of Microsoft COM’s IUnknown interface, specifically the QueryInterface() method. Same idea. For this to work, all structures returned by ListContainers() must implement both ContainerInfo as well as Container.
func MyOpenContainer(containers []ContainerInfo, name string) (Container, error) {
var myContainer Container
for _, c := range containers {
if c.Name() == name {
// We have a ContainerInfo; we want a Container.
// All ContainerInfo interfaces returned by ListContainers()
// should also implement Container as well. We just need
// to tell Go that we're interested in it.
return c.(Container), nil
}
}
return nil, fmt.Errorf("Container not found")
}
This method basically invokes a HEAD request against the store endpoint. The result is a collection of metadata related to the store. The metadata collected includes the following structure fields:
type StoreMetadata struct {
ContainerCount int
BytesUsed int
}
The StoreMetadata structure implements the MetadataManager interface, thus granting easy access to Any custom metadata keys will have their X-Account-Meta- prefixes stripped, as usual.
meta, err := store.Metadata()
if err != nil {
panic(err)
}
fmt.Println("Your account has", meta.ContainerCount, "containers total,")
fmt.Println("and you're consuming a total of ", meta.BytesUsed, "bytes.")
Altering custom metadata can be performed by filling in your own map[string]string type and passing it to the SetMetadata() method. Omitting a field from the metadata map will cause it to retain its existing value, while listing it will cause it to be overwritten.
For example:
sm, err := store.Metadata()
if err == nil {
if sm.Custom["backedup"] == "false" {
// perform store back-up here.
err = store.SetMetadata(map[string]string{
"backedup": "true",
})
}
}
if err != nil {
panic(err)
}
Before continuing onto the topic of objects, I’d like to discuss the MetadataManager interface, used to create, read, update, and delete custom metadata on different objects. As illustrated in sections 2 and 3, this interface can exist on different kinds of objects. The metadata dealt with is scoped to these specific objects.
The MetadataManager interface is defined as follows:
type MetadataManager interface {
CustomValue(string) string
CustomValues() (map[string]string, error)
SetCustomValue(string, string) error
SetCustomValues(map[string]string) error
DeleteCustomValues([]string) error
}
It bears repeating that this interface works only with unadorned or unscoped key values. For example, if you call SetCustomValue("foo", "bar") on a Store, the key will be set with a custom header of X-Account-Meta-foo, but on a container, the header will appear as X-Container-Meta-foo. This enhances security by preventing header forgery through casual means.
Many applications require inspecting only a single custom value; however, others may require more efficient to many values in bulk. For this reason, two methods exist for obtaining a custom value.
Applications may use the CustomValue() method to inspect the value of a given custom key. If the key does not exist, "" (empty string) is returned, as per Go community norms.
toBool := map[string]bool{"true": true}
md, err := myContainer.Metadata()
if err != nil {
panic(err)
}
isBackedUp := toBool[md.CustomValue("backedup")]
Semantically, CustomValue(key) may be thought of as a shortcut for:
mapping, err := entity.CustomValues()
if err != nil {
return ""
}
return mapping[key]
Acquisition of a lot of custom headers may, depending on the provider’s implementation, consume a bit of time due to large numbers of client/server round-trip interactions. To help amortize this overhead, Gophercloud provides the CustomValues() method (note: plural) as a means of acquiring the entire set of custom values attached to the entity’s metadata.
md, err := myContainer.Metadata()
if err != nil {
panic(err)
}
settings := md.CustomValues()
for _, k := range []string{
"title", "subject", "isbn", "copyright",
} {
v := settings[k]
if v != "" {
fmt.Println("The book's %s is \"%s\"", k, v)
}
}
Similar to retrieving custom settings, you may also set or change custom data in one of two ways, depending on your bandwidth requirements.
It frequently happens that only a single metadata setting needs to be altered. The SetCustomValue() method achieves this with a minimum of syntax.
md, err := myContainer.Metadata()
if err != nil {
panic(err)
}
err = md.SetCustomValue("backedup", "false")
if err != nil {
panic(err)
}
If the custom setting did not exist prior to this call, and no error is returned, the setting will be created. If the setting existed previously, it’s old value is replaced with the new value.
Altering custom metadata can be performed by filling in your own map[string]string type and passing it to the SetCustomValues() method (note plural). Omitting a field from the metadata map will cause it to retain its existing value, while listing it will cause it to be overwritten.
md, err := myContainer.Metadata()
if err != nil {
panic(err)
}
err = md.SetCustomValues(map[string]string{
"title": "Jonathan Livingston Seagull",
"subject": "Overcoming adversity through tenacity, conviction, and finding your true home.",
"isbn": "0-380-01286-3",
"copyright": "Bach, Richard",
})
if err != nil {
panic(err)
}
If no error is returned, any previously undefined settings will now have settings created for them. Any previously defined settings will have their existing values replaced with their corresponding new values.
Sometimes, you don’t want to overwrite a piece of metadata, you want to delete it all-together. The DeleteCustomValues() method performs this action. Instead of taking a map, it just takes an slice of strings, each corresponding to a metadata key you want to delete:
err := myContainer.DeleteCustomValues([]string{"backedup"})
if err != nil {
panic(err)
}
Objects contain arbitrary blobs of binary data and thus best approximates files in the local filesystem. However, they have very different semantics from commonly available filesystems. For instance, from the server’s perspective, an object can only be read from or written to sequentially; some types of objects need to be written as a single record, while other kinds of objects can be written in chunks; etc. Those old enough to remember the concept of "sequential datasets" on IBM mainframes will see many similarities.
According to Openstack’s API reference, six operations may be performed involving objects.
-
Get data
-
Get object metadata
-
Copy an object to another with a new name.
-
Create or update the content and metadata for a specified object.
-
Update custom metadata.
-
Delete the object permanently.
Due to the sheer diversity involved with downloading and uploading content, we’ll require a somewhat unconventional storage access model. One would think, for example, that given a Container, you could just ask the Container to GET an object, and be done with it. However, this won’t cover some of the more complex cases, such as ranged GETs, or GETs involving entity tags, and it certainly doesn’t address CDN management.
Thus, I collate different classes of functionality into different, hopefully orthogonal, interfaces.
-
Basic Downloader — This interface allows a client to download an object entirely into local memory, and provides io.Reader and io.Seeker interfaces on the buffer. Use the io.Closer interface to release the buffer. Note that io.Writer isn’t supported on these objects.
-
Chunked Downloader — This interface is used in a similar way to the basic downloader; but it has run-time implications. Being that data is retrieved a piece at a time, the buffer requirements can potentially be much smaller than for the basic downloader. However, it also means that a lot more traffic to the remote server happens as well, so downloading the same-sized file may take quite a bit longer. However, when dealing with very large objects, memory consumption concerns usually trump download performance, and stream-based processing of the data is preferred.
-
Basic Uploader — This interface allows the creation of an object entirely in memory, complete with io.Reader; io.Writer, io.Seeker, and io.Closer interfaces. The object isn’t sent to the server until the object is closed. Note that the io.Reader is provided so the client can read previously written bytes, and is not suitable for reading previously existing data. Thus, it cannot be used to update an existing object in-place.
-
Chunked Uploader — This interface allows the creation of extremely large objects, but at the cost of not being able to seek. Each chunk written through the writer results in HTTP traffic; thus, no internal buffering occurs.
As you can imagine, the basic access methods better serve relatively small object sizes (a few megabytes or smaller), while the chunked access methods better serve larger files (e.g., tens of megabytes or larger). Note that support for DLO, SLO, or other forms of aggregate object creation are not yet supported (directly) in this CFC; thus, maximum object size will depend upon your provider’s limits. In the case of Rackspace, that limit is set to 5GB.
The first access method will involve acquiring an object using the simplest possible method: a simple GET against the object’s endpoint. This will copy the entire object into a memory buffer, which is managed using our own implementations of reader, seeker, and closer interfaces:
reader, err := myContainer.BasicObjectDownloader(gophercloud.ObjectOpts{
Name: "/test/object/1",
})
if err != nil {
panic(err)
}
defer reader.Close()
ioutil.Copy(toFile, reader)
Note that the io.Writer interface is not supported by this access method.
|
Note
|
Everything beyond this point MAY be a Rackspace extension! None of the attributes, headers, or other features described below are described in the OpenStack API docs that I can see. (Certainly not on http://api.openstack.org/api-ref.html). |
The application may elect to download a proper subset of the object as well:
reader, err := myContainer.BasicObjectDownloader(gophercloud.FindOpts{
Name: "/test/object/1",
Offset: 10,
Length: 5,
})
The Offset field, if specified, indicates where to start the download from. The Length field indicates how much data to retrieve in the download. These options affect the Range: MIME header in the HTTP request as follows:
Offset Length Equivalent Range Header 0 0 not present; download the entire object. 0 5 Range: bytes=0-5 10 0 Range: bytes=10- 10 5 Range: bytes=10-15 -10 0 Range: bytes=-10 -10 5 Error: combination not supported.
When using ranges, be aware that byte offset 0 of the buffer will correspond to the first downloaded byte. Thus, invoking reader.Seek(5, 0) on an object downloaded at offset 10 will reset the reader to the 15th byte of the object.
Issue: Should we alter this behavior so that Seek() is aware of the downloaded origin? In this system, Seek() would issue an error if an attempt is made to seek outside the downloaded byte range.
If you’re attempting to download an especially large object, you can manually manipulate offset and length options while closing and re-opening the object. This isn’t terribly convenient, and absolutely will generate a lot of run-time garbage which the collector will have to deal with more frequently. It’s more efficient to maintain a sliding window on behalf of the client application.
reader, err := myContainer.ChunkedObjectDownloader(gophercloud.FindOpts{
Name: "/test/object/2",
})
if err != nil {
panic(err)
}
defer reader.Close()
As you can see, the approach to getting a reference to the object is virtually identical with basic object acquisition. The difference at this point is that no HTTP traffic has transpired at this point (except, perhaps, discovering basic metadata like object size), and a fixed size buffer has been allocated to hold incoming object data. This buffer will be re-used as required, thus reducing the burden on the garbage collector significantly.
The size of the buffer may be specified if the default proves too inefficient for your needs. The example below illustrates how to request a 10MB buffer:
reader, err := myContainer.ChunkedObjectDownloader(gophercloud.FindOpts{
Name: "/test/object/2",
BufferSize: 10485760,
})
Note that the Length option has no effect with the chunked object provider (length is determined by buffer size). However, the Offset field can be used to save a seek:
reader, err := myContainer.ChunkedObjectDownloader(gophercloud.FindOpts{
Name: "/classic-vm/commodore64/node12/memdump",
Offset: 40960,
})
Assuming the file is a memory dump for a cloud-hosted Commodore 64 instance, the reader will be positioned to read the C64 BASIC ROM image. Otherwise, you’d need to seek explicitly:
reader, err := myContainer.ChunkedObjectDownloader(gophercloud.FindOpts{
Name: "/classic-vm/commodore64/node12/memdump",
})
if err != nil {
panic(err)
}
err = reader.Seek(40960, 0)
if err != nil {
panic(err)
}
Remember that HTTP requests for actual data do not occur with this access method until bytes are read from the buffer. Thus, as bytes are read, or if seeking beyond the bounds of the current buffer, the chunked provider will automatically refill the buffer on your behalf.
Both basic and chunked providers support conditional downloads. If a server refuses to deliver content, an appropriate reason will be returned as an error, which can be explicitly confirmed:
reader, err := myContainer.BasicObjectDownloader(gophercloud.FindOpts{
Name: "/cached-images/lolcat-graph.jpg",
IfMatch: []string{oldETag},
})
if err != nil {
if err == gophercloud.WarnNotModified {
// If control reaches here, data has not been modified.
// Ignore, or just re-use existing data.
// NOTE: Nothing was downloaded, so no need to close
// the reader.
}
// Otherwise, another error occurred. Deal with accordingly, or, ...
panic(err)
}
// If control reaches here, the graph was updated. Redisplay and cache its etag.
io.Copy(toFile, reader)
oldETag = reader.ETag()
reader.Close()
Supported predicate options include:
-
IfMatch
-
IfNoneMatch
-
IfModifiedSince
-
IfUnmodifiedSince
More than one predicate may be specified, as per RFC2616. All such options are strings, and assume their native, RFC2616 formats.
When uploading an object, it’s convenient to build the object in memory first, and only send it when it’s completely formed.
writer, err := myContainer.BasicObjectUploader(gophercloud.ObjectOpts{
Name: "/cached-images/lolcat-graph.jpg",
})
if err != nil {
panic(err)
}
io.Copy(writer, fromFile)
err = writer.Commit()
if err != nil {
// Object was not uploaded for some reason. Deal with the error, or,
panic(err)
}
writer.Close()
When opening a basic uploader, the writer will point into an empty chain of buffers. As data is deposited into the object, it accumulates in one or more buckets as required. When the client closes the writer, it will dispense with all those buckets. The Commit() method actually attempts to upload the object data. Since the buckets are available for it to examine, it may do so in a single HTTP request, with appropriate headers. Thus, this approach represents the fastest, most convenient method of uploading an object to a cloud files server.
Note that the returned writer supports the seeker interface; this lets the software creating the object seek back in the file to update previously unset fields. For example, writing out EA-IFF-85a, RIFF, or AIFF files requires this ability. However, it is an error to seek beyond the bounds of the object already written, with the sole exception of Seek(0, 2), which positions the write-pointer at the very end of the object, suitable for appending.
When uploading an extremely large object, using chunked encoding is preferred for reasons of low memory consumption, but it impacts upload performance somewhat due to greater reliance of HTTP traffic.
You can use either the basic or chunked approaches to upload the components of a really large object (say, in excess of 5GB) using dynamic large object (DLO) support. Supporting DLO though additional access mechanisms becomes an exercise in managing component objects utilizing one of the two methods discussed previously. In the interests of time, I’ve omitted such details here, and will (with interest) record my idea for supporting DLO in a subsequent CFC.
I admit that I have not had sufficient time to read through and fully understand Static Large Object specifications yet. I’ll address this in a future CFC if sufficient interest exists.
Not supported in this CFC. If sufficient interest exists, I can address this with a future CFC.
Not supported in this CFC. If sufficient interest exists, I can address this with a future CFC.