Hidden (aka Private) attributes

[vvcephei]

What is the Issue?

I'd like to consider the notion of "private" attributes. These are defined as attributes not visible by default via the sor api, databus, or stash. API keys with write permission would be able to assert a query parameter that they wish to see private fields.

Private attributes have three main use cases that I'm aware of right now:

1. Writers may need to create attributes for use in conditional deltas in order to implement MVCC or other writer coordination protocols. These fields probably don't have any value to readers and would only serve to pollute reader attentional space and open the possibility of harmful coupling.
2. Table owners may wish to write certain processing state information into the document for permanent audit-like use cases. Like the coordination attributes, these attributes will pollute the interface and probably result in harmful coupling.
3. Owners are currently free to add new fields to their documents in the faith that new attributes should not break existing readers. However, deprecating and removing fields has no clear path. Being able to toggle attributes off and back on again provides a valuable mechanism in the context of a larger deprecation protocol.

How to Test and Verify

1. Check out the project; run Emo
2. Write some documents with hidden attributes.
3. Verify that you cannot see the hidden fields unless you specify showHiddenFields=true
4. Verify you cannot specify showHiddenFields=true unless you have update permission to the table
5. Verify you cannot see hidden fields in databus events or in stash
6. Write some conditional deltas, conditioned on the values of hidden attributes.
7. Verify Emo correctly computes the conditions (hidenness has no impact on conditional logic).

Risk

Level

Medium

We are introducing a new concept to Emo. This is something we should do very, very carefully. Try to erase my use case from your mind and think about what a new user will make of this feature. Does the documentation make the intended usage obvious? Is the feature sufficiently intuitive?

Issue Checklist

• Make sure to label the issue.

• Well documented description of use-cases and bugs.

[vvcephei]

I did some design work, which I copy/paste here:

GET /sor/1/{table}/{key}?showHidden=true|false

PUT /bus/1/{subscription}?showHidden=true|false <<< 'alwaysTrue()'

NESTED========================================

if {..,{"~hidden":{..,"sourceVersion": lte(44) } } } then {.., "status": "APPROVED" } end

{
  "title": "Yay!",
  "text": "it works great",
  "rating" 4,
  "~id": "1234",
  "~table": "documents",
  "~version": 14,
  "~hidden": {
    "sourceVersion": 43,
    "author": "content-processing",
    "provenance": {
      "2017-01-01T00:00:00.000Z": "initial submit",
      "2017-01-01T00:00:01.000Z": "photo upload",
      "2017-01-01T00:00:30.000Z": "publish",
      "2017-01-03T00:09:00.000Z": "moderation approve",
      "2017-02-01T00:12:00.000Z": "client remove"
    }
  }
}

NAMESPACE========================================

if {..,{"~hidden.sourceVersion": lte(44) } } then {.., "status": "APPROVED" } end

{
  "title": "Yay!",
  "text": "it works great",
  "rating" 4,
  "~id": "1234",
  "~table": "documents",
  "~version": 14,
  "~hidden.sourceVersion": 43,
  "~hidden.author": "content-processing",
  "~hidden.provenance": {
    "2017-01-01T00:00:00.000Z": "initial submit",
    "2017-01-01T00:00:01.000Z": "photo upload",
    "2017-01-01T00:00:30.000Z": "publish",
    "2017-01-03T00:09:00.000Z": "moderation approve",
    "2017-02-01T00:12:00.000Z": "client remove"
  }
}

SIGIL========================================

if {..,{"_sourceVersion": lte(44) } } then {.., "status": "APPROVED" } end

{
  "title": "Yay!",
  "text": "it works great",
  "rating" 4,
  "~id": "1234",
  "~table": "documents",
  "~version": 14
  "_sourceVersion": 43,
  "_author": "content-processing",
  "_provenance": {
    "2017-01-01T00:00:00.000Z": "initial submit",
    "2017-01-01T00:00:01.000Z": "photo upload",
    "2017-01-01T00:00:30.000Z": "publish",
    "2017-01-03T00:09:00.000Z": "moderation approve",
    "2017-02-01T00:12:00.000Z": "client remove"
  }
}

The nested strategy makes delta operations more complicated to write without offering substantially more informaion.

The sigil strategy is very compact, but introduces more UX complexity, since you have to remember what "_" means.

I think the namespace strategy achieves the best balance of compactness and self-documentation.

Note that this is an implicit re-definition of "~" fields from being "emo implicits" to being more generally "emo-special" in some way or another, since
hiddenness isn't an implicit property of the document. So I'll have to remember to update the documentation accordingly.

I think I want to leave the door open to letting writers create databus subscriptions with hidden fields included, but I don't think I want to implement it right away. Let's see if anyone actually wants it.