Skip to content
This repository has been archived by the owner on Oct 29, 2019. It is now read-only.

need rationale in spec for fully qualified DID references as the value of "id" fields #97

Closed
dhh1128 opened this issue Aug 6, 2018 · 23 comments
Labels
discuss Wider discussion in an issue or meeting required before merging

Comments

@dhh1128
Copy link
Contributor

dhh1128 commented Aug 6, 2018

After a reading of the spec, I am somewhat mystified by the usage of fully qualified DID references as the value of an "id" field. We have stuff like this from example 8:

 "service": [{
    "id": "did:example:123456789abcdefghi;openid",

... instead of:

 "service": [{
    "id": "openid",

... and stuff like this from example 5:

  "publicKey": [... {
    "id": "did:example:123456789abcdefghi#keys-2",
    "type": "Ed25519VerificationKey2018",
    "owner": "did:example:pqrstuvwxyz0987654321",
    "publicKeyBase58": "H3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
  }, {

When I would have expected:

  "publicKey": [... {
    "id": "keys-2",
    "type": "Ed25519VerificationKey2018",
    "owner": "did:example:pqrstuvwxyz0987654321",
    "publicKeyBase58": "H3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
  }, {

This second chunk of code is particularly interesting as an example because it references an owner that is NOT the owner of the provided DID Doc. The normative URI for such a key is that other owner's DID doc (perhaps at did:example:pqrstuvwxyz0987654321#keys-407). The fact that we allow a fully qualified DID reference here makes me think this ought to be possible--but apparently we have a constraint that every value of "id" inside the doc must refer to an item inside the doc, despite being fully qualified?

It feels like we are confusing the old SGML/XML/HTML notions of ID and IDREF; we're using an IDREF where we should be using an ID. In html, the value of a link take is an ID, and the hyperlink that points to it is an IDREF. You'd never look in an html doc and see <a name="http://mydomain.com/path/to/my/doc#fragment"...>; you'd just expect to see <a name="fragment">.

Now, perhaps this analysis is missing something and there is a compelling reason why we need fully qualified values. If that is the case, then I think the spec needs a paragraph or two explaining why. My major point is that such a pattern is surprising enough that it needs to be justified and documented. My minor point is that it may be the wrong decision in the first place.

@msporny
Copy link
Contributor

msporny commented Aug 7, 2018

It feels like we are confusing the old SGML/XML/HTML notions of ID and IDREF; we're using an IDREF where we should be using an ID.

The model in JSON-LD has been simplified from the SGML/XML/HTML notions because those latter mechanisms have led to many confusions and mistakes from a developer/author perspective. While SGML/XML/HTML have multiple types of identifiers in the documents... for example creating formA <form name="foo"> and then formB <form id="foo"> and then formC <form resource="#foo"> and then doing vs <a href="#foo"> have led to developers getting very confused about what that href is supposed to point to (note: there are at least three different valid answers depending on document ordering and depending on which application is processing the HTML). So, we need to avoid doing that to people working with DID Documents.

There is only one valid answer in JSON-LD for id: The value is either an absolute URI (e.g. did:example:1234;foo) or it is a relative URL (e.g ;foo or #foo, foo, etc). If it's a relative URL, you take the base URL for the document (which, for DID Documents, is the DID itself) and you append the value in id to the base. More on @base processing in JSON-LD here: https://json-ld.org/spec/latest/json-ld/#base-iri

So, all that to say that "id": "#keys-2", is a valid expression and will be appended to the base URL (which is the DID). The same goes for "service": [{ "id": ";openid", ... and it's more explicit than name and id in HTML, both of which only exist due to historical baggage reasons.

The reason the specs don't use relative URLs is because I'm concerned about @base being injected into the document and causing all sorts of havoc. It's an attack vector, so rather than allow relative URLs from the beginning for those sorts of identifiers, I thought it might be best to use absolute URLs in the beginning (be explicit and restrictive to start... and then we can loosen up the requirements in time). Also, keep in mind that DID Documents are not meant to be authored by humans, like HTML was.

@msporny
Copy link
Contributor

msporny commented Aug 7, 2018

That said, happy to put the rationale in the spec... just trying to give some background here first to see if you have a preference on exactly what language is placed into the specification.

@dhh1128
Copy link
Contributor Author

dhh1128 commented Aug 23, 2018

@msporny Are some words missing from your explanation? The paragraph that begins "So, all that to say" doesn't make sense. Do you mean "all that to say that "id": "#keys-2" IS VALID" or "...IS INVALID"?

@msporny
Copy link
Contributor

msporny commented Aug 23, 2018

Are some words missing from your explanation?

Yes. I had to re-read the entire thread to figure out what I was trying to say 16 days ago. Your interpretation is correct. I've fixed the original text in that respect... but there was another error in my response... id: "foo" is also valid, so to make sure I was correct in my interpretation of what would happen I started reading RFC 3986, specifically the section on reference resolution: https://tools.ietf.org/html/rfc3986#section-5

... and after doing some testing with a few URI resolvers, it looks like there is a bigger problem wrt. resolving relative URLs in DID documents. I think it may be a bug in URL resolvers as the types of URIs we use look more like URNs than they do URLs... and it looks like the base parsing rules for URIs assumed something that looked more like a URL.

So, it's going to take a couple of hours to really figure out what the right answer is. More specifically, and in the worst case, if we can redefine base processing for DID Documents. In the best case, we just tripped over a bug in a number of the URL processing libraries, which isn't surprising given that I wouldn't expect anyone has tried to do relative URIs wrt. URNs.

/cc @dlongley @gkellogg

@dhh1128
Copy link
Contributor Author

dhh1128 commented Aug 29, 2018

@msporny I am happy to hear that fragments (fragment-only IDs) are supported.

However, if it is true that it is valid to make the value of an "id" property equal just to a fragment and not to a full qualified URI, then I am scratching my head a bit. The spec seems to be incompatible with this permissiveness in section 4.3 when it provides the algorithm for processing publicKey values: "If the id property of the object matches value, set result to the object." I started a PR that changes this, but did not submit it, pending the resolution of this issue. Here's a link to the relevant line: dhh1128@2217d78#diff-eacf331f0ffc35d4b482f1d15a887d3bL1245

I am also puzzled about how you would decide whether the value you are looking at is a fragment or is fullly qualified. Do you have to inspect the value and guess, based on whether it begins with did:...?

@dlongley
Copy link
Contributor

This works with the WHATWG URL parser, btw:

const did = 'did:v1:nym:1234';
if(new URL('#key', did).toString() === new URL('did:v1:nym:1234#key', did).toString()) {
  console.log('works!');
}

@dlongley
Copy link
Contributor

@dhh1128,

I am also puzzled about how you would decide whether the value you are looking at is a fragment or is fullly qualified.

Absolute URLs have a colon (:) in them. If an id doesn't have a colon, it's relative to the document identifier (in this case, the DID).

@peacekeeper
Copy link
Member

Are we actually saying anywhere in the spec that the base IRI of the DID Document is the DID itself? I created an issue for this topic in DID Resolution here: w3c/did-resolution#9

@Fak3
Copy link

Fak3 commented Sep 28, 2018

refs #72 #90

@csuwildcat
Copy link
Contributor

Hey folks, we are running into a need for fragment/id-only values in this field, and it wasn't clear what the outcome was. Can someone clarify that for me?

@msporny
Copy link
Contributor

msporny commented Jan 26, 2019

Hey folks, we are running into a need for fragment/id-only values in this field, and it wasn't clear what the outcome was. Can someone clarify that for me?

This is what you should do if you just want fragment id only values:

  "id": "#foobar"

Does that answer your question, @csuwildcat?

@csuwildcat
Copy link
Contributor

csuwildcat commented Jan 26, 2019 via email

@dhh1128
Copy link
Contributor Author

dhh1128 commented Feb 19, 2019

@msporny

There is only one valid answer in JSON-LD for id: The value is either an absolute URI (e.g. did:example:1234;foo) or it is a relative URL (e.g ;foo or #foo, foo, etc). If it's a relative URL, you take the base URL for the document (which, for DID Documents, is the DID itself) and you append the value in id to the base. More on @base processing in JSON-LD here: https://json-ld.org/spec/latest/json-ld/#base-iri

This would imply that foo is a valid relative URL. But it can't be, if the algorithm for combining a fragment with the base is pure concatenation. We'd end up combining DID "did:example:abc" with "foo" and getting "did:example:abcfoo", and not being able to tell where, if at all, it contains a fragment. So I am assuming that A) the correct answer is to combine, without any delimiter; and B) the ID of something inside a doc must begin with a character that is NOT a valid character in a DID value, so we can tell that we're now in the fragment portion. This is compatible with your summary for @csuwildcat that we're using "id": "#foobar", but would require us to update the ABNF to make it clear that the fragment must begin a particular way. I'm not convinced that this is easier or less error-prone than simply saying that "#" is a delimiter and we should be doing "id": "foo" when declaring the id, and reference_to_id: "#foo" when referencing it.

@dhh1128
Copy link
Contributor Author

dhh1128 commented Jul 15, 2019

After thinking about this for a while, and after another interesting exchange on this topic here, I'd like to record my new perspective. @peacekeeper

As I understand it, @msporny is advocating strongly for the notion that all id properties in a DID doc should be absolute/fully qualified URIs--and part of his reasoning is that the handling of relative URIs has been the basis of much confusion, and possibly the basis of some security vulnerabilities, in HTML. Is that a faithful summary, Manu?

I would point out that, while this choice is a legal one per specs like RFC3986, and while it may be a good one for the reasons Manu is offering, it is NOT an obvious one. It clearly violates the principle of least surprise. The majority of web developers who encounter the DID spec--and there are LOT of them--have never seen an example.com/foo.html, in their entire careers, that looks like this:

<h3 id="http://example.com/foo.html#my-cool-topic">My Cool Topic</h3>

I guess that part of the reason why this is (almost?) unheard of in HTML is that it would mean that every time HTML content gets moved, all the id properties inside that content would need to be rewritten. In the HTML content type, fragment handling means that all ids are declared as document-unique only--not globally unique--and that looking up the fragment means finding an id or name or resource property inside the doc. RFC 3986 explicitly allows fragment handling to be done in other ways, but my own HTML experience so strongly colored my reading of the DID spec that I had a very, very hard time liberating myself from that mental model. Even though the DID spec has language requiring a different way--no gluing an id onto the document base URI, but simply matching id value while ignoring base--I still didn't get it, for over a year, after multiple readings of the DID spec and multiple conversations in comment streams like this one. I mention this because I want to make it clear just how much potential for confusion is associated with this choice.

I believe the theory is (and Manu, here I'm trying to fairly articulate your reasoning, but please correct me if I don't get it right) that HTML's issue (absolute IDs would need rewriting each time a doc moves) doesn't apply in the same way to DID docs, because DID doc location can never change. The "location" is the DID itself, which is assumed to be rooted on a global and never-relocatable source of truth.

I think this is is a seductive mental model, but ultimately limiting.

In did:peer DID docs, the location of the DID is not a ledger, but rather a DID doc. The DID doc can be copied anywhere; if Alice and Bob each use peer DIDs and each have 3 agents, then each of their docs will exist in at least 6 locations. If a peer DID is used by a corporation that has dozens or hundreds of agents, the proliferation explodes. And the resolution process doesn't go back to some source of truth (even to a local microledger) to look up an ID; it looks inside the document itself--whichever instance and version of the doc happens to be at hand/cached. This makes relative references inside a DID doc a lot more rational; the sort of external resolution implied by an absolute URI doesn't really apply.

It's not just peer DIDs that have this property, though. If a ledger that roots DIDs ever gets forked, then all DIDs from the common ancestor ledger exist in two places, not just one. We can force one of the forks to update all id properties in all DID docs that it stores. And perhaps we should. But until we do, the same data exists in more than one place, as with peer DID docs--and resolution has to take place within a context; the absolute URI doesn't actually eliminate that.

A similar issue can exist in "network of networks" situations. We have not specified what happens if two different instances of a given blockchain both register the same DID. I know that some methods are requiring different prefixes to eliminate this ambiguity -- did:sov and did:sov:test are Sovrin's mainnet and buildernet, respectively -- but unless the spec says this behavior is required, there is nothing to stop a big corporation from registering the same DID did:sov on, say, the "findy" instance of Indy that's running in Finland, and on the Sovrin mainnet, and on another Sovrin or Indy instance running in China or Mexico (both of which are scenarios that are/have been explored). And even if the big corporation agrees not to do this, there is nothing to stop one ledger from importing transactions from another; China could do this as a way to keep all DID resolution inside the Great Firewall, for example.

But even if you discount peer DIDs, network forking, and cross-registered DIDs as anomalies, there is still another reason why I think the mental model of all IDs being absolute URIs is flawed: time. See issue #238 .

The point of absolute hyperlinks is to break the association between the link and the context; "#my-cool-topic" could point to lots of different things, whereas "https://example.com/foo.html#my-cool-topic" is unambiguous. But #my-cool-topic in an HTML doc has a subtle virtue with respect to time: when it is resolved against the current document, it is guaranteed to refer to a view of that doc where reference and referent are aligned in time. When it is resolved by going out to the internet, doing a DNS resolution, fetching a resource, and then doing fragment resolution, the temporal guarantee disappears. This is a common reason for broken hyperlinks.

I would like a DID doc to have the same quality. If I am looking at a cached version of a DID doc, and I'm trying to resolve a reference to a key's id as it appears in the authentication section, I don't want it to be the case that the cached version of the DID doc is considered a tolerable-but-less-than-truly-canonical source of truth about what that id actually means. Rather, I want it to be the case that the cached DID doc is actually more canonical than whatever is on the ledger, because what's on the ledger isn't guaranteed to align perfectly with the timing that I need. In other words, resolving the id property relative to a doc, not absolutely against a ledger that serves the doc, is more correct.

If you buy that argument, then I claim absolute URIs are actually undesirable, except for external references. All references inside a DID doc should be relative, always. This is convenient for peer DIDs. It solves the problem of rewriting DID docs when a ledger forks. It creates no problems for cross-registration (though I hope cross-registration is avoided for other reasons). And it makes temporal alignment of references to ids, and of values of ids, guaranteed. The only time you'd ever have an absolute reference to a local id is if you wanted to force external resolution. This is in fact exactly how things work in HTML, except that we can clean up HTML's muddy rules about base.

Perhaps there are other ways to resolve these concerns. I think I could be talked into an absolute URI strategy if the planets aligned right. But whatever strategy we take, I hope at least the following takeaway can be agreed to:

The spec needs to explain why it's handling ids, fragments, and relative/absolute URIs so differently from HTML, so the large population of web developers out there aren't surprised or subtly non-conforming.

@talltree
Copy link
Contributor

@dhh1128 Thank you for taking the time to explain your reasoning all the way through. I find your logic to be quite compelling. Just to make sure I fully understand: the "base URI" for a DID document would always the subject DID that is value of the one-and-only required property in a DID document—the top-level id element. Yes?

This would be true no matter where you found the DID document (although you would separately need to determine if that location was in fact authoritative for that DID).

If so, then you could always unambiguously resolve the value of any fragment contained in other id element in the DID document by appending it to the subject DID. No exceptions. No reason to ever look outside the DID document.

Do I have that right?

@dhh1128
Copy link
Contributor Author

dhh1128 commented Jul 16, 2019 via email

@msporny
Copy link
Contributor

msporny commented Jul 16, 2019

Just to make sure I fully understand: the "base URI" for a DID document would always the subject DID that is value of the one-and-only required property in a DID document—the top-level id element. Yes?

So, here's the issue with that... what about did:web? That is, DID Methods that are based off of the DNS system (that change locations/domains)?

All this stuff works for DLTs where the BASE doesn't change... but not for DID Methods where BASE might change.

Also, to be clear - I'm on the fence wrt. this decision and have been for a while. There are benefits and drawbacks to each approach and I wish the decision was easier than it is. If we support relative references, it makes developer lives easier but at the cost of potential security vulnerabilities. If we assert strict references, it (arguably) leads to more verbose DID Documents. I'm sure there are other benefits/drawbacks that I'm not thinking about right now, but in any case, this isn't a decision we should make lightly and I expect it'll be contentious because there is no clear answer (at least, that I can see).

@dhh1128
Copy link
Contributor Author

dhh1128 commented Jul 16, 2019

@msporny , WRT your question "what about did:web", my response is: Yes, what about it? How does making IDs absolute make anything different or better for did:web? That's framed rhetorically, but I'll propose an answer: it doesn't.

If I am an employee of AcmeCorp and register a did:web off of AcmeCorp's web site, and then AcmeCorp is acquired by BiggerFishCorp, what should happen to my DID? It might be the case that did:web:acmecorp.com/mydid should change to be did:web:biggerfishcorp.com/acme/mydid, or that those two DIDs should become synonyms for a transition period, or that BiggerFishCorp wants all Acme DIDs to become invalid. I can imagine arguments for all three positions. Making IDs absolute is convenient with one of these positions, but not the others. Shouldn't Acme and BiggerFishCorp be able to make it work however they want, since did:web's purpose is to tie DID management to web site management?

Set aside the acquisition scenario for a moment, and just think about web-based methods in the abstract.

Web URIs offer good enough resolution for most practical purposes, and DNS is a wonderful invention. I'm not down on it in any way. But a fully qualified web URI can be multihomed, and there is no guarantee that two web servers will return the same content for it at any given point in time. This is particularly true during a period of time when DNS records are updating. Even if you discount the issues with misaligned-but-theoretically-equivalent servers, and the issues with how DNS record changes propagate, you still have the problem of multiple layers of local DNS caches. On any given computing device, there are layers upon layers of DNS caching that are difficult to control in a fine-grained way, that undermine the sort of precision you're hoping to achieve with the absolute URI requirement. You simply cannot make a DNS-based resolution safer or more precise than the properties of ambiguity and latency that are inherent in DNS, no matter what. And I think that's okay. I'm not dissing did:web. I'm just asserting that requiring absolute URIs is a bit like chasing the leprechaun's gold at the far end of the rainbow--we'll keep running over the hill, hoping to find the treasure, only to be disappointed. Requiring absolute URIs as node id properties doesn't materially improve the security or precision of did:web -- but it does make the whole system more fragile and less adaptable to different usage patterns.

@talltree
Copy link
Contributor

I want to quickly address one passage in @dhh1128 's previous comment:

If I am an employee of AcmeCorp and register a did:web off of AcmeCorp's web site, and then AcmeCorp is acquired by BiggerFishCorp, what should happen to my DID? It might be the case that did:web:acmecorp.com/mydid should change to be did:web:biggerfishcorp.com/acme/mydid, or that those two DIDs should become synonyms for a transition period, or that BiggerFishCorp wants all Acme DIDs to become invalid. I can imagine arguments for all three positions.

Stop right there. DIDs do not change. Ever. Period. Full stop. End of story.

DIDs effectively meet all the functional requirements of URNs. To quote from that spec (emphasis added):

URN namespaces inherit certain rights and responsibilities by the
nature of URNs, in particular:

  1. They uphold the general principles of a well-managed URN
    namespace by providing persistent identification of resources and
    unique assignment of names in accordance with a common
    definition.

In short, the purpose of a URN is to provide an identifier for a resource that never needs to change and will never need to be assigned to another resource. That was the first of the four requirements we started with for the original DID spec three years ago. (Resolvability, cryptographic verifiability, and no dependence on a centralized registration authority were the other three.)

So the whole question of the DID for a DID document ever changing should be taken completely and permanently off the table. (That does not, of course, mean that the contents of the DID document can't change—with the exception of static DID documents, that's one of the primary purposes of DID documents.)

That doesn't mean that, to use the did:web: example, that a DID based on a domain name couldn't stop working, and that the DID subject couldn't be assigned a new DID based on a new domain name as Daniel suggests. But:

  1. Those are two different DIDs, not "one DID changing into another". Synonymity between them (which is such a hard problem we put it out of scope for the first DID spec) would need to be proven in some other way.
  2. The fact that a domain name (or IP address) can go away or be reassigned is precisely why did:web: is, IMHO, such a bad idea. There is no requirement for persistence of URIs in the Web as a whole, so there's no way did:web: DIDs can meet the first requirement of a DID.

It was my realization earlier in this thread that once the top-level id property of a DID document is initialized with a valid DID, then that DID could never, ever change—that made me realize the benefits of just using fragments for the value of id properties inside the DID document. Note: That doesn't mean that we should disallow the use of a fully-qualified URI as the value of an id property (in fact that is required when, for whatever reason, the base URI for the fully-qualified URI is not the subject DID, but some external DID, or even a non-DID URI).

But it definitely convinces me that unqualified fragments should not just be allowed as the value of id properties in the DID document, but that we should actively encourage developers to use them. It's easier to read, shorter, and leverages the widely adopted standard Web practice of using fragments to address secondary resources (e.g., bookmarks) within HTML documents.

@csuwildcat
Copy link
Contributor

csuwildcat commented Jul 17, 2019 via email

@dhh1128
Copy link
Contributor Author

dhh1128 commented Jul 17, 2019

Thank you for the comment, @talltree. Yes, that was sloppy of me to talk about a DID changing, and it needed correction. What I meant in my did:web scenario is that the person who owns the DID thinks of the evolution of state as if their DID changed, when what really happens underneath is that an old DID is retired and a new DID is used. And I agree that, following DID canon, there should be no reason to make this transition since both DIDs are theoretically referring to the same resource--which makes did:web somewhat of an odd duck.

@msporny
Copy link
Contributor

msporny commented Jul 17, 2019

@csuwildcat wrote:

After reading all this, I'm just hoping we're arriving at an agreement that we will allow (continue to allow?) ID fragments without the proceeding DID GUID string.

I think so, yes... at least, I think it'll be very difficult to prevent that... and that's totally fine if you're using a DLT (as far as we can tell). The question is, do fragment identifiers cause problems for non-DLT-based DID Methods. My concern is that they do, but I haven't been able to articulate a concrete attack where that's an issue (due to workload... I haven't even had time to read @dhh1128's comments above).

... but, it sounds like requiring the entire DID is not a thing... and if we were to make it a thing, DID Resolvers could always modify the "just fragment identifiers" in a DID Document to be the entire DID plus fragment. So, I don't think you have to worry about it @csuwildcat ... I'm not seeing strong pressure to go more strict.

@jandrieu
Copy link
Contributor

Closing as we have adopted this issue in the new DIDWG repo.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
discuss Wider discussion in an issue or meeting required before merging
Projects
None yet
Development

No branches or pull requests

9 participants