OAS 3.2 (and 3.3?) release planning

[handrews]

After struggling to figure out the right scope for 3.2, it occurred to me that a very quick, useful, and easy-to-implement 3.2, followed by a 3.3 that solves some more intractable problems that would otherwise have delayed 3.2 significantly, might be a better approach.

It was good that we did not put out a longer road map prior to the patch releases. Their scope was not clear, and it took us longer than expected to finish. But if we plan 3.2 and 3.3 together, that would be a pro-path-to-Moonwalk road map that would give our community some clarity. And this 3.2 "core work" as shown below would actually be quite easy to finish despite looking substantial, as much of it is already done.

3.2 core work

This is what I think 3.2 should have in it no matter what we decide regarding 3.3. Aside from the XML stuff (which, as noted, we should only do if we have a champion willing to step up) and the little bits and pieces, most of this is already well-thought-through or even already written in PR form. If this is our 3.2 release, we should be able to finish it very quickly.

• Hierarchical, categorizable tags
  • Accepted proposal:
    • 2024-09-01-Tags-Improvement
  • Issues resolved:
    • Feature: sub-tags grouping #1367
    • Introduce Summary Field on Tag Object #2843
• Referencing and identification
  • Self-identifying documents
    • Accepted proposal:
      • 2024-08-01-Self-Identification
    • Issues resolved:
      • OAS3 Tag/template to define external-components URL #1979
  • URI-references for Security Requirement -> Security Scheme
    • Previously-approved PR (was punted due to branch changes and me considering a more comprehensive solution):
      • Allow URI-references for Security Requirements (3.2.0) #3821
    • Issues resolved
      • Partially: Use URI references for Security Requirements in 3.2 #3776
    • Encouraging some security guard-rails on de-referencing
      • Add requirements or recommendations about allow/deny lists for reference target retrieval #4037
• Data modeling
  • Streaming JSON media types
    • Previously approved PR (was punted due to branch changes and me considering a more comprehensive solution):
      • Support streaming JSON formats such as json-seq #3735
    • Issues resolved:
      • Support application/json-seq and similar JSON-based sequential formats #3730
  • Server-Sent Event (SSE) text/event-stream format
    • Just treat it as a JSON object with no nesting- even easier than streaming JSON support
    • Discussion resolved:
      • An SSE Proposal #4171
  • XML (IF we can find a champion for the work; it is separate from everything else and could easily be punted to 3.3)
    • Add top level XML code generation hints #1652
    • How to represent XML elements with attributes #630
    • Formalize how to express null value in xml #3959
• HTTP
  • HTTP methods
    • @ralfhandl and I have pretty nearly converged on a good and easy solution here:
      • initial idea
      • later tweaks
    • Issues resolved:
      • Support all HTTP Methods #1747
  • Empty responses _(clarification, maybe a field to make emptiness explicit?)
    • Explicity mention that a media type object can be empty #3967
    • What is the meaning of an empty "content" definition? #2875
• Security
  • Best practices
    • remove "Optional OAuth2 security" examples #3777
    • Add info to security considerations about outdated security practices, and link in new versions #3603
  • Device authentication, OAuth2 metadata, and deprecation _(these were done on the old v3.2.0-dev branch and just need porting)
    • Migrate relevant 3.2 commits to the new branch structure #4188
  • Possible Other enhancements (I need some help figuring out the scale of these and whether they fit in 3.2 or 3.3)
    • Enhancing specification to describe token presentation mechanisms for OAuth 2.0 #3612
    • Add tokenExchange grant type to list of allowed grant types for oauth2 shape #3709
    • Add token endpoint auth method to the tokenURL in the securitySchemes #4023
    • Dynamic Scopes #3983
• Bits and pieces (minor things, we can take or leave each one independently)
  • Descriptions and names
    • description/summary for Paths Object #3722
    • Add 'name' field to servers #1821
    • OAS 3.0 and 3.1 schemas require the Response Object description property but allow it to be empty #3538
  • Examples
    • Proposal: Having an example for an entire operation in one place (request + response) #859
    • Examples in a cURL-like format, preferably at the PathItem Object #3874
  • Path templating
    • Formally define (ABNF) 3.x URL templating behavior #3256
  • API governance
    • Clarification on the purpose of the API license? #726
    • Add x-api-version to extension registry #4212 (we don't seem to have an issue for just adding api-version so using this one for now)
  • Maintainence
    • YAML JSON Schema is not enough for roundtrip-safe #2951
    • draft-bhutton-json-schema-00 is deprecated , Should the specification point to the 01 ? #3934
    • RFC 7231 has been obsoleted by RFC 9110 #3800

3.2 work if we are NOT going to also plan 3.3

These are things that I think we need to fix sooner rather than later, but the easier fixes add a lot of complexity to areas that are already far too complex. Incorporating these into 3.2 will make for more difficult-to-maintain bits of duct tape and will not get us closer to Moonwalk. We'll also have to spend time debating which sub-optimal duct-tape solutions to accept, which I find unappealing. It's always hard to do those because no one will feel satisfied with it.

This is why I think we should do these more comprehensively as explained in the 3.3 section further down:

• Media types
  • Multipart (this needs a solution like letting the encoding field use keys like "0", "1", "2", etc. to correlate Encoding Objects with positional, unnamed parts. And maybe "*" to match positions with no numbered encoding. That's the least-gross option I've come up with so far, and it's not pretty.)
    • Support for multipart/mixed (and possibly better multipart/* support in general) #3721
    • Support multipart/byteranges for 206 responses #3725
• Parameters
  • Query String (as much as I really want to support in: queryString and just use the HTTP message body solution for application/x-www-form-urlencoded to solve these, it would be far better to provide a simpler solution that addresses a lot more of our parameter problems and moves us closer to Moonwalk)
    • How to disallow unknown query parameter? #2511
    • Support for arbitrary query strings #1502
  • Headers (similar problem as for query string- modeling it all at once- but then we'd have two complex changes to two complex parameter areas, and still wouldn't have really solved these problems as we also have users who have relationships among parameters with different in values)
    • is there a way to describe that one of a couple of headers is required? #4174
  • Path (this would require some gross duct tape involving allowReserved, which is already a confusing area)
    • Use semicolon(;) in Path Parameters #1710
  • Headers (all of these boil down to modeling header structures, which we could do similar to how we handle JSON Streaming or form data, but they are more complex than JSON Streaming, and trying to build on the form data approach is extremely unappealing given the complexity of that area already, and how many things are built into it that are completely unsuitable for headers, and would require lots of "if you are doing headers, field X SHALL be ignored" etc.)
    • How to state parameter requirements in HTTP headers? #2458
    • How to define Header parameter values? #3827
    • Parameters for Media Types #2342
    • Question: Create Links from "complex" headers #2402
    • Make it easier to define link headers #667
    • Enhance Response cache-control header #2784
    • Use wildcard or regex in cookie name in Cookie Authentication #2296
    • Support for structured-headers de/serialization #1980
• Security
  • CIBA (we have solutions already in the old v3.2.0-dev branch for these, but as @SensibleWood noted in issue Representation of Client-Initiated Backchannel Authentication (CIBA) in 3.2.0 #4106, it would be much better to reconsider this comprehensively)
    • ciba-grant-3.2.0.md #3615 (note that if we were to port this, some of the field anchor names do not match the final field names and would need fixing)
  • Other enhancement (I think these would be easier if we re-think things comprehensively as proposed for CIBA, etc., but this is not my area of expertise at all)
    • Enhancing specification to describe token presentation mechanisms for OAuth 2.0 #3612
    • Add tokenExchange grant type to list of allowed grant types for oauth2 shape #3709
    • Add token endpoint auth method to the tokenURL in the securitySchemes #4023
    • Dynamic Scopes #3983

3.3

If we immediately move on to 3.3, it would be centered around two completely new mechanisms built alongside the existing features, which would be frozen (whether they would be deprecated is another discussion we can have if and when it is relevant- it doesn't really matter for now). These would address all of the issues in the previous section.

The idea here is to try out something more moonwalk-ish that can be implemented in a straightforward manner.

For Security:

• I'd basically defer to @SensibleWood and the ideas he's got going in this issue:
  • Representation of Client-Initiated Backchannel Authentication (CIBA) in 3.2.0 #4106
• These other issues might fit best here, too, although if any are simple and unrelated to 4106 they could go into 3.2
  • Enhancing specification to describe token presentation mechanisms for OAuth 2.0 #3612
  • Add tokenExchange grant type to list of allowed grant types for oauth2 shape #3709
  • Add token endpoint auth method to the tokenURL in the securitySchemes #4023
  • Dynamic Scopes #3983

For parameter and media type data modeling, my thoughts are are along the following lines, heavily influenced by the Moonwalk work on figuring out what tasks are involved in implementing the OAS.

NOTE: The purpose of this next lengthy section is to explain why we should consider a particular approach for 3.3. It is not intended to resolve the exact approach, which will need its own discussion and/or proposal.

First, let's review the problems:

The current approach to data modeling involves various combinations of the Schema, Encoding, Media Type, Parameter, and/or Header Objects:

• while Parameter and Header are never used together, each can be used with all of the other three, and all of the other three are used in some HTTP message body scenarios
• the Encoding, Header, and Parameter Objects have extremely complex field interactions, which are slightly different in each despite mostly having the same field names
• the Encoding Object places complex conditionals on behavior for form-urlencoded vs form-data
• we have always claimed that the Encoding Object supports multipart/mixed, but it never has (we dropped the claim in 3.0.4 and 3.1.1, but we know from Slack and issues that this support is needed)
• HTTP headers cannot be modeled in any meaningful way, which is a major pain point; however, none of the current mechanisms are well-suited to modeling their structure
• Many configurations of these objects appear valid but are not - sometimes they are very invalid, meaning they cannot possible produce outcomes that comply with HTTP and other standards

In truth, there are three parts to data modeling. Of these three, modeling mostly works, but we treat mapping inconsistently, and conflate it with encoding

• Modeling the in-memory data, which is done with Schema Objects
  • This is basically fine. Or, at least, we're not revamping the Schema Object again, so it's fine enough
• Mapping to non-JSON structures (ignoring XML which is handled differently)
  • This is extremely complex, involving style, explode, contentType (for multipart, and maybe sort-of for form-urlencoded?), special treatment of array schemas (for multipart only), and maybe contentMediaType and contentSchema (except that they overlap with other fields and are sometimes ignored)
  • style and explode also automatically trigger encoding/escaping behavior that is frequently undesirable or outright wrong
  • style and explode leverage RFC6570, but also include non-RFC6570 options that don't play nicely with RFC6570's assumptions
  • we are only able to map named things, because the encoding field correlates Schema and Encoding Objects by property (part) name; however, we need to handle sequential formats and formats like headers that have both a sequential and named aspect
  • we have blocked off some functionality of style and explode in some areas where it would be helpful, for various reasons – even when these reasons are good, the result is confusingly inconsistent behavior
• Encoding/escaping data
  • For JSON, this is so trivial no one even thinks about it (e.g. a " needs to be represented as either "\"" or with a unicode hex escape that I can't currently recall
  • For non-JSON, this is pretty complex, involving allowReserved, contentEncoding, rules that we inherit from other standards, and/or the Encoding Object's headers field
  • URI percent-encoding can be enabled even if no obviously-encoding-related fields are used, which is confusing
  • For URLs, the RFC6570-derived behavior mostly works, but has some odd corners. Particularly in that it does not address things like using + as an escape for space in form-urlencoded data at all, and I imagine some APIs use that convention
  • For non-URLs, the RFC6570-derived behavior is often completely wrong, e.g. cookies with binary data are base64-encoded, not percent-encoded, and while HTTP header parameter values might be percent-encoded, the primary value generally is not

We would be better off making a single-schema approach to parameters (as proposed for Moonwalk!) that addresses these three parts in a comprehensive way. I do not mean coming up with some elaborate universal way to map any JSON Schema to any other format. I mean something that is the same for all sequential formats, and the same for all named formats, and when you have something like an HTTP header that has a primary value followed by parameters (essentially a tuple where the 2nd entry is an object), it is consistent with the basic sequential and named rules. Furthermore, the mapping would be handled separately from the encoding, so it is never possible (or at least not without really working hard at it) to use the wrong kind of encoding/escaping by accident.

While this would not be trivial, it would produce something far easier to implement than any patchwork of duct tape fixes to the current pile of Objects involved, solve far more of our users' problems, and produce valuable feedback for Moonwalk before we lock in a new major version.

Open questions

• We could do something to expand the use of referencing, but do we want to?
  • Objects with unusual behavior
    • Tag part only Use URI references for Security Requirements in 3.2 #3776 (URI-references to Tag Objects are hard because they are in an array, which makes JSON Pointer fragments rather fragile)
    • Link Object: require referencing operation with unambiguous path template #4084 (it's not clear to me that it's worth fixing anything in the Link Object that would add complexity, given that it does not seem to be used much and might be more-or-less obsolete with Arazzo)
    • Did we break 3.2 compatibility by removing the Path Item $ref special case? #3734 (do we want to try again to improve the Path Item Object $ref? One option would just be to say that, for Path Item Objects, a Reference Object can overwrite any Path Object field (not just summary and description), which would move the $ref from the Path Item Object to the Reference Object and leverage the Reference Object's existing behavior)
  • Objects that do not support referencing
    • Consolidated $ref-to-Some Object feature request #3853 (one thing that we'd need to consider is avoiding other scenarios like with operationRef where the behavior depends on parent structures that might not be unique)
  • Non-Objects (we have generally leaned towards not doing these- perhaps Overlay helps?)
    • Examples referencing other examples #3902
    • allowing $ref in descriptions #2697

[handrews]

Updating with thoughts from today's TDC call:

• We should not start major work on 3.3 if it will in any way delay 3.2. The 3.3 plan exists so if people who are not going to work on 3.2 (e.g. because we're not doing a lot of security in 3.2, but are making it a major focus of 3.3) know what they can start thinking about. But we probably won't make a v3.3-dev branch until 3.2.0 goes out. 3.3 work would stay as proposals, just like we managed some 3.2 work as proposals while doing the 3.0.4 and 3.1.1 releases
• The 3.2 work that is already approved in some form (proposal, pre-branch-rework PR, etc.) is mandatory (unless we find big problems with one of them), while other things can be punted to 3.3. This is extra-true for something like XML that is self-contained and needs a champion. If none step up for 3.2, that gets punted to 3.3
• There is some debate about whether to start a data modeling / mapping registry during 3.2 or just do mappings like JSON Text Sequences as one-offs within the specification text and then make a registry that only targets the new 3.3 approach

[baywet]

@handrews I thought that during the meeting we also discussed "specifying which characters can be used for parameters names" (i.e. ABNF for this section) but I can't find any reference to that work here? Did I misunderstand? Thanks for the help!

[handrews]

@baywet it's under the "3.2 core" section under "Path templating".

[baywet]

Thanks! "I wasn't fully up yet" will be my excuse here 😂