From 6ab5888d05ef3f445e26a1ca81081a7b912016ed Mon Sep 17 00:00:00 2001 From: Tab Atkins-Bittner Date: Mon, 8 May 2023 16:09:57 -0700 Subject: [PATCH] [css-syntax-1] First draft of parser rewrite to support new Nesting behavior. Significant editorial changes to support a better token stream abstraction. --- css-syntax-3/Overview.bs | 936 ++++++++++++++++++++------------------- 1 file changed, 473 insertions(+), 463 deletions(-) diff --git a/css-syntax-3/Overview.bs b/css-syntax-3/Overview.bs index dc6e2f743b3..7297436d8f3 100644 --- a/css-syntax-3/Overview.bs +++ b/css-syntax-3/Overview.bs @@ -19,11 +19,15 @@ Ignored Vars: +b, -b, foo 
-spec:css-text-decor-3; type:property; text:text-decoration
+spec:css-cascade-5; type:dfn; text:property
 spec:css-color-4; type:property; text:color
+spec:css-text-decor-3; type:property; text:text-decoration
 spec:css-transforms-1; type:function; text:translatex()
 spec:encoding-1; type:dfn; text:get an encoding
-spec:html; type:element; text:a
+spec:html; type:element;
+	text:a
+	text:style
+	text:script
 spec:infra; type:dfn;
 	text:string
 	text:list
@@ -442,11 +446,11 @@ Tokenization
 
 	To tokenize a stream of code points
 	into a stream of CSS tokens |input|,
-	repeatedly consume a token from |input|
+	repeatedly [=tokenizer/consume a token=] from |input|
 	until an <> is reached,
 	pushing each of the returned tokens into a stream.
 
-	Note: Each call to the consume a token algorithm
+	Note: Each call to the [=tokenizer/consume a token=] algorithm
 	returns a single token,
 	so it can also be used "on-demand" to tokenize a stream of code points during parsing,
 	if so desired.
@@ -892,7 +896,7 @@ Definitions
 		

 			The representation of a token
 			is the subsequence of the input stream
-			consumed by the invocation of the consume a token algorithm
+			consumed by the invocation of the [=tokenizer/consume a token=] algorithm
 			that produced it.
 			This is preserved for a few algorithms that rely on subtle details of the input text,
 			which a simple "re-serialization" of the tokens might disturb.
@@ -935,7 +939,7 @@ Tokenizer Algorithms
 

 Consume a token

 
-	This section describes how to consume a token from a stream of code points.
+	This section describes how to consume a token from a stream of code points.
 	It will return a single token of any type.
 
 	Consume comments.
@@ -1687,103 +1691,15 @@ Consume the remnants of a bad url
 

 Parsing

 
-	The input to the parsing stage is a stream or list of tokens from the tokenization stage.
-	The output depends on how the parser is invoked,
-	as defined by the entry points listed later in this section.
-	The parser output can consist of at-rules,
-	qualified rules,
-	and/or declarations.
-
-	The parser's output is constructed according to the fundamental syntax of CSS,
-	without regards for the validity of any specific item.
-	Implementations may check the validity of items as they are returned by the various parser algorithms
-	and treat the algorithm as returning nothing if the item was invalid according to the implementation's own grammar knowledge,
-	or may construct a full tree as specified
-	and "clean up" afterwards by removing any invalid items.
+	The CSS parser converts a [=token stream=]
+	(produced by the tokenization process,
+	defined earlier in this spec)
+	into one or more of several CSS constructs
+	(depending on which parsing algorithm is invoked).
 
 	The items that can appear in the tree are:
 
-	

-		
[=at-rule=]
-		

-			An [=at-rule=] has a name,
-			a prelude consisting of a list of component values,
-			and an optional block consisting of a simple {} block.
-
-			Note: This specification places no limits on what an at-rule's block may contain.
-			Individual at-rules must define whether they accept a block,
-			and if so,
-			how to parse it
-			(preferably using one of the parser algorithms or entry points defined in this specification).
-
-		
qualified rule
-		

-			A qualified rule has
-			a prelude consisting of a list of component values,
-			and a block consisting of a simple {} block.
-
-			Note: Most qualified rules will be style rules,
-			where the prelude is a selector [[SELECT]]
-			and the block a list of declarations.
-
-		
declaration
-		

-			Conceptually, declarations are a particular instance of
-			associating a property or descriptor name with a value.
-			Syntactically, a declaration has a name,
-			a value consisting of a list of component values,
-			and an important flag which is initially unset.
-
-			Declarations are further categorized
-			as property declarations or descriptor declarations,
-			with the former setting CSS [=properties=]
-			and appearing most often in qualified rules
-			and the latter setting CSS [=descriptors=],
-			which appear only in at-rules.
-			(This categorization does not occur at the Syntax level;
-			instead, it is a product of where the declaration appears,
-			and is defined by the respective specifications defining the given rule.)
 
-		
component value
-		

-			A component value is one of the [=preserved tokens=],
-			a [=function=],
-			or a [=simple block=].
-
-		
preserved tokens
-		

-			Any token produced by the tokenizer
-			except for <>s,
-			<{-token>s,
-			<(-token>s,
-			and <[-token>s.
-
-			Note: The non-[=preserved tokens=] listed above are always consumed into higher-level objects,
-			either functions or simple blocks,
-			and so never appear in any parser output themselves.
-
-			Note: The tokens <}-token>s, <)-token>s, <]-token>, <>, and <> are always parse errors,
-			but they are preserved in the token stream by this specification to allow other specs,
-			such as Media Queries,
-			to define more fine-grained error-handling
-			than just dropping an entire declaration or block.
-
-		
function
-		

-			A function has a name
-			and a value consisting of a list of component values.
-
-		
simple block
-		
{}-block
-		
[]-block
-		
()-block
-		

-			A simple block has an associated token (either a <[-token>, <(-token>, or <{-token>)
-			and a value consisting of a list of component values.
-
-			[={}-block=], [=[]-block=], and [=()-block=] refer specifically
-			to a [=simple block=] with that corresponding associated token.
-	

 
 
 
-

-Definitions

+

+CSS Parsing Results

 
-	

-		
current input token
+	The result of parsing can be any of the following
+	(or lists of these):
+
+	

+		: rule
+		:: A [=rule=] is either an [=at-rule=]
+			or a [=qualified rule=].
+
+		
[=at-rule=]
 		

-			The token or component value currently being operated on, from the list of tokens produced by the tokenizer.
+			An [=at-rule=] has a name which is a [=string=],
+			a prelude consisting of a list of [=component values=].
+			[=Block at-rules=]
+			(ending in a {}-block)
+			will additionally have
+			a list of [=declarations=]
+			and a list of child [=rules=].
+			([=Statement at-rules=],
+			ending in a semicolon,
+			will not.)
 
-		
next input token
+		
qualified rule
 		

-			The token or component value following the current input token in the list of tokens produced by the tokenizer.
-			If there isn't a token following the current input token,
-			the next input token is an <>.
+			A qualified rule has
+			a prelude consisting of a list of component values,
+			a list of declarations,
+			and a list of child rules.
+
+			Note: Most qualified rules will be style rules,
+			where the prelude is a selector [[selectors-4]]
+			and its declarations are [=properties=].
 
-		
<>
+		
declaration
 		

-			A conceptual token representing the end of the list of tokens.
-			Whenever the list of tokens is empty,
-			the next input token is always an <>.
+			A declaration has a name which is a [=string=],
+			a value consisting of a list of [=component values=],
+			and an important flag which is initially unset.
+
+			Declarations are further categorized
+			as property declarations or descriptor declarations,
+			with the former setting CSS [=properties=]
+			and appearing most often in qualified rules
+			and the latter setting CSS [=descriptors=],
+			which appear only in at-rules.
+			(This categorization does not occur at the Syntax level;
+			instead, it is a product of where the declaration appears,
+			and is defined by the respective specifications defining the given rule.)
 
-		
consume the next input token
+		
component value
 		

-			Let the current input token be the current next input token,
-			adjusting the next input token accordingly.
+			A component value is one of the [=preserved tokens=],
+			a [=function=],
+			or a [=simple block=].
 
-		
reconsume the current input token
+		
preserved tokens
 		

-			The next time an algorithm instructs you to consume the next input token,
-			instead do nothing
-			(retain the current input token unchanged).
+			Any token produced by the tokenizer
+			except for <>s,
+			<{-token>s,
+			<(-token>s,
+			and <[-token>s.
+
+			Note: The non-[=preserved tokens=] listed above are always consumed into higher-level objects,
+			either functions or simple blocks,
+			and so never appear in any parser output themselves.
+
+			Note: The tokens <}-token>s, <)-token>s, <]-token>, <>, and <> are always parse errors,
+			but they are preserved in the token stream by this specification to allow other specs,
+			such as Media Queries,
+			to define more fine-grained error-handling
+			than just dropping an entire declaration or block.
+
+		
function
+		

+			A function has a name
+			and a value consisting of a list of [=component values=].
+
+		
simple block
+		
{}-block
+		
[]-block
+		
()-block
+		

+			A simple block has an associated token (either a <[-token>, <(-token>, or <{-token>)
+			and a value consisting of a list of component values.
+
+			[={}-block=], [=[]-block=], and [=()-block=] refer specifically
+			to a [=simple block=] with that corresponding associated token.
+	

+
+

+Token Streams

+
+	A token stream is a [=struct=]
+	representing a stream of [=tokens=] and/or [=component values=].
+	It has the following [=struct/items=]:
+
+	

+		: tokens
+		:: A [=list=] of [=tokens=] and/or [=component values=].
+
+			Note: This specification assumes, for simplicity,
+			that the input stream has been fully tokenized
+			before parsing begins.
+			However, the parsing algorithms only use one token of "lookahead",
+			so in practice tokenization and parsing
+			can be done in lockstep.
+
+		: index
+		:: An index into the [=token stream/tokens=],
+			representing the progress of parsing.
+			It starts at 0 initially.
+
+			Note: Aside from [=token stream/marking=],
+			the [=token stream/index=] never goes backwards.
+			Thus the already-processed prefix of [=token stream/tokens=]
+			can be eagerly discarded as it's processed.
+
+		: marked indexes
+		:: A [=stack=] of index values,
+			representing points that the parser might return to.
+			It starts empty initially.
 	

 
+	Several operations can be performed on a [=token stream=]:
+
+	

+		: next token
+		:: The item of [=token stream/tokens=] at [=token stream/index=].
+
+			If that index would be out-of-bounds past the end of the list,
+			it's instead an <>.
+
+		: empty
+		:: A token stream is [=token stream/empty=]
+			if the [=next token=] is an <>.
+
+		: consume a token
+		:: Let |token| be the [=token stream/next token=].
+			Increment [=token stream/index=],
+			then return |token|.
+
+		: discard a token
+		:: If the [=token stream=] is not [=empty=],
+			increment [=token stream/index=].
+
+		: mark
+		:: Append [=token stream/index=] to [=token stream/marked indexes=].
+
+		: restore a mark
+		:: [=stack/Pop=] from [=token stream/marked indexes=],
+			and set [=token stream/index=] to the popped value.
+
+		: discard a mark
+		:: [=stack/Pop=] from [=token stream/marked indexes=],
+			and do nothing with the popped value.
+
+		: discard whitespace
+		:: While the [=next token=] is a <>,
+			[=discard a token=].
+
+		: process
+		:: To [=token stream/process=],
+			given a following list of token types and associated actions,
+			perform the action associated with the [=next token=].
+			Repeat until one of the actions returns something,
+			then return that.
+	

+
+	An <> is a conceptual token,
+	not actually produced by the tokenizer,
+	used to indicate that the [=token stream=] has been exhausted.
+
 
 
 

@@ -2002,24 +2061,20 @@ Parser Entry Points

 
 		To normalize into a token stream a given |input|:
 
-		1. If |input| is a list of CSS tokens,
-			return |input|.
-
-		2. If |input| is a list of CSS component values,
-			return |input|.
+		1. If |input| is already a [=token stream=],
+			return it.
 
-			Note: The only difference between a list of tokens
-			and a list of component values
-			is that some objects that "contain" things,
-			like functions or blocks,
-			are a single entity in the component-value list,
-			but are multiple entities in a token list.
-			This makes no difference to any of the algorithms in this specification.
+		2. If |input| is a list of CSS [=tokens=] and/or [=component values=],
+			create a new [=token stream=]
+			with |input| as its [=token stream/tokens=],
+			and return it.
 
 		3. If |input| is a [=string=],
 			then [=filter code points=] from |input|,
 			[=tokenize=] the result,
-			and return the final result.
+			then create a new [=token stream=]
+			with those tokens as its [=token stream/tokens=],
+			and return it.
 
 		4. Assert: Only the preceding types should be passed as |input|.
 	
@@ -2182,7 +2237,7 @@ Parse a stylesheet
 
 		
    
 			
  1. 
-				If |input| is a byte stream for stylesheet,
+				If |input| is a byte stream for a stylesheet,
 				[=decode bytes=] from |input|,
 				and set |input| to the result.
 
@@ -2191,7 +2246,9 @@ Parse a stylesheet
 				and set |input| to the result.
 
 			
  2. 
-				Create a new stylesheet, with its location set to |location| (or null, if |location| was not passed).
+				Create a new stylesheet,
+				with its location set to |location|
+				(or null, if |location| was not passed).
 
 			
  3. 
 				Consume a list of rules from |input|,
@@ -2235,15 +2292,14 @@ Parse a rule
 				and set |input| to the result.
 
 			
  4. 
-				While the next input token from |input| is a <>,
-				consume the next input token from |input|.
+				[=token stream/Discard whitespace=] from |input|.
 
 			
  5. 
-				If the next input token from |input| is an <>,
+				If the next token from |input| is an <>,
 				return a syntax error.
 
 				Otherwise,
-				if the next input token from |input| is an <>,
+				if the next token from |input| is an <>,
 				consume an at-rule from |input|,
 				and let rule be the return value.
 
@@ -2254,11 +2310,10 @@ Parse a rule
 				return a syntax error.
 
 			
  6. 
-				While the next input token from |input| is a <>,
-				consume the next input token from |input|.
+				[=token stream/Discard whitespace=] from |input|.
 
 			
  7. 
-				If the next input token from |input| is an <>,
+				If the next token from |input| is an <>,
 				return rule.
 				Otherwise, return a syntax error.
 		

@@ -2279,11 +2334,10 @@ Parse a declaration
 				and set |input| to the result.
 
 			
  • 
-				While the next input token from |input| is a <>,
-				consume the next input token.
+				[=token stream/Discard whitespace=] from |input|.
 
 			
  • 
-				If the next input token from |input| is not an <>,
+				If the next token from |input| is not an <>,
 				return a syntax error.
 
 			
  • 
@@ -2293,44 +2347,15 @@ Parse a declaration
 		
 	
 
-
    
-Parse a style block's contents
    
-
-	Note: This algorithm parses the contents of [=style rules=],
-	which need to allow nested style rules
-	and other [=at-rules=].
-	If you don't need nested [=style rules=],
-	such as in ''@page''
-	or in ''@keyframes'' child rules,
-	use [=parse a list of declarations=].
-
-	
    
-		To parse a style block's contents from |input|:
-
-		
      
-			
    1. 
-				[=Normalize=] |input|,
-				and set |input| to the result.
-
-			
    2. 
-				Consume a style block's contents from |input|,
-				and return the result.
-		
    
-	
    
-
 
    
 Parse a list of declarations
    
 
 	Note: Despite the name,
-	this actually parses a mixed list of declarations and at-rules,
+	this actually parses a mixed list of declarations and rules.
 	as CSS 2.1 does for ''@page''.
 	Unexpected at-rules (which could be all of them, in a given context)
 	are invalid and will be ignored by the consumer.
 
-	Note: This algorithm does not handle nested [=style rules=].
-	If your use requires that,
-	use [=parse a style block's contents=].
-
 	
    
 		To parse a list of declarations from |input|:
 
@@ -2357,11 +2382,10 @@ Parse a component value
 				and set |input| to the result.
 
 			
  • 
-				While the next input token from |input| is a <>,
-				consume the next input token from |input|.
+				[=token stream/Discard whitespace=] from |input|.
 
 			
  • 
-				If the next input token from |input| is an <>,
+				If |input| is [=token stream/empty=],
 				return a syntax error.
 
 			
  • 
@@ -2369,11 +2393,10 @@ Parse a component value
 				and let value be the return value.
 
 			
  • 
-				While the next input token from |input| is a <>,
-				consume the next input token.
+				[=token stream/Discard whitespace=] from |input|.
 
 			
  • 
-				If the next input token from |input| is an <>,
+				If |input| is [=token stream/empty=],
 				return value.
 				Otherwise,
 				return a syntax error.
@@ -2386,16 +2409,11 @@ Parse a list of component values
 	
    
 		To parse a list of component values from |input|:
 
-		
      
-			
    1. 
-				[=Normalize=] |input|,
-				and set |input| to the result.
+		1. [=Normalize=] |input|,
+			and set |input| to the result.
 
-			
    2. 
-				Repeatedly consume a component value from |input| until an <> is returned,
-				appending the returned values (except the final <>) into a list.
-				Return the list.
-		
    
+		2. [=Consume a list of component values=] from |input|,
+			and return the result.
 	
    
 
 
    
@@ -2404,25 +2422,19 @@ Parse a comma-separated list of component values
    
 	
    
 		To parse a comma-separated list of component values from |input|:
 
-		
      
-			
    1. 
-				[=Normalize=] |input|,
-				and set |input| to the result.
+		1. [=Normalize=] |input|,
+			and set |input| to the result.
 
-			
    2. 
-				Let list of cvls be an initially empty list of component value lists.
+		2. Let |groups| be an empty [=list=].
 
-			
    3. 
-				Repeatedly consume a component value from |input| until an <> or <> is returned,
-				appending the returned values (except the final <> or <>) into a list.
-				Append the list to list of cvls.
+		3. While |input| is not [=token stream/empty=]:
 
-				If it was a <> that was returned,
-				repeat this step.
+			1 [=Consume a list of component values=] from |input|,
+				with <> as the stop token,
+				and append the result to |groups|.
+			2. [=Discard a token=] from |input|.
 
-			
    4. 
-				Return list of cvls.
-		
    
+		4. Return |groups|.