Add support for Text fragment feature (#1545) #1600

thiru-appitap · 2024-12-25T08:15:03Z

Text Fragment feature implementation pull request. The feature follows the published URL Fragment Text Directives specification (https://wicg.github.io/scroll-to-text-fragment/).

(base) lychee % lychee -vv --include-text-fragments https://developer.mozilla.org/en-US/docs/Web/URI/Fragment/Text_fragments
...
[DEBUG] tdirective: "text=From%20the%20foregoing%20remarks%20we%20may%20gather%20an%20idea%20of%20the%20importance
[DEBUG] status: Completed
[DEBUG] result: "From the foregoing remarks we may gather an idea of the importance"
[200] https://mdn.github.io/css-examples/target-text/index.html#:~:text=From%20the%20foregoing%20remarks%20we%20may%20gather%20an%20idea%20of%20the%20importance
[DEBUG] tdirective: "text=linked%20URL,-'s%20format"
[DEBUG] status: Completed
[DEBUG] result: "linked URL"
[DEBUG] tdirective: "text=Deprecated-,attributes,attribute"
[DEBUG] status: Completed
[DEBUG] result: "attributes charset Deprecated Hinted at the character encoding of the linked URL. Note:This attribute"
[200] https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#:~:text=linked%20URL,-'s%20format&text=Deprecated-,attributes,attribute
[DEBUG] tdirective: "text=downgrade:-,The%20Referer,be%20sent,-to%20origins"
[DEBUG] status: Completed
[DEBUG] result: "The Referer header will not be sent"
[200] https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#:~:text=downgrade:-,The%20Referer,be%20sent,-to%20origins
[DEBUG] tdirective: "text=linked%20URL,defining%20a%20value"
[DEBUG] status: Completed
[DEBUG] result: "linked URL as a download. Can be used with or without a filename value: Without a value, the browser will suggest a filename/extension, generated from various sources: The Content-Disposition HTTP header The final segment in the URL path The media type (from the Content-Type header, the start of a data: URL, or Blob.type for a blob: URL) filename: defining a value"
[200] https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#:~:text=linked%20URL,defining%20a%20value
...

If the fragment directive is not found, a TextDirectiveNotFound error will be returned.

Below changes are completed:

Fragment Directive parser uses fancy-regex

this package was added as a dependency

a new flag, include-text-fragments is added to support the feature

this is a deviation from the original feature request (which asked for using the text-fragments flag itself)

Fragment (Text) Directive feature is tested on LTR sites only
new UrlExt trait is implemented to enhance Url's to support Fragment Directive
Support for multiple text fragment directives (for example, #:~:text=linked%20URL,-'s%20format&text=Deprecated-,attributes,attribute)
tests are added for validating the feature
cargo clippy & cargo tests were executed

thiru-appitap · 2024-12-25T09:09:00Z

I missed to run the clippy across the test modules - the related lint failure issues are now fixed and ready for review!

mre · 2024-12-25T11:24:31Z

lychee-lib/src/utils/url.rs

+        );
+        match url {
+            Ok(url) => {
+                eprintln!(


This could be an assertion

mre · 2025-01-10T13:47:09Z

lychee-lib/src/checker/website.rs

+                let mut status = Status::new(&response, self.accepted.clone());
+                if self.validate_text_fragments && has_fragment_directive {
+                    if let Ok(res) = response.text().await {
+                        info!("checking fragment directive...");
+                        if let Some(fd) = req_url.fragment_directive() {
+                            info!("directive: {:?}", fd.text_directives);
+                            match fd.check(&res) {
+                                Ok(stat) => {
+                                    status = stat;
+                                }
+                                Err(e) => {
+                                    return e.into();
+                                }
+                            }
+                        }
+                    }
+                }
+                status


Maybe we can move that into a function/method?

some tests for that part would also be nice

mre · 2025-01-10T13:47:53Z

lychee-lib/src/client.rs

+        assert!(res.status().is_success());
+
+        // start with suffix
+        println!("\ntesting start with suffix...");


you'll probably remove the println!s right?

mre · 2025-01-10T13:48:47Z

lychee-lib/src/extract/html/fragdirtok.rs

+
+use crate::types::TextDirective;
+
+const BLOCK_ELEMENTS: &[&str] = &[


This is a long list. Does that mean we'd have to maintain the HTML keywords here? Maybe we can avoid that as it would be an uphill battle.

mre · 2025-01-10T13:50:24Z

lychee-lib/src/extract/html/fragdirtok.rs

I like that this functionality is isolated in its own module. But it's a looot of code. 😅 Not sure what to do here, but at least the ratio of code/tests could be improved.

Perhaps we could move it into a separate crate or use an upstream crate for that? I think it would be a nice library to maintain individually as more applications could profit from it

mre · 2025-01-10T13:50:54Z

lychee-lib/src/extract/html/fragdirtok.rs

+        let mut all_directives_found = false;
+        let directive = td.directive.borrow();
+
+        'directive_loop: while !all_directives_found {


The labels make it quite hard to read. Have you considered any alternatives?

mre · 2025-01-10T13:53:19Z

lychee-lib/src/types/text_fragment.rs

+pub(crate) const FRAGMENT_DIRECTIVE_DELIMITER: &str = ":~:";
+pub(crate) const TEXT_DIRECTIVE_DELIMITER: &str = "text=";
+
+pub(crate) const TEXT_DIRECTIVE_REGEX: &str = r"(?s)^text=(?:\s*(?P<prefix>[^,&-]*)-\s*[,$]?\s*)?(?:\s*(?P<start>[^-&,]*)\s*)(?:\s*,\s*(?P<end>[^,&-]*)\s*)?(?:\s*,\s*-(?P<suffix>[^,&-]*)\s*)?$";


Where does that regex come from? Did you write it yourself? If there's an "official" regex for those text fragments, we could perhaps add a link to the reference.

mre · 2025-01-10T13:54:17Z

lychee-lib/src/utils/url.rs

@@ -23,10 +26,61 @@ pub(crate) fn find_links(input: &str) -> impl Iterator<Item = linkify::Link> {
    LINK_FINDER.links(input)
 }

+/// Fragment Directive feature trait
+/// we will use the extension trait pattern to extend the Url to support Text Fragment feature


mre · 2025-01-10T13:55:12Z

lychee-lib/src/utils/url.rs

@@ -23,10 +26,61 @@ pub(crate) fn find_links(input: &str) -> impl Iterator<Item = linkify::Link> {
    LINK_FINDER.links(input)
 }

+/// Fragment Directive feature trait
+/// we will use the extension trait pattern to extend the Url to support Text Fragment feature


Suggested change

/// we will use the extension trait pattern to extend the Url to support Text Fragment feature

/// We will use the extension trait pattern to extend [`url::Url`] to support the text fragment feature

mre · 2025-01-10T13:55:55Z

lychee-lib/src/utils/url.rs

+/// Fragment Directive feature trait
+/// we will use the extension trait pattern to extend the Url to support Text Fragment feature
+pub(crate) trait UrlExt {
+    /// Checks if the url has a fragment and if the fragment is has the fragment directive delimiter embedded


Suggested change

/// Checks if the url has a fragment and if the fragment is has the fragment directive delimiter embedded

/// Checks if the url has a fragment and if the fragment has the fragment directive delimiter embedded

mre · 2025-01-10T13:56:27Z

lychee-lib/src/utils/url.rs

+}
+
+impl UrlExt for Url {
+    /// Returns whether the URL has fragment directive or not


Suggested change

/// Returns whether the URL has fragment directive or not

/// Checks whether the URL has fragment directive or not

mre

Left some comments. I like the overall structure. Good work so far!

thiru-appitap · 2025-01-12T21:45:25Z

Left some comments. I like the overall structure. Good work so far!

@mre
I'll take a look at each of the comments and work to address it - thank you!

thiru-appitap and others added 3 commits December 25, 2024 01:56

Fragment Directive feature commit

7e1e2e1

Merge branch 'lycheeverse:master' into text-fragment

e3e09d6

verbose message clean-up

1085f7c

HU90m mentioned this pull request Dec 25, 2024

feature request: check text fragments #1545

Open

thiru-appitap added 2 commits December 25, 2024 03:28

cargo fmt fixes commit

584f460

clippy warnings fixed for test modules

b6d5b5a

mre reviewed Jan 10, 2025

View reviewed changes

lychee-lib/src/utils/url.rs

);

match url {

Ok(url) => {

eprintln!(

Copy link

Member

mre Dec 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be an assertion

mre reviewed Jan 10, 2025

View reviewed changes

mre requested changes Jan 10, 2025

View reviewed changes

Merge branch 'lycheeverse:master' into text-fragment

4ebb039

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Text fragment feature (#1545) #1600

Add support for Text fragment feature (#1545) #1600

thiru-appitap commented Dec 25, 2024

thiru-appitap commented Dec 25, 2024

mre Dec 25, 2024

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre Jan 10, 2025

mre left a comment

thiru-appitap commented Jan 12, 2025


		use crate::types::TextDirective;

		const BLOCK_ELEMENTS: &[&str] = &[

	/// we will use the extension trait pattern to extend the Url to support Text Fragment feature
	/// We will use the extension trait pattern to extend [`url::Url`] to support the text fragment feature

	/// Checks if the url has a fragment and if the fragment is has the fragment directive delimiter embedded
	/// Checks if the url has a fragment and if the fragment has the fragment directive delimiter embedded

	/// Returns whether the URL has fragment directive or not
	/// Checks whether the URL has fragment directive or not

Add support for Text fragment feature (#1545) #1600

Are you sure you want to change the base?

Add support for Text fragment feature (#1545) #1600

Conversation

thiru-appitap commented Dec 25, 2024

thiru-appitap commented Dec 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mre left a comment

Choose a reason for hiding this comment

thiru-appitap commented Jan 12, 2025