HTML Diff: Where possible, diff regions of the page independently

[Mr0grog]

I thought I’d written this idea down somewhere before, but cannot find it.

It might be nice to have the HTML differ use some heuristics to identify major regions of a page (e.g. main content [distinguised by tag, class name, etc.], headers, footers) and, if it can find the same region on both sides of the diff, diff each of those regions (and the regions between them) independently. This could help ensure, for example, that diffs in menus don't bleed together with diffs of the body content.

Having these heuristics around would also undoubtedly be useful in auto-classifying changes (e.g. changes only involved menus).

[Mr0grog]

Side note: it’s also possible this would help clean up diffs like a recent change to EPA’s page layout, where menus moved from after the body content to before it—depending on the particulars of the body content, we currently might identify the whole body as being changed or instead identify the whole menu as being changed. I imagine this technique of splitting up the diffing would ensure that this would always show menu as being the part that changed.

Example: https://monitoring.envirodatagov.org/page/2a2cd62e-ded7-4ecc-9749-804ea3e06a0d/9bfd1b57-5872-467f-b556-34cee538493a..0ef9613e-d38d-4aa2-97e7-01f39acf6f17

[Mr0grog]

Definitely still an idea worth working on.

[Mr0grog]

This Microsoft research paper covers an interesting way of using layout information to visually segment a page: https://www.microsoft.com/en-us/research/publication/vips-a-vision-based-page-segmentation-algorithm/

Some things about it might be tough to incorporate easily, though:

• It needs layout information, so we’d need to use headless Chrome or something similar,
• It can construct visual blocks from non-contiguous sections of the DOM, so reconstructing a page from it might be hard. It could be a visual layer on top, or maybe a “map” for another, simpler differ to do several diffs of subsections of the page based on it.
• It would almost certainly need a lot of updating for the web of today, as opposed to the web of 2003. (New tags, inline SVG, etc.)

[Mr0grog]

There is also some useful existing work based on real-world data from government sites at https://github.com/edgi-govdata-archiving/web-monitoring-task-sheets/blob/main/analyst_sheets/normalize.py

Not nearly as generic as the MS paper referenced earlier, though.