Extracting embedded JSON objects with rogue elements

[forthrin]

What problem are you trying to solve?

Does -- or could -- Nokogiri have a built-in way of extracting inline JSON objects in HTML, including handling of rogue elements that appear in the wild? (undefined and function are the only ones I've encountered at least.)

Fixing undefined is easy, but inline functions can be large and complex, and thus unfeasible to distinguish from the surrounding JSON with regular expressions. I haven't seen an out-of-the-box solution for this anywhere so far.

Please show your code!

doc='<script>foo={"a": 1, "b": undefined, "c": function(){}};</script><script>bar={"baz": 2};</script>'
Nokogiri.HTML5(doc).css('script').each do |item|
  next unless (match = item.text.match(/foo=(?<json>.*?);/))

  clean = match['json'].gsub(/\bundefined\b/, 'null').gsub(/function\(\)\{\}/, 'null')
  puts json = JSON.parse(clean)
end

Would be nice to be able to do someting like this do get the JSON object:

json = Nokogiri.HTML5(doc).css('script').json('foo')

Environment

 nokogiri:
      version: 1.14.1
      platform: arm64-darwin22
      description: ruby 3.2.1 (2023-02-08 revision 31819e82c8) [arm64-darwin22]

[flavorjones]

Hi, @forthrin, thanks for asking this question!

Nokogiri specifically wraps XML and HTML parsers, and so unfortunately has no capability to parse Javascript.

That said, you may want to look at something like https://github.com/nene/rkelly-remix which is a pure-Ruby javascript parser! You should be able to use it to examine the contents of a <script> tag:

#! /usr/bin/env ruby

require "bundler/inline"

gemfile do
  source "https://rubygems.org"
  gem "rkelly-remix"
  gem "nokogiri"
end

require "rkelly"

markup = '<script>foo={"a": 1, "b": undefined, "c": function(){}};</script><script>bar={"baz": 2};</script>'
html_doc = Nokogiri.HTML5(markup)

script = html_doc.at_css("script")
script.content # => "foo={\"a\": 1, \"b\": undefined, \"c\": function(){}};"

RKelly::Parser.new.parse(script.content).to_sexp.map do |k,v|
  pp [k,v]
end
# >> [:expression,
# >>  [:op_equal,
# >>   [:resolve, "foo"],
# >>   [:object,
# >>    [[:property, :"\"a\"", [:lit, 1]],
# >>     [:property, :"\"b\"", [:resolve, "undefined"]],
# >>     [:property, :"\"c\"", [:func_expr, "function", [], [:func_body, []]]]]]]]

Hope this helps?

[forthrin]

Thanks for your prompt and helpful reply. I agree that identifying inline (and semi-invalid) JSON however common, is ostensibly out of scope for an HTML parser. It looks like your suggested library does the job. However, two questions:

1. Is this a full-fledged JS parser? If so, this seems a bit supersized for the very specific job at hand (to simply identify and replace the inline function with null). Have you seen any lightweight alternatives to this anywhere? Shouldn't a dumb tokenizer that simply counts parantheses +/- until they are all zero followed by a semicolon work for most cases? (Ignoring idiotic cases with escaped parantheses, etc. which I would happily let fail and move on.)

2. A name change for the library seems due.

[stevecheckoway]

2. A name change for the library seems due.

Indeed! Looks like the project it was forked from changed the name years ago.