Cannot see lxml parser for Beautiful Soup, even after pip install

[margaretkennedy]

Problem: Cannot see lxml parser for Beautiful Soup, even after pip install

If you have already imported the bs4 module before installing lxml, bs4 will not see lxml in the list of available parsers.

On the first import of bs4, the initialization of the module occurs, loading all existing parsers. This initialization does not re-occur after future pip installs.

To confirm what parsers bs4 is able to see, run:

import bs4
print(bs4.builder.builder_registry.builders)

Example output where lxml is NOT recognized:

[<class 'bs4.builder._htmlparser.HTMLParserTreeBuilder'>]

Example output where lxml IS recognized:

[<class 'bs4.builder._lxml.LXMLTreeBuilder'>, <class 'bs4.builder._lxml.LXMLTreeBuilderForXML'>, <class 'bs4.builder._htmlparser.HTMLParserTreeBuilder'>]

[margaretkennedy]

This is a known issue.

Solution:

1. Restart Docker container:

docker-compose down
docker-compose up -d

2. Install both Python modules before running an import on bs4.

For example:

import os
os.system("pip install bs4 lxml")
import bs4
print(bs4.builder.builder_registry.builders)