This package provides a python client for crawling chemical compounds and their properties on PubChem.
You can install the PubChem API Crawler directly with pip :
pip install pubchem-api-crawlerOr you can clone the project from github and install it locally using poetry with
poetry installExample notebooks showing how to use the library are available in the notebooks directory. To run the notebooks, run
poetry run jupyter laband select the notebook in the browser window.
The main entry point for PubChem API Crawler is the Molecular Formula Search function of Pubchem which lets you retrieve compounds given a molecular formula search input.
For example, if you wanted to find all compounds on PubChem containing carbon, hydrogen, aluminium and bore, you would use :
from pubchem_api_crawler import MolecularFormulaSearch
df = MolecularFormulaSearch().search(["C1-", "H1-", "B1-", "Al1-"], allow_other_elements=False, properties=["MolecularFormula", "CanonicalSMILES"])| CID | MolecularFormula | CanonicalSMILES | |
|---|---|---|---|
| 0 | 168084494 | CH5AlB2 | [BH].[BH].C[Al] |
| 1 | 163556649 | C16H14AlB | [B]CCC1=C2CCC=CC2=C(C3=CC=CC=C31)[Al] |
| 2 | 161576177 | C27H30AlB | [H+].[B-](C1=CC=CC=C1)(C2=CC=CC=C2)(C3=CC=CC=C3)C4=CC=CC=C4.C[Al](C)C |
| 3 | 160352291 | C6H15AlB | [B].CC[Al](CC)CC |
| 4 | 159123289 | C10H28AlB2 | [B](C)C.[B](C)C.CCCC.C[Al]C |
| 5 | 158802573 | C11H29AlB | B(C)(C)C.CCCC.CC[Al]CC |
| 6 | 158250967 | C3H9AlB | [B].C[Al](C)C |
| 7 | 158044531 | C2H6AlB | [B].C[Al]C |
| 8 | 157093180 | C3H9AlB | B(C)(C)C.[Al] |
| 9 | 156888304 | C12H14AlB | [B]C1=CC=CC=C1C2CCCCC2[Al] |
| 10 | 129859217 | C2H6AlB | [B].C[Al]C |
| 11 | 129657578 | C2H6AlB | [B-].C[Al+]C |
| 12 | 129657197 | CH3AlB2 | [B-].[B-].C[Al+2] |
| 13 | 59992955 | C7H9AlB | [BH2].C1=CC=C(C=C1)C[Al] |
| 14 | 22996618 | C12H30AlB | B(CC)(CC)CC.CC[Al](CC)CC |
| 15 | 19734271 | C8H18AlB | [B-].CC(C)C[Al+]CC(C)C |
| 16 | 155575130 | C8H8AlB | [B]C1=CC(=C(C=C1C)[Al])C |
The valid inputs for Molecular Formula Search are described here.
The general MF query syntax consists of a series of valid atomic symbols
(please consult your periodical chart), each optionally followed by either
a number or a range.
The generic range syntax is "[atomic symbol][low count]-[high count]",
repeated for every specified element. Elements may be written in
arbitrary order.
Examples:
1. C7-8: represents compounds with seven or eight carbons.
2. C-7: represents compounds with up to seven carbons.
3. C7-: represents compounds with seven or more carbons
4. C or C1: represents compounds with exactly one carbon
5. C-: represents any number of carbons, including none.
The search input must be provided as a list of [atomic symbol][low count]-[high count] strings to the search method.
Note: specifying an open ended high count (i.e. C2-) does not seem to work correctly on PubChem. It is recommended to always specify a high count (i.e. C2-500).
By default, the molecular formula search will return the cids of the matching compounds. Optionally, a list of properties can also be requested. The list of valid compound properties which can be requested is available here.
Aditionnaly, the allow_other_elements option lets you choose to allow other elements to be present in addition to those specified.
REST requests made to PubChem time out after 30s. Therefore, searches that are too broad will timeout on the server and raise an error. To overcome this limitation, it is possible to use PubChem's Async REST API. If your search request times out, you should retry it via the Async REST API with the _async=True parameter :
df = MolecularFormulaSearch().search(["C1-", "H1-"], allow_other_elements=False, properties=["MolecularFormula", "CanonicalSMILES"], _async=True)When using PubChem's REST API, you can only retrieve computed compound properties (list is available here).
If you want to retrieve experimental properties annotations, you can use the Annotations class of PubChem API Crawler. The list of annotation headings (and their types) for which PubChem has any data is available here.
PubChem API Crawler offers two ways to get annotations. You can get annotations for specific compounds individually by giving their cids. But there are no batch methods to fetch annotations on PubChem, so this requires sending a REST request per compound, which can be quite slow if you want to get properties for a lot of compounds. The alternative is to get all the data that PubChem has for a given annotation heading.
The get_compound_annotations method will get a specific annotation heading for the given cids (if heading is unspecified, it will get the Experimental Properties section).
from pubchem_api_crawler import Annotations
Annotations().get_compound_annotations(356, heading='Heat of Combustion')| Heat of Combustion Value |
Heat of Combustion Reference |
CID |
|
|---|---|---|---|
| 0 | 1,302.7 kg cal/g mol wt at 760 mm Hg and 20 °C | Weast, R.C. (ed.) Handbook of Chemistry and Physics. 69th ed. Boca Raton, FL: CRC Press Inc., 1988-1989., p. D-278 | 356 |
The get_annotations method will get all available data on PubChem for a given heading.
from pubchem_api_crawler import Annotations
Annotations().get_annotations("Autoignition Temperature")| SourceName | SourceID | URL | Value | Reference | CID | |
|---|---|---|---|---|---|---|
| 0 | Hazardous Substances Data Bank (HSDB) | 30 | https://... | 270 °C (518 °F) | Fire Protection Guide to Hazardous Materials. ... | 4510 |
| 1 | Hazardous Substances Data Bank (HSDB) | 35 | https://... | 928 °F (498 °C) | National Fire Protection Association; Fire Protection Guide ... | 241 |
| 2 | Hazardous Substances Data Bank (HSDB) | 37 | https://... | 871 °F (466 °C) | National Fire Protection Association; Fire Protection Guide ... | 2537 |
| 3 | Hazardous Substances Data Bank (HSDB) | 39 | https://... | 772 °F (411 °C) | Fire Protection Guide to Hazardous Materials. ... | 7835 |
| 4 | Hazardous Substances Data Bank (HSDB) | 40 | https://... | 867 °F (463 °C) | National Fire Protection Association; Fire Protection Guide ... | 176 |
You should first check the rate limits that PubChem imposes on requests to its API. On top of those dynamic request throttling policies, you should not send more than 5 requests per second to the PubChem REST API.
By default, PubChem API Crawler sets a rate limit of 5 calls per 3 seconds on REST API calls. These settings can be modified either by setting environment variables RATE_LIMIT_CALLS (integer) and RATE_LIMIT_PERIOD (integer, in seconds) or by creating a .env file in your working directory where those variables are set.
If you enable logging for the PubChem API Crawler namespace with log level set to DEBUG, the library will report request throttling status in the logs after each request.
Enable logging before calling the library's functions to see debugging and info messages.
import logging
logger = logging.getLogger('pubchem_api_crawler')
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
logger.addHandler(ch)