How do I fine-tune the extraction of the table from this PDF?

[jakobdo]

Hello again,
yesterday I got some interesting information from @samkit-jain , thanks again.

But I am still not sure how to fine-tune the table settings, when I don't get the "table" structure I want.

I have this PDF: https://cjp-rbi-icis.s3.eu-west-1.amazonaws.com/wp-content/uploads/sites/7/2021/03/18154706/2021-ICIS-Publishing_v5.pdf

I want the information from page 3 (as an example)

If I use the "default" settings, only the "header": When we publish is selected as a table.
If I use:
vertical_strategy: text
horizontal_strategy: lines
keep_blank_chars: True
I get this result:

This is very close the the result I want, I "just" need the last row and I want the Sector and Region to be splitted into 2 columns.

If I use:
vertical_strategy: text
horizontal_strategy: text
It looks like the last row is included, but then I get to many columns in the result.

[samkit-jain]

Hi @jakobdo For the first case where you are using the extraction strategy as

{
    "vertical_strategy": "text",
    "horizontal_strategy": "lines",
    "keep_blank_chars": True
}

but are failing to get the last row, you may use the intersection_y_tolerance as

{
    "vertical_strategy": "text",
    "horizontal_strategy": "lines",
    "keep_blank_chars": True,
    "intersection_y_tolerance": 5
}

It will give you the result as

You may find all the available table extraction settings at https://github.com/jsvine/pdfplumber#table-extraction-settings.

Unfortunately, I have not been able to come up with a table extraction strategy that works for this table without using the explicit_vertical_lines option. Text-based vertical separators are a challenging problem as it is very difficult to make them work for all the tables. For using the explicit vertical lines setting, I would suggest the following approach as I believe it can be extended to all the tables in the PDF you shared: From the first row of the table (the header row), get the leftmost X-axis coordinates of the column names and then use those as your vertical separators and run the table extraction. This will work because all the columns are left-aligned. Though, you would need to add a mechanism for ensuring that SCHEDULED PUBLICATION on the 3rd page is read as a single word otherwise you'll get 2 vertical separators instead of 1.