Multiple Tables of banded shaded rows with varying number of lines in row

[ramakrse]

Describe the bug

PDF has multiple tables across the documents. Tables are shaded/banded rows with varying lines in row

Code to reproduce the problem

Load the PDF file with pdfplumber

plumber_file = pdfplumber.open(pdf_file)
pdf_page = plumber_file.pages[29-1] #127 #67
im = pdf_page.to_image()

Table settings.

ts = {
"vertical_strategy": "lines",
"horizontal_strategy": "lines",
'intersection_tolerance': 32
}
im.debug_tablefinder(ts)

PDF file

Using the Public available pdf
https://www.mtu-solutions.com/content/dam/mtu/technical-information/operating-instructions/diesel/mtu-series-1600/marine/MS15029_01E.pdf/_jcr_content/renditions/original./MS15029_01E.pdf

Expected behavior

To identify the tables in each page properly. Here there are two tables

Actual behavior

playing with intersection_tolerance variable to handle more lines in a row, it detect one table, Space between tables also consider as row. Not able to detect two tables properly

Screenshots

Environment

• Python 3.10.11
• pdfplumber version: [e.g., 0.9.0]
  -Collab notebook

Additional context

Add any other context/notes about the problem here.

[jsvine]

Hi @ramakrse, and interesting example. I'd suggest something like the following:

• Use the properties of the page's .rects to identify the table headers (which seem to be a different color than the other rects) and the gray rects
• Use that information to determine the extents of the tables (i.e., where each table begins and ends)
• Passing that information to page.crop(...) twice, so that each crop focuses just on one of the tables
• then using cropped_page.extract_table(...)

[ramakrse]

Thanks @jsvine for your suggestion.
I spend some time on the suggested methods. However .rects doe not give a clue to identify header information from color. As I am new to this library, maybe, I am not getting there. Request help on the same.

Also, .extract_table will be adjusted based on the number of lines in the cell, It is not possible to hard code ts
settings

[jsvine]

However .rects doe not give a clue to identify header information from color. As I am new to this library, maybe, I am not getting there. Request help on the same.

I think the non_stroking_color property of each rect should help. It reflects the filling color for each rect. For the headers, the value is (0.5, 0.25, 0, 0.1), representing a light blue color in CMYK, and the height is 15.9685 units.

It turns out that there's another set of rects with the same color, but just 1 unit high, marking the end of those tables, which may also be a helpful piece of information in constructing your program logic.

[ramakrse]

Thanks for your input. I have progressed with your suggestion.

1. pdf_page.rects has duplicate information. each rectangle is listed twice. It may be fixed.
2. filtered by non_stroking_color value of (0.5, 0.25, 0, 0.1) and processed to get bounding box of table
3. Used simple Table setting as both horizontal and vertical strategy as lines. It gives table 1 only one table with one-row header only and table 2, it gives 4 tables, with varying columns.

I am looking for help, how do we handle varying number of lines of text in row. As I need to process multiple pages of the document, for each page setting ts specific will be painful.

below here code base on debug visualization

# Load the PDF file with pdfplumber
plumber_file = pdfplumber.open(pdf_file)
pdf_page = plumber_file.pages[29-1] 
rect_info = pdf_page.rects
rectangles_bbox = []
rectangles_info=[]
table_header_color=(0.5, 0.25, 0, 0.1)
for idx, rect in enumerate(rect_info):
    print("index: {} Bbox: {}".format(idx, rect))
    #Check whether it is headers and footer of table
    if rect['non_stroking_color'] == table_header_color: 
        #Get the bounding box of rectangle
        temp_rect_bb = (rect['x0'],rect['top'],rect['x1'],rect['bottom']) 
        #Check rect is duplicated in .rects
        if rect not in rectangles_info:
            rectangles_info.append(rect)
        if temp_rect_bb not in rectangles_bbox: #Check if boundingbox alread there
            rectangles_bbox.append(temp_rect_bb)
print('Rectangle Info Count: {}'.format(len(rect_info)))        
print('Unique Rectangle Info Count: {}\n'.format(len(rectangles_info)))

#Get the table bounding box 
table_reactangle_bbox = []
n_idx = len(rectangles_bbox)
if (n_idx%2) == 0: #Check whether it is even - header and footer line
    for idx in range(0, n_idx, 2):
        temp_rect = (rectangles_bbox[idx][0],rectangles_bbox[idx][1],rectangles_bbox[idx+1][2],rectangles_bbox[idx+1][3])
        table_reactangle_bbox.append(temp_rect)
else :
    raise Exception('No of identified {} rectangles is not even'.format(n_idx))

print('Table Bounding Box')
for idx, rect in enumerate(table_reactangle_bbox):
    print('{}: Table Bounding Box: {}'.format(idx,rect))

ts = {
    "vertical_strategy": "lines",
    "horizontal_strategy": "lines",
}
table_idx = 0 #1
pdf_page_filtered = pdf_page.crop(table_reactangle_bbox[table_idx])
tables = pdf_page_filtered.extract_table(ts)
if (len(tables) >0):
    print('Extracted Table Information')
    for idx, table in enumerate(tables):
        print('Table: {} and {}'.format(idx, table))


im = pdf_page_filtered.to_image()
im.debug_tablefinder(ts)

Output #1 - Table 1
Rectangle Info Count: 16
Unique Rectangle Info Count: 4

Table Bounding Box
0: Table Bounding Box: (121.88976, 211.12442, 561.25985, 443.71497999999997)
1: Table Bounding Box: (121.88976, 455.05358, 561.25985, 727.89611)
Extracted Table Information
Table: 0 and ['Abbr.', 'Meaning']

Output -Table 2

[jsvine]

Hmmm, I'm not sure I understand the question about "varying number of lines of text in row". What specific problems is the varying number of lines of text in row causing?

[ramakrse]

Table#1, most of row has one line height, except row 3,5 which has 2 lines height.

Table#2 header has single line height, rest has 5 lines heights.

These kinds of variation across pages. What should ts settings that deals with these variations

[jsvine]

Thanks. I don't believe that should pose a problem, nor require any change to the table settings. But if there are specific problems you see with the extraction related to the difference in rows heights, please do provide those details here.