extract_table omits the last row that is incomplete

[chengtie]

I have the following pdf file, and I would like to extract the tables and rows. Note that the last row (numbered by 6) in the first page has not finished, the resting is on the second page. And I notice that the extract_table function omits the last row in the first page.

So is there way to be able to extract that incomplete row of the first page as well, so that I could combine it with the first row of the second page?

SAMPLE PDF.pdf

Here is my code:

def process_pdf(file_path, output_dir):
    # Extract file name (without extension) from the file path
    file_name_prefix = os.path.splitext(os.path.basename(file_path))[0]
    
    headers = []
    prev_row = None  # Variable to store the previous row
    with pdfplumber.open(file_path) as pdf:
        # Loop through all the pages in the PDF
        for page_number, page in enumerate(pdf.pages):
            # Extract the table from the current page
            table = page.extract_table()
            
            if not headers:  # If headers are not yet extracted
                for row in table:
                    if None not in row:  # Change this condition based on your criteria
                        headers = row
                        start_row = table.index(row) + 1
                        break
            
            # If headers are still not extracted (i.e., a valid header row wasn't found), skip this page
            if not headers:
                continue
            
            # Loop through the rows in the table starting from the appropriate row
            for row_number, row in enumerate(table[start_row:], start=start_row):
                print(row[0])
                print(row[1])
                print(row[2])
                if row[0] is None or row[0] == '':  # Check if the first cell is empty
                    if prev_row:  # If there is a previous row to concatenate with
                        row = [p if r is None else r for p, r in zip(prev_row, row)]  # Concatenate the rows
                else:
                    prev_row = row  # Save the current row as previous row
                
                process_row(row_number, row, headers, page_number, output_dir, file_name_prefix)

[cmdlineluser]

Hi @chengtie

As I understand it, the reason it is omitted is because there is no "bottom" line closing off the final row.

What I've done in cases like this is to use the .find_*() methods to return the table objects.

We can then take each horizontal line from the Row to build a list of explicit_horizontal_lines to pass to the .extract_*() method.

Using the position of the "lowest" character on the page is one approach for generating your own "bottom" line.

from operator import itemgetter

[...]

explicit_horizontal_lines = set().union(*((row.bbox[1], row.bbox[3]) for row in page.find_table().rows))
explicit_horizontal_lines.add(max(page.chars, key=itemgetter("bottom"))["bottom"])

table = page.extract_table(dict(
   explicit_horizontal_lines=explicit_horizontal_lines, 
   horizontal_strategy="explicit"
))

# Last row
print(table[-1])

['6',
 'GEN',
 '10/23/2023',
 '9/18/2023',
 'QUIT CLAIM\nDEED',
 'LASTNAME, FIRST',
 'LASTNAME, FIRST',
 'Section:34\nTownship:80\nRange:21 Qtr\nSection:SE Qtr\nQtr Section:SE',
 '2023-\n00005028',
 '',
 '2023-\n00004572',
 '',
 '3']

[chengtie]

Thank you for the help.

I tried your code, it returned an error: AttributeError: 'Page' object has no attribute 'find_table'.

Additionally, pip show pdfplumber returned Version: 0.10.2.

[cmdlineluser]

How odd. Do you have a .find_tables() method?

[chengtie]

I do have .find_tables(). explicit_horizontal_lines = set().union(*((row.bbox[1], row.bbox[3]) for row in (page.find_tables())[0].rows)) works.

I have another question now. My current code is as follows. It performs process_row over the last row of the first page which is incomplete, and then it concatenates that row and the first row of the second page. I don't want to perform process_row over the last incomplete row. How could I check if the last row of a page is incomplete or not? Is it possible to check this by using explicit_horizontal_lines (i.e., check if a final horizontal line is added or not)?

def process_pdf(file_path, output_dir):
    # Extract file name (without extension) from the file path
    file_name_prefix = os.path.splitext(os.path.basename(file_path))[0]
    
    headers = []
    prev_row = None  # Variable to store the previous row
    with pdfplumber.open(file_path) as pdf:
        # Loop through all the pages in the PDF
        for page_number, page in enumerate(pdf.pages):

            explicit_horizontal_lines = set().union(*((row.bbox[1], row.bbox[3]) for row in (page.find_tables())[0].rows))
            explicit_horizontal_lines.add(max(page.chars, key=operator.itemgetter("bottom"))["bottom"])

            table = page.extract_table(dict(
                explicit_horizontal_lines=explicit_horizontal_lines, 
                horizontal_strategy="explicit"
            ))

            # Extract the table from the current page
            # table = page.extract_table()
            print(table)
            
            if not headers:  # If headers are not yet extracted
                for row in table:
                    if None not in row:  # Change this condition based on your criteria
                        headers = row
                        start_row = table.index(row) + 1
                        break
            
            # If headers are still not extracted (i.e., a valid header row wasn't found), skip this page
            if not headers:
                continue

            if page_number != 0:
                start_row = 1

            # Loop through the rows in the table starting from the appropriate row
            for row_number, row in enumerate(table[start_row:], start=start_row):
                print(row[0])
                print(row[1])
                print(row[2])
                if row[0] is None or row[0] == '':  # Check if the first cell is empty
                    if prev_row:  # If there is a previous row to concatenate with
                        row = [p if not r else (r if not p else p + "\n" + r) for p, r in zip(prev_row, row)]  # Concatenate the rows
                else:
                    prev_row = row  # Save the current row as previous row
                
                process_row(row_number, row, headers, page_number, output_dir, file_name_prefix)

[cmdlineluser]

The method I've usually seen for this is to combine the records afterwards.

In this example, if the "row_id" column is empty - it is part of the previous row.

In "DataFrame" terms, you could "forward fill" that column allowing you to .groupby() and combine the columns.

import pandas as pd

[...]

tables = [pd.DataFrame(table) for table in tables]
tables[0] = tables[0].drop(0).reset_index(drop=True)

df = pd.concat((table.drop(0) for table in tables), ignore_index=True)
df.columns = ["row_id"] + list(tables[0].iloc[0, 1:])

>>> df
   row_id Index                              Date Filed Document\nDate                       Kind  .
0       1   GEN  10/23/2023\nDocument\nDate\n10/18/2023     10/18/2023             WARRANTY\nDEED  .
1       2   GEN  10/23/2023\nDocument\nDate\n10/20/2023     10/20/2023                   MORTGAGE  .
2       3   GEN  10/23/2023\nDocument\nDate\n10/20/2023     10/20/2023                   MORTGAGE  .
3       4   GEN  10/23/2023\nDocument\nDate\n10/18/2023     10/18/2023  SATISFACTION\nOF MORTGAGE  .
4       5   GEN  10/23/2023\nDocument\nDate\n10/18/2023     10/18/2023  SATISFACTION\nOF MORTGAGE  .
5       6   GEN                              10/23/2023      9/18/2023           QUIT CLAIM\nDEED  .
6                            Document\nDate\n09/18/2023                                            .
7       7   GEN  10/23/2023\nDocument\nDate\n10/20/2023     10/20/2023             WARRANTY\nDEED  .
8       8   GEN  10/23/2023\nDocument\nDate\n10/20/2023     10/20/2023             WARRANTY\nDEED  .
9       9   GEN  10/23/2023\nDocument\nDate\n10/12/2023     10/12/2023         POWER OF\nATTORNEY  .
10     10   GEN  10/23/2023\nDocument\nDate\n10/23/2023     10/23/2023                   MORTGAGE  .

[11 rows x 13 columns]

>>> df['row_id'].replace('', None).ffill()
0      1
1      2
2      3
3      4
4      5
5      6   # These rows now have the same id
6      6   # and can be merged
7      7
8      8
9      9
10    10
Name: row_id, dtype: object