Repeat characters and unable to extract tables

[zmingxie]

Hi all,

I've been using this pdfplumber tool to extract data from the TREB monthly reports for personal use. It's been working great for the past reports (e.g. April 2022), but I'm having issues with its May 2022 report

After some diggings, it looks like pdfplumber is having problems extracting words and table data from page 3. Here is the code snippet from my troubleshooting:

import pdfplumber

pdf = pdfplumber.open("mw2205.pdf")

# The issue is with page[2] and onward and debug this page for now
p2 = pdf.pages[2]

# Draw word boxes
words = p2.extract_words(x_tolerance=0, y_tolerance=0)
im = p2.to_image()
im.reset().draw_rects(words)

# Debug table
im.reset().debug_tablefinder({})

• May report word racts:

• April report word racts:

• May report table:

• April report table:

I can't figure out what changed that is causing this extraction issue. Could anyone give me some hints? Thanks!

[jsvine]

Hi @zmingxie, and very interesting! I examined pdf.page[2].chars for the May report, it it appears that there are a bunch of invisible, very large A, b, and c characters, all with char["size"] == 33.84. That would explain the strange artifacts you're seeing when you run im.reset().draw_rects(words). You can verify this a few ways, including by inspecting words or .chars, or by opening the PDF and searching for "Abc". To achieve your goals, I'd do something like this:

p2_fixed = p2.filter(lambda obj: obj.get("size", 0) < 33.84)
im2_fixed = p2_fixed.to_image()
im2_fixed.reset().debug_tablefinder()

Does that give you the results you expect?

[jsvine]

Whoops, try < 33 instead of < 33.84; it seems I overlooked some other unusually large characters of the same kind. Doing that gets me this:

Is there an easy way to filter all invisible characters from a page?

Unfortunately not always (or at least not in a simple way), since there are different ways that characters can look hidden — depending on background color, what other objects are being drawn (imagine an opaque rectangle being drawn on top of a set of characters), et cetera.

[jsvine]

Note: This seems to handle the word-extraction issue, but May report still differs from the April report with respect to the table extraction, and that seems to stem from a large number of .rects objects in the new report. I'd recommend looking into the different properties of those rectangles, and seeing if you can devise a filter to select only those that you really want.

[zmingxie]

I just try this again with their July report(https://trreb.ca/files/market-stats/market-watch/mw2207.pdf), and I'm still experiencing the same issue :(

Out of curiosity, if I convert this page to an image first, would it help to eliminate those invisible characters?

[jsvine]

Hi @zmingxie, and sorry to hear that they keep changing the PDF structure. Converting to an image does not modify the PDF — it's just a convenience method to help visually debug what's in the page. I suggest looking at the page.chars values, and using the visually debugging to understand the attributes of the characters that are giving you trouble.

[zmingxie]

Alright, I will give that a try again. Thanks for all the advice and quick responses!