Help parsing semi-structured data with multiple indentations #1000
Replies: 3 comments 4 replies
-
Hi @saveyak Appreciate your interest in the library. Regex is a good idea. But, you could also do a mix of both regex and tables.
|
Beta Was this translation helpful? Give feedback.
-
It's also a bit tricky as individual records can span multiple pages e.g. "The blaze engulfs" starts on page 2 and continues to page 3. There may be some useful ideas here: Merge all pages into a 1-page PDF documentAn approach I've found helpful is to combine everything into a single page pdf, the simplest way I found was to use import pypdf
pdf = pypdf.PdfReader('Downloads/Atlanta_Public_Schools_GA_sample.pdf')
width = max(page.mediabox.right for page in pdf.pages)
height = sum(page.mediabox.top for page in pdf.pages)
blank = pypdf.PageObject.create_blank_page(None, width, height)
blank.merge_page(pdf.pages[-1])
bottom = pdf.pages[-1].mediabox.bottom
height = pdf.pages[-1].mediabox.height
for page in reversed(pdf.pages[:-1]):
blank.merge_translated_page(page, bottom, height)
bottom = page.mediabox.bottom
height += page.mediabox.height
out = pypdf.PdfWriter()
out.add_page(blank)
out.write('page.pdf')
|
Beta Was this translation helpful? Give feedback.
-
Update on this: Jeremy Singer-Vine very kindly reached out to me and helped with this problem. We were able to find a solution that uses this strategy: More detail: First, find the subheader (the part that says the date and number of books removed.) Find the range of vertical positions that defines where that section sits on the page.
Next, crop the page to that section.
Next, within that section, find the vertical positions of each individual book.
Within each individual book, identify the title, the middle section, and the bottom section.
Finally, put all the functions together.
The only issue remaining is that about 6% of books listed get cut off because they're at the bottom of the page, so as a result some data is missing. But, since we at least still get the title, author and date removed, this isn't a big deal to me. I very much appreciate everyone's help with this problem! |
Beta Was this translation helpful? Give feedback.
-
I'm parsing a PDF that shows books removed or lost from various school libraries in a public school district. (See sample pages here.) The format is tricky to parse because there are multiple indentations, and also headers that cover multiple books:
My current strategy mostly relies on regular expressions. I convert everything to text, then I plan to use regex to find the author, title, etc. I have gotten this far so far:
Before I spend a lot of time crafting regexes, I want to know if there is an alternative way to do this. I used debug_tablefinder just to see what it would look like, and I did notice that the program was able to recognize rectangles around at least some of the variables I want to grab. Is there a way to get it to recognize what essentially tables within tables within tables?
If it seems like regex will be my best best, then I'd like tips on how to grab the site from which the book was removed and the weeded/lost status. Both seem much harder to grab than the author or the title because the site can spill over multiple lines, and then weeded/lost status doesn't follow any particular keyword. (Right now every status says the same thing but in the larger PDF, some of them say stuff like "Was lost by ____ on ____" or "Was deleted." I do think they all start with "Was" but I'm not sure.)
Beta Was this translation helpful? Give feedback.
All reactions