How to extract the text from academic PDF

[rooooolin]

Hi, first of all, many thanks for this library.
I have encountered a problem when i extract the text from academic pdf.
Many academic PDFs have text arranged in the left and right columns as shown in the following figure. How can I know if a line is divided into left and right. Or How should I extract such a PDF.

[cmdlineluser]

Hi @rooooolin

I think the relevant search term is "columnar" or "columns"

There are a few previous discussions on the topic which may help if you haven't found them?

#890 (comment)

[rooooolin]

Thank you so much.

It helps me a lot

But I still want to know，is there a way to determine whether a line is divided or not? Actually, I am unsure whether the extracted PDF document contains such instances on a given page.

Thank you once again.

[cmdlineluser]

Are you talking about the lines under "Article Info" and "Abstract"?

Seems like it should be possible.

Is the PDF publically available? I can find https://www.mdpi.com/2504-446X/7/6/382/pdf but it is in a different format.

[rooooolin]

Yeah, the formatting of the lines under "Article Info" and "Abstract" Is different. For me , the key to solving this problem lies in how to automatically distinguish the formatting of the page or line.

For instance, when processing two PDFs simultaneously, such as https://arxiv.org/pdf/1512.03385.pdf and https://www.mdpi.com/2504-446X/7/6/382/pdf.

Pdfplumber needs to identify the lines (excluding the title) in the first document that have two columns, so that the lines or pages can be cropped into left and right sections for further processing.

left = page.crop(...）
right = page.crop(...）

In the case of the second PDF, Pdfplumber can determine that the lines have only one column, allowing it to extract the text directly using page.extract_text().

[cmdlineluser]

Oh right.

I haven't check through all the previous discussions, so it's possible I'm repeating a previous suggestion.

I think the basic approach for that type of thing would be to count the x0, x1 positions of each word and keep those above a certain count threshold.

Something like:

from collections import Counter

count = Counter()

for word in page.extract_words():
    count[round(word['x0'])] += 1
    count[round(word['x1'])] += 1
    
explicit_vertical_lines = [ 
   pos for pos, count in counter.most_common() if count > 10 
]

im = page.to_image(300)
im.draw_vlines(explicit_vertical_lines, stroke_width=5).save('vlines.png')

Although anything involving a hardcoded threshold value is open to false positives/negatives.

[rooooolin]

oh thanks for you help, this is what i needed