while extracting japanese text from pdf to text file, it is showing unwanted noise with japanese text.

[rohit-humne]

This is my code:-

import pymupdf

doc = pymupdf.open("output_S19990401A10010001001.pdf") # open a document
out = open("output5_S19990401A10010001001.txt", "wb") # create a text output
for page in doc: # iterate the document pages
    text = page.get_text().encode("utf16") # get plain text (is in UTF-8)
    out.write(text) # write text of page
    out.write(bytes((12,))) # write page delimiter (form feed 0x0C)
out.close()

output:-

i hope that i can find a solution.

Thanks in advance!

[JorjMcKie]

You did not share the file.
But anyway, did you check whether this also happens without your encoding change?

[rohit-humne]

@JorjMcKie
Thanks for your reply!

this is a pdf file
S19990401A10010001001.pdf

when i tried with other code instead of red marks as you can seen in ouput image, it is showing (cid:2) kind of characters.

[rohit-humne]

@JorjMcKie
Thank you!

if in case, i want to ignore this glyph characters, how can i do it?

[JorjMcKie]

Use a text extraction flags that does not try to find replacements for the so-called Invalid Unicode 0xFFFD (�), e.g. flags = pymupdf.TEXTFLAGS_TEXT & ~pymupdf.TEXT_CID_FOR_UNKNOWN_UNICODE.
Then text = page.get_text(flags=flags).
Finally text = text.replace(chr(0xfffd), "").

[rohit-humne]

Thank you! @JorjMcKie,

i am very new to programming. I have just manage to extract data from pdf into text file and then remove cid:x characters from the text file and save it into another textfile.

This is the code:-

from pdfminer.converter import PDFPageAggregator
from pdfminer.layout import LAParams, LTContainer, LTTextBox, LTTextLine, LTChar
from pdfminer.pdfinterp import PDFPageInterpreter, PDFResourceManager
from pdfminer.pdfpage import PDFPage
import re

def pdfminer_config(line_overlap, word_margin, char_margin,line_margin, detect_vertical):
    laparams = LAParams(line_overlap=line_overlap,
                        word_margin=word_margin,
                        char_margin=char_margin,
                        line_margin=line_margin,
                        detect_vertical=detect_vertical)
    resource_manager = PDFResourceManager()
    device = PDFPageAggregator(resource_manager, laparams=laparams)
    interpreter = PDFPageInterpreter(resource_manager, device)
    return (interpreter, device)

def find_textboxes(layout_obj):
    if isinstance(layout_obj, LTTextBox):
        return [layout_obj]
    if isinstance(layout_obj, LTContainer):
        boxes = []
        for child in layout_obj:
            boxes.extend(find_textboxes(child))
        return boxes
    return []

def find_textlines(layout_obj):
    if isinstance(layout_obj, LTTextLine):
        return [layout_obj]
    if isinstance(layout_obj, LTTextBox):
        lines = []
        for child in layout_obj:
            lines.extend(find_textlines(child))
        return lines
    return []

def find_characters(layout_obj):
    if isinstance(layout_obj, LTChar):
        return [layout_obj]
    if isinstance(layout_obj, LTTextLine):
        characters = []
        for child in layout_obj:
            characters.extend(find_characters(child))
        return characters
    return []

def write_text(text_file, text):
    text_file.write(text)

text_file = open('output11.txt', 'w')
with open("./S19990401A100100010011.pdf", 'rb') as f:
    interpreter, device = pdfminer_config(line_overlap=0.5, word_margin=0.1, char_margin=2, line_margin=0.5, detect_vertical=True)
    for page in PDFPage.get_pages(f):
        interpreter.process_page(page)  # ページを処理する。
        layout = device.get_result()  # LTPageオブジェクトを取得。
        boxes = find_textboxes(layout)
        for box in boxes:
            write_text(text_file, box.get_text().strip())
text_str = '(cid:2)'
if 'cid' in text_str.lower():
	text_str = text_str.strip('(')
	text_str = text_str.strip(')')
	ascii_num = text_str.split(':')[-1]
	ascii_num = int(ascii_num)
	text_val = chr(ascii_num)  # 66 = 'B' in ascii            
        
text_file.close()

with open("output11.txt", "r") as infile, open("output11_1.txt", "w") as outfile:
    content = infile.read()
    cleaned_content = re.sub(r"\(cid:\w+\)", "", content)
    outfile.write(cleaned_content)
re.sub(r"\(cid:\w+\)", "", content) #this will remove that pattern

output11_1.txt
output11.txt

if you see output11_1.text file still some noise is present....
as well as i want to extract text from this kind of pdfs from the folder and save into the same folder as same name, how can i imporve this code?

[JorjMcKie]

This script does not use PyMuPDF but pdfminer to access PDFs.
We do not give advice for this package.

[rohit-humne]

Thank you! @JorjMcKie