to_image function is slow

[chopin1998]

as you can see, for resolution=300, render a page need ~700ms on intel 13700K.

a normal pdf reader(qpdfview) ls fast more

[cmdlineluser]

It looks like wand.image (which uses ghostscript?) is what does the conversion (because PIL cannot read PDF files?):

https://github.com/jsvine/pdfplumber/blob/stable/pdfplumber/display.py#L56-L83

I have been using pypdfium2 which has a .to_pil() method:

https://pypdfium2.readthedocs.io/en/stable/readme.html#support-model

It seems to be around 3-4 times faster in single page parsing but also parses multiple pages concurrently.

Basic comparison of the first 3 pages from a "complex" pdf:

with codetiming.Timer(): 
     pdf.pages[0].to_image(resolution=300)
     pdf.pages[1].to_image(resolution=300)
     pdf.pages[2].to_image(resolution=300)

# Elapsed time: 24.9851 seconds

with codetiming.Timer():
    images = list(pdf.render(
        pdfium.PdfBitmap.to_pil,
        page_indices = [0, 1, 2],
        scale = 300/72,  # 300dpi resolution
    ))

# Elapsed time: 4.1711 seconds

This appears to be the only place wand is used in pdfplumber so it seems like it could be swapped out easily.

[jsvine]

Thanks for this helpful comparison, @cmdlineluser! I'll take a closer look at pypdfium2 to evaluate the feasibility/desirability of swapping out Wand in favor of it. Some things I'll be looking for (these are mostly notes-to-self, but you're also welcome to chime in):

• How much of a hassle (if at all) is installing pypdfium2?
• Does installing it automatically also download the binaries it requires?
• What is the memory/time overhead of instantiating pypdfium2 on a PDF? (I.e., the prep before you can call the .render(...) method.)
• How does the quality of pypdfium2's image output compare to Wand/ghostscript's?

[jsvine]

Thanks again, @cmdlineluser. Your comment motivated me to fully swap out Wand for pypdfium2 in v0.10.0, now available. As noted in b049373, it seems like a improvement on several fronts (quality, speed, installation).

[cmdlineluser]

Glad it was of interest @jsvine - I've just upgraded to test it out.

Is there a way to pass no_smoothtext=False ?

https://github.com/jsvine/pdfplumber/blob/stable/pdfplumber/display.py#L57

I see it's listed under Modifiable but I can't seem to set it from .to_image()

Just comparing to the last images I created with 0.9.0 - the only difference I could notice was the text "choppiness".

no_smoothtext=True

no_smoothtext=False

The loop to create the images has gone from 11s to 0.9s which is pretty nice!

[jsvine]

Thanks! That looks like a mistake on my part, and I can look into fixing this. Are you able to share the PDF, so that I can test the results?

[cmdlineluser]

Oops - sorry.

It was from another user in a previous discussion: #934

https://github.com/jsvine/pdfplumber/files/12026548/Dummy.pdf

[jsvine]

Thanks! That's helpful. Here's how the relevant method would be rewritten, with the ability to pass a new antialias: bool parameter:

    img: PIL.Image.Image = pypdfium2.PdfDocument._process_page(
        # Modifiable arguments
        page_ix,
        input_data=src,
        password=password,
        scale=resolution / 72,
        no_smoothtext=not antialias,
        no_smoothpath=not antialias,
        no_smoothimage=not antialias,
        # Non-modifiable arguments
        renderer=pypdfium2._helpers.page.PdfPage.render,
        converter=pypdfium2.PdfBitmap.to_pil,
        prefer_bgrx=True,
        pass_info=False,
        need_formenv=False,
        mk_formconfig=None,
    )

I'm torn on whether to set the default antialias=False or antialias=True. On the one hand, Judging by the tests below (using a crop of the PDF you linked to), the file sizes with antialiasing are noticeably larger (presumably because antialiasing uses a larger range of colors, less amenable to PNG's compression):

Resolution	Quantize?	With AA (bytes)	Without AA (bytes)	With:Without
72	Y	3578	1986	1.8x
72	N	10879	2038	5.3x
150	Y	8274	4062	2.0x
150	N	29549	5020	5.9x
300	Y	18150	9124	2.0x
300	N	57041	12289	4.6x

... based on:

import pdfplumber
from pathlib import Path

pdf = pdfplumber.open("Dummy.pdf")
page = pdf.pages[0]
cropped = page.crop((60, 30, 420, 220 ))

for aa in [ True, False ]:
    _aa = str(aa).lower()[0]
    for res in [ 72, 150, 300 ]:
        im = cropped.to_image(
            resolution=res,
            antialias=aa,
        )
        for q in [ True, False ]:
            _q = (str(q).lower())[0]
            dest = Path(f"results/{res:03d}-aa-{_aa}-q-{_q}.png")
            im.save(dest, quantize=q)
            size = dest.stat().st_size
            print(f"{dest.stem}: {size}")

On the other hand, the antialiased image quality is noticeably better:

72 DPI

Quantized

Without Antialiasing

With Antialiasing

Not Quantized

Without Antialiasing

With Antialiasing

150 DPI

Quantized

Without Antialiasing

With Antialiasing

Not Quantized

Without Antialiasing

With Antialiasing

300 DPI

Quantized

Without Antialiasing

With Antialiasing

Not Quantized

Without Antialiasing

With Antialiasing

---

My general inclination is to default to not antialiasing / smaller files, while making the antialias=True option clear in the documentation. One core motivation: The role of the visual tools in pdfplumber are more for strategizing than for production-quality rendering. But I'm open to other perspectives. Thoughts?

[mara004]

Hello,

Thanks for switching to pypdfium2 (I'm the maintainer). I found this repo through GH dependents network and took a quick look at commit b049373:
Please note that PdfDocument._process_page is private API and you should never use it directly. Use PdfPage.render instead.
You can retrieve pages from a pdf using the .get_page() method or index access. Please change to that.
(_process_page will either be renamed and receive breaking changes, or more likely disappear entirely with the next major release.)

Kind regards

[mara004]

_process_page may be removed since its caller, the parallel rendering method PdfDocument.render mentioned by @cmdlineluser, has issues.

We can't really provide an API in that form, since it's important that all post-processing (e.g. saving PIL images) also happens inside worker jobs, otherwise the main process handler becomes a bottleneck, and causes images to queue up in memory.
There are further concerns, like overhead from process initialization and data transfer.

I think it's best to outsource parallelization to callers, who should use it only after careful testing that it really provides an advantage in the specific env/scenario.

[jsvine]

Thanks for flagging @mara004, I'll change this!

[jsvine]

Thanks again. Now fixed in 28f4ebe, will be part of the next release. (Unless there's a reason to push the release urgently, @mara004?)

[mara004]

The change looks good, thanks!
Since you don't currently have upper version bounds for pypdfium2, it would be good to release this before the next version of pypdfium2 comes out (which will be API-breaking with major version increment).
I would estimate the code base to be ready in about 2 to 6 weeks, depending on how much we want to change and how much time I have, so not really urgent.

[jsvine]

Thanks again. Now released in v0.10.2. Really appreciate your heads up, and best of luck with your next release.