Can you please add linear to write() - IMPLEMENTED

[jloehel]

PyMuPDF/fitz/fitz.i

Lines 1685 to 1688 in b0babae

	write(int garbage=0, int clean=0,
	int deflate=0, int deflate_images=0, int deflate_fonts=0,
	int ascii=0, int expand=0, int pretty=0, int encryption=1,
	int permissions=-1, char *owner_pw=NULL, char *user_pw=NULL)

I am not sure if this is on purpose but it would be nice to write also linearized files. :-)

[zufj]

Hey,

Apparently it is a limitation from the underlying mupdf library, see #674. So this change won't come soon.

We used temporary files to deal with it and the performance was fine on our setup.

Cheers

[JorjMcKie]

Apparently it is a limitation from the underlying mupdf library

Linearization as implemented by MuPDF requires the output device to support the seek operation, which does not work with memory buffers.

After this had been detected by #674 I disabled the linear option for bytes output (which I will rename to doc.tobytes btw.).
I can't image this will change ever.

I never bothered to try alternatives like io.BytesIO: A considerable amount of coding would be required to present this as a file to MuPDF (out of the box, they only support filename-based files or memory as output devices), and then it is unclear whether seek operations would go through in this case ...

So it looks like temporary files are the way to go for the time being.

[jloehel]

Do you know why the buffer_seek function has never been implemented? If save would support file-like objects like BytesIO it would be fine for me too. But to trick fopen(), you need to write the bytes anyway to disk. Nothing changed! I will focus on the buffer_seek function.

[JorjMcKie]

I remember I did use MuPDF's option to user-define an own output device. Requires providing function slots for all the disk operations - seek being among them.
If using a bytearray as the storage device, I image one can mimic everything that may come up ...
If that in fact works, we would be there ...

[jloehel]

I remember I did use MuPDF's option to user-define an own output device. Requires providing function slots for all the disk operations - seek being among them.
If using a bytearray as the storage device, I image one can mimic everything that may come up ...
If that in fact works, we would be there ...

Sounds interesting. Is there any documentation about such user-defined output devices?

[JorjMcKie]

Sounds interesting. Is there any documentation about such user-defined output devices?

Wenn du das so nennen willst: ja - wofür hat man Source Code!

mupdf-1.18.0/include/mupdf/fitz/output.h - critical line: 84
and
mupdf-1.18.0/source/fitz/output.c, line 230: function buffer_seek throws the error.

So, seriously, if one more or less duplicates this code and knows what needs to be done instead of buffer_seek ...

[jloehel]

I understand, thanks for your explanation. I will play with it.

[JorjMcKie]

Let me convert this thread to a discussion ... will be more adequate.

[JorjMcKie]

Thanks to everyone for participating in this ... and getting me ignited!
I just developed a way to output a PDF to a Python file pointer.
This does include BytesIO objects and hence a way "writing" the linear PDF format to memory.
I will include this in the next version (1.18.7).

[zufj]

Great news! Happy to test it as soon as it is out. Thank you!

[jloehel]

Awesome! 💯

[JorjMcKie]

Thank you all for the feedback. Wheel generation is underway. For those of you using OSX or some Linux and who would like to test it, may download their version from the resp. branch of this repo.
Should have the results in half an hour or so.
If someone needs a Windows version, I would have to upload it to this thread.

The change works like this:

1. doc.write has no API changes, except it now allows the incremental parameter.
2. doc.save now accepts as first positional parm:
   • a string like before
   • a pathlib.Path like before
   • a Python file pointer, either created via out = open("file.pdf", "wb") or via out = io.BytesIO(). The last option is used by doc.write, which returns out.getvalue() after doing a doc.save(out, ...).

[jloehel]

Hi @JorjMcKie Finally I have tried your new implementation it works fine but the PDF looks broken afterwards. Only the first page is visible. If I open it with evince:

>>> A lot of CRITICAL errors here <<<
...
(evince:221338): Poppler-CRITICAL **: 14:43:02.526: void poppler_page_get_size(PopplerPage*, double*, double*): assertion 'POPPLER_IS_PAGE (page)' failed

(evince:221338): Poppler-CRITICAL **: 14:43:02.526: void poppler_page_get_size(PopplerPage*, double*, double*): assertion 'POPPLER_IS_PAGE (page)' failed
[1]    221338 abort (core dumped)  evince foo.pdf

[JorjMcKie]

you could try and test acrobat with it
it wouldn't be the first poppler error I see

[jloehel]

you could try and test acrobat with it
it wouldn't be the first poppler error I see

Sorry, I can't test it very easily with the Adobe Acrobat Reader. The linux support have ended after 9.x. and the old build for 9.5.5 does not run anymore. Maybe I get it working with wine. I have tested the PDF with some other readers:

mupdf: no issues
evince (poppler): coredump
atril (poppler): only first page is correct. rest is broken. shows a lot of critical errors, but no coredump
oowriter: shows only the first page
qpdfview: issues to parse the xref section

"Error: Failed to parse XRef entry [1]."
"Error: Kid object (page 2) is wrong type (none)"
"Error: Failed to parse XRef entry [5]."
"Error: Kid object (page 2) is wrong type (none)

gv: no issues

I guess it's a mupdf lib issue.

[jloehel]

Short reproducer:

from io import BytesIO
import fitz

INPUT_FILE = "test.pdf"
MATRIX = fitz.Matrix(2, 2)
QUALITY = 70


def page2jpg(idx):
    with fitz.open(INPUT_FILE) as document:
        pixmap = document[idx].getPixmap(alpha=False, matrix=MATRIX)
        image = BytesIO(
            pixmap.pillowData(
                format="JPEG", optimize=True, quality=QUALITY, progressive=True
            )
        )
        return document[idx].rect, image


def main():
    with fitz.open(INPUT_FILE) as document_in, fitz.open() as document_out:
        for idx in range(len(document_in)):
            rect, image = page2jpg(idx)
            new_page = document_out.newPage(idx)
            new_page.setMediaBox(rect)
            new_page.insertImage(rect, stream=image)
        document_out.save("test_out.pdf", deflate=1, garbage=4, linear=True)


if __name__ == '__main__':
    main()

[JorjMcKie]

I have tried to reproduce with a somewhat simplified snippet:

>>> import fitz
>>> doc=fitz.open("v110-changes.pdf")
>>> out=fitz.open()
>>> for page in doc:
	np = out.new_page(width=page.rect.width,height=page.rect.height)
	jpg = page.get_pixmap().pil_tobytes("JPEG")
	np.insert_image(page.rect, stream=jpg)

	
5
9
13
17
>>> out.ez_save("x.pdf",linear=True)
>>> 

I am on Windows, but I am using WSL. So I tried the resulting x.pdf with the following PDF viewers:

• Adobe Acrobat: success
• PDF XChange Viewer: success
• Foxit Reader: success
• Nitro PDF: success
• Sumatra PDF: success
• MuPDF: success
• Linux evince: only displays 1st page, but admits that there are 4

Then I also usd pdftopng x.pdf img, a utility from XPDF, which is the library underneath poppler (or used to be): successfully converted the 4 pages to PNG images.

So I guess the bottom line is, that the error cannot be reproduced with a lot of PDF readers. Among them the standard Adobe.
Some notes:

• used the new "snake_case" method names
• used new method ez_save ("easy save") which does garbage=3, deflate=True as default.

Everything remains intact if I replace the last snippet line with

>>> pdfbytes=out.tobytes(garbage=3,deflate=True,linear=True)
>>> output=open("x.pdf","wb")
>>> output.write(pdfbytes)
212716
>>> output.close()
>>>

[jloehel]

Thank you very much for your effort. It looks really more like a weird evince/poppler issue. The most user will open the documents anyway in a Browser.