PyMuPDF collapsing whitespace text in some spans (no flags)

[GaryGen]

Description of the bug

I am encountering cases where the spans in some of text of some PDFs is collapsing without any whitespace. While less than 1% of the spans are affected, it is very noticeable where it is happening.

In Googling this issue, I see that this it typically related to people specifying TEXT_INHIBIT_SPACES. However, I am not specifying any flags. I did try adding TEXT_PRESERVE_LIGATURES, TEXT_PRESERVE_WHITESPACE, and TEXT_PRESERVE_SPANS to the get_text call, but none of these had any affect.

How to reproduce the bug

Shared Colab notebook

!pip install PyMuPDF
import fitz #PyMuPDF
import requests

def download_file(url, filename):
  """Downloads a file from a given URL and saves it locally."""
  try:
    response = requests.get(url, stream=True)
    response.raise_for_status()  # Raise an exception for bad status codes

    with open(filename, 'wb') as file:
      for chunk in response.iter_content(chunk_size=8192):
        file.write(chunk)

    print(f"File downloaded successfully as '{filename}'.")

  except requests.exceptions.RequestException as e:
    print(f"Error downloading file: {e}")

url = "https://ia904601.us.archive.org/24/items/asme-y-14.5-2018-dimensioning-and-tolerancing/ASME-Y14.5-2018-Dimensioning-and-Tolerancing.pdf"

filename = "sample.pdf"
download_file(url, filename)

pdf_doc = fitz.open(filename)
pdf_page = pdf_doc.load_page(15)
page_dict = pdf_page.get_text('dict')
bad_block = page_dict['blocks'][1]
bad_line = bad_block['lines'][4]
bad_span = bad_line['spans'][0]
print(bad_span['text'])

formerlyinSection1hasbeenreorganizedinto

PyMuPDF version

1.24.13

Operating system

Linux

Python version

3.10

[JorjMcKie]

This is a typical "Discussions" item.
Transferring ...

[JorjMcKie]

This happens when the PDF creator stored the text with insufficient gaps between words.
Please accept that ultimately no words or lines or whatever higher level structures are stored in a PDF, but only single characters. Often enough, spaces are not stored at all. So we need logic to synthesize spaces where we think this makes sense.
MuPDF (base library) uses a number of criteria to judge whether it should generate a space between "formerly" and "in". Part of these criteria are the font size, the character properties in the used font, and more.

Here, the distance between the right border of "y" in formerly and the left border in in is 1.29216 points. This is much less than the width of character y, 4.8384 and even less than 50% of the width of character "i" in in, 2.6688.
Therefore, MuPDF concluded that this is not enough to create a synthetic space at this point.