Relative path problem when calling Pandoc Python filter in subfolders (MSYS2 on Windows) #9328

sdbbs · 2024-01-10T17:03:52Z

sdbbs
Jan 10, 2024

I have installed pandoc-3.1.11-windows-x86_64.msi (+ pandoc-crossref-Windows-v0.3.17.0.7z), on Windows 10, but I use pandoc almost exclusively through a MSYS2 (or MINGW64) bash shell; so pandoc reports:

$ which pandoc
/c/Program Files/Pandoc/pandoc

$ pandoc --version
pandoc.exe 3.1.11
Features: +server +lua
Scripting engine: Lua 5.4
User data directory: C:\Users\[USER]\AppData\Roaming\pandoc
...

The Python I have in this environment is:

$ python --version
Python 3.11.6

$ which python
/usr/bin/python

$ cygpath -w $(which python)
C:\msys64\usr\bin\python.exe

Now, I have a collection of Pandoc lua filter scripts, which I want to keep in a subfolder; I also have a my-pandoc.yaml, which is not necessarily in the same subfolder as the scripts; but I'd like to set up things, so if I copy the .yaml and the filter scripts folder elsewhere, they would still work. This works fine with lua filter scripts - however, something goes wrong when I want to use a Python filter script.

Here is a test setup:

$ cd /tmp/
$ readlink -f .
/tmp
$ cygpath -w $(readlink -f .)
C:\msys64\tmp

$ mkdir /tmp/pandoc_test
$ cd /tmp/pandoc_test/
$ mkdir -p scripts/pandoc

$ cat > scripts/pandoc/span-test.lua <<'EOF'
function Span (el)
  if not(el == nil) then
    print("Hey, lua got a span:", el)
  end
  return el
end
EOF

$ cat > scripts/pandoc/python-test.py <<'EOF'
#! /usr/bin/env python
import subprocess
import os
import sys
import shutil

def do_process(key, value, fmt, meta):
  if key == 'Span':
    print("Hey, python got a span", value)
    return Span(value)

if __name__ == "__main__":
    toJSONFilter(do_process)
EOF

$ mkdir -p docs/test_example

$ cat > docs/my-pandoc.yaml <<'EOF'
pdf-engine: xelatex
# NOTE https://github.com/jgm/pandoc/issues/6152 - resource path should be colon separated?
resource-path: [".:../scripts/pandoc:${.}/../scripts/pandoc"]
filters:
- ${.}/../scripts/pandoc/span-test.lua
#- ${.}/../scripts/pandoc/python-test.py
EOF

$ cat > docs/test_example/test_example.md <<'EOF'
---
title: "Example test - hello world"
author: Me
date: \today

---

# Just a test example

Here we have:
<span id="01">Hello</span>
<span id="02">World</span>

EOF

$ tree /tmp/pandoc_test/
/tmp/pandoc_test/
├── docs
│   ├── my-pandoc.yaml
│   └── test_example
│       └── test_example.md
└── scripts
    └── pandoc
        ├── python-test.py
        └── span-test.lua

I use ${.} because https://pandoc.org/MANUAL.html notes that is what I need:

${.} will resolve to the directory containing the defaults file itself. This allows you to refer to resources contained in that directory:

Note that, to begin with, the call to python script is commented. So now, if I do:

$ cd /tmp/pandoc_test/docs/test_example

$ pandoc --defaults=../my-pandoc.yaml --toc test_example.md -o test_example.pdf
Hey, got a span:        Span ("01",[],[]) [Str "Hello"]
Hey, got a span:        Span ("02",[],[]) [Str "World"]

$ du -hs *
1.0K    test_example.md
16K     test_example.pdf

So, without the Python script, building pdf using pandoc in this folder/path organization works fine.

So, let's remove the comment character from the my-pandoc.yaml file, so we end up line noted below, and then save the file:

...
- ${.}/../scripts/pandoc/python-test.py

... and try the build again:

$ pandoc --defaults=../my-pandoc.yaml --toc test_example.md -o test_example.pdf
Hey, got a span:        Span ("01",[],[]) [Str "Hello"]
Hey, got a span:        Span ("02",[],[]) [Str "World"]
python: can't open file '/tmp/pandoc_test/docs/test_example/C:\\msys64\\tmp\\pandoc_test\\docs/../scripts/pandoc/python-test.py': [Errno 2] No such file or directory
Error running filter C:\msys64\tmp\pandoc_test\docs/../scripts/pandoc/python-test.py:
Filter returned error status 2

So, somehow, from the ${.}/../scripts/pandoc/python-test.py in the .yaml file, python got the first argument - the script path - as /tmp/pandoc_test/docs/test_example/C:\\msys64\\tmp\\pandoc_test\\docs/../scripts/pandoc/python-test.py - so:

First the absolute path as per MSYS2/bash/Unix path: /tmp/pandoc_test/docs/test_example/,
Then to that, appended the absolute path as per Windows C:\\msys64\\tmp\\pandoc_test\\docs
And finally, to that, appended the script path from the .yaml file (after the ${.}), which is /../scripts/pandoc/python-test.py

Not surprising, then, that python reports error "can't open file ... No such file or directory".

I recall, in earlier versions of python (I think it was 2.13), I had tried entering an absolute ("mixed" mode, with Windows drives but forward slashes) path for the python script, as found via:

$ cygpath -m $(readlink -f ../../scripts/pandoc/python-test.py)
C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py

... in the .yaml:

...
- C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py

... but if I try it in the same example as above, I get again:

$ pandoc --defaults=../my-pandoc.yaml --toc test_example.md -o test_example.pdf
Hey, got a span:        Span ("01",[],[]) [Str "Hello"]
Hey, got a span:        Span ("02",[],[]) [Str "World"]
python: can't open file '/tmp/pandoc_test/docs/test_example/C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py': [Errno 2] No such file or directory
Error running filter C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py:
Filter returned error status 2

... so it does not work in the most recent version.

But even if it did, I would not want to use it, because it then breaks my entire approach of:

... I'd like to set up things, so if I copy the .yaml and the filter scripts folder elsewhere, they would still work.

Finally, if I just use the MSYS2 Unix path, that is:

...
- /c/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py

... in the .yaml file, then I get a different error:

$ pandoc --defaults=../my-pandoc.yaml --toc test_example.md -o test_example.pdf
Hey, got a span:        Span ("01",[],[]) [Str "Hello"]
Hey, got a span:        Span ("02",[],[]) [Str "World"]
Error running filter /c/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py:
Could not find executable /c/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py

... which apparently means that the .py script has finally been found, but pandoc does not recognize it as an executable -- even if MSYS2 actually presents all scripts (text files) that start with a shebang as executable:

$ ls -la /tmp/pandoc_test/scripts/pandoc/python-test.py
-rwxr-xr-x 1 [USER] None 259 Jan 10 17:20 /tmp/pandoc_test/scripts/pandoc/python-test.py*

# ... and calling from MSYS2 works:
$ /tmp/pandoc_test/scripts/pandoc/python-test.py
Traceback (most recent call last):
  File "/tmp/pandoc_test/scripts/pandoc/python-test.py", line 13, in <module>
    toJSONFilter(do_process)
    ^^^^^^^^^^^^
NameError: name 'toJSONFilter' is not defined

So - how can I have a relative path reference to a Python filter script for Pandoc, in a Pandoc .yaml file?

Answered by sdbbs

Jan 11, 2024

Right - I think I can confirm now, that this:

if the MSYS2 version of Python sees an argument which does not look like an UNIX absolute path (i.e. it does not start with a / slash, and has a colon), then it prepends the UNIX path of the current working directory to that path/argument

.... was indeed the culprit.

Note that in MSYS2, you might end up with multiple Python installations, depending on the shell you use:

Under MSYS2 shell of MSYS2, there is this Python (as noted in OP):

$ python --version
Python 3.11.6

$ which python
/usr/bin/python

$ cygpath -w $(which python)
C:\msys64\usr\bin\python.exe

Under MINGW64 shell of MSYS2, there is this Python:
```
$ python --version
Python 3.11…
```

View full answer

jgm · 2024-01-10T17:17:32Z

jgm
Jan 10, 2024
Maintainer

Do you get the same problem if you are producing html rather than pdf? That would help rule out Text.Pandoc.PDF as a culprit.

0 replies

sdbbs · 2024-01-10T17:26:13Z

sdbbs
Jan 10, 2024
Author

Thanks @jgm :

Do you get the same problem if you are producing html rather than pdf? That would help rule out Text.Pandoc.PDF as a culprit.

If I recall correctly, all I need to do is to change the output filename extension from .pdf to .html - so for this entry in .yaml file:

...
- ${.}/../scripts/pandoc/python-test.py

... I still get the same error for .html output:

$ pandoc --defaults=../my-pandoc.yaml --toc test_example.md -o test_example.html
Hey, got a span:        Span ("01",[],[]) [Str "Hello"]
Hey, got a span:        Span ("02",[],[]) [Str "World"]
python: can't open file '/tmp/pandoc_test/docs/test_example/C:\\msys64\\tmp\\pandoc_test\\docs/../scripts/pandoc/python-test.py': [Errno 2] No such file or directory
Error running filter C:\msys64\tmp\pandoc_test\docs/../scripts/pandoc/python-test.py:
Filter returned error status 2

0 replies

jgm · 2024-01-11T03:18:23Z

jgm
Jan 11, 2024
Maintainer

It looks like the way ${.} gets resolved is by doing

takeDirectory <$> canonicalizePath defaultsFilePath

Maybe this behaves in unexpected ways on Windows -- I'm sorry, I don't have a Windows box set up to test on.

0 replies

jgm · 2024-01-11T03:20:23Z

jgm
Jan 11, 2024
Maintainer

If you have ghc installed, you could try running ghci and then

> :m + System.Directory System.FilePath
> takeDirectory <$> canonicalizePath defaultsFilePath

where for defaultsFilePath put (in "") the path to the defaults file just as specified on the command line.

0 replies

sdbbs · 2024-01-11T13:40:53Z

sdbbs
Jan 11, 2024
Author

Thanks @jgm :

It looks like the way ${.} gets resolved is by doing
takeDirectory <$> canonicalizePath defaultsFilePath
Maybe this behaves in unexpected ways on Windows -- I'm sorry, I don't have a Windows box set up to test on.

No problem, thanks for taking a look!

However, I do not think this is solely ${.} which is to blame, because 1) lua works with it fine; and 2) python shows a problem, even if I use absolute path (though in "mixed mode") - C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py in yaml (i.e. ${.} would not be used for the python script.

If you have ghc installed, you could try running ghci and then
> :m + System.Directory System.FilePath
> takeDirectory <$> canonicalizePath defaultsFilePath
where for defaultsFilePath put (in "") the path to the defaults file just as specified on the command line.

Sure! I did not have ghc installed, but after a bit of a mess ( https://stackoverflow.com/questions/77800125/completely-avoid-use-of-appdata-when-installing-haskell-stack-on-windows ) managed to install it; here is what I got, running from the /tmp/pandoc_test/docs/test_example directory:

$ STACK_ROOT='C:\HaskellStack' /c/HaskellStack/stack.exe ghci

Note: No local targets specified, so a plain ghci will be started with no package hiding or package options.

      You are using snapshot: lts-22.5

      If you want to use package hiding and options, then you can try one of the following:

      * If you want to start a different project configuration
        than C:\HaskellStack\global-project\stack.yaml, then you can use stack init to create a new stack.yaml for the
        packages in the current directory.

      * If you want to use the project configuration
        at C:\HaskellStack\global-project\stack.yaml, then you can add to its 'packages' field.

Configuring GHCi with the following packages:
GHCi, version 9.6.3: https://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from C:\Users\[USER]\AppData\Local\stack\ghci-script\2a3bbd58\ghci-script
ghci> :m + System.Directory System.FilePath
ghci> takeDirectory <$> canonicalizePath "../my-pandoc.yaml"
"C:\\msys64\\tmp\\pandoc_test\\docs"

Looks reasonable, since cygpath -w $(dirname $(readlink -f ../my-pandoc.yaml)) from the same location also responds with C:\msys64\tmp\pandoc_test\docs

I'm starting to suspect, that if the MSYS2 version of Python sees an argument which does not look like an UNIX absolute path (i.e. it does not start with a / slash, and has a colon), then it prepends the UNIX path of the current working directory to that path/argument; I'll see if I can confirm that ...

0 replies

sdbbs · 2024-01-11T14:49:59Z

sdbbs
Jan 11, 2024
Author

Right - I think I can confirm now, that this:

if the MSYS2 version of Python sees an argument which does not look like an UNIX absolute path (i.e. it does not start with a / slash, and has a colon), then it prepends the UNIX path of the current working directory to that path/argument

.... was indeed the culprit.

Note that in MSYS2, you might end up with multiple Python installations, depending on the shell you use:

Under MSYS2 shell of MSYS2, there is this Python (as noted in OP):

$ python --version
Python 3.11.6

$ which python
/usr/bin/python

$ cygpath -w $(which python)
C:\msys64\usr\bin\python.exe

Under MINGW64 shell of MSYS2, there is this Python:

$ python --version
Python 3.11.7

$ which python
/mingw64/bin/python

$ cygpath -w $(which python)
C:\msys64\mingw64\bin\python.exe

Also, my Python script in the OP was very wrong, here is the corrected python-test.py:

#! /usr/bin/env python
import subprocess
import os
import sys
import shutil
from pandocfilters import toJSONFilter, Str, Para, Image, Span, attributes

def do_process(key, value, fmt, meta):
  #sys.stderr.write(f"{key=}, {value=}, {fmt=}, {meta=}\n")
  if key == 'Span':
    sys.stderr.write("Hey, python got a span, key: {} value: {} fmt: {} meta: {}\n".format(key, value, fmt, meta))
    # see also: [Pandoc filter to convert Critic Markup to Spans (which will be converted to Tracked Changes in Docx).](https://gist.github.com/HeirOfNorton/7dbcaa7c22297fe1b303)
    retval = Span(value[0], value[1])
    #sys.stderr.write("retval {}\n".format(retval))
    return retval

if __name__ == "__main__":
  #print("main python (print)") # no printout; AND causes "Error in $: Failed reading: not a valid json value at 'mainpython(print)"!!
  sys.stderr.write("main python (stderr)\n") # works
  toJSONFilter(do_process)

So - if I use this my-pandoc.yaml:

pdf-engine: xelatex
# NOTE https://github.com/jgm/pandoc/issues/6152 - resource path should be colon separated?
resource-path: [".:../scripts/pandoc:${.}/../scripts/pandoc"]
filters:
- ${.}/../scripts/pandoc/span-test.lua
- ${.}/../scripts/pandoc/python-test.py

... then, if I run the pandoc command in the MSYS2 shell of MSYS2, I get a failure:

MSYS /tmp/pandoc_test/docs/test_example
$ pandoc --defaults=../my-pandoc.yaml --toc test_example.md -o test_example.pdf
Hey, got a span:        Span ("01",[],[]) [Str "Hello"]
Hey, got a span:        Span ("02",[],[]) [Str "World"]
python: can't open file '/tmp/pandoc_test/docs/test_example/C:\\msys64\\tmp\\pandoc_test\\docs/../scripts/pandoc/python-test.py': [Errno 2] No such file or directory
Error running filter C:\msys64\tmp\pandoc_test\docs/../scripts/pandoc/python-test.py:
Filter returned error status 2

... however, if I run the pandoc command in the MINGW64 shell of MSYS2, it succeeds:

MINGW64 /tmp/pandoc_test/docs/test_example
$ pandoc --defaults=../my-pandoc.yaml --toc test_example.md -o test_example.pdf
Hey, got a span:        Span ("01",[],[]) [Str "Hello"]
Hey, got a span:        Span ("02",[],[]) [Str "World"]
main python (stderr)
Hey, python got a span, key: Span value: [['01', [], []], [{'t': 'Str', 'c': 'Hello'}]] fmt: latex meta: {'author': {'t': 'MetaInlines', 'c': [{'t': 'Str', 'c': 'Me'}]}, 'date': {'t': 'MetaInlines', 'c': [{'t': 'RawInline', 'c': ['tex', '\\today']}]}, 'title': {'t': 'MetaInlines', 'c': [{'t': 'Str', 'c': 'Example'}, {'t': 'Space'}, {'t': 'Str', 'c': 'test'}, {'t': 'Space'}, {'t': 'Str', 'c': '-'}, {'t': 'Space'}, {'t': 'Str', 'c': 'hello'}, {'t': 'Space'}, {'t': 'Str', 'c': 'world'}]}}
Hey, python got a span, key: Span value: [['02', [], []], [{'t': 'Str', 'c': 'World'}]] fmt: latex meta: {'author': {'t': 'MetaInlines', 'c': [{'t': 'Str', 'c': 'Me'}]}, 'date': {'t': 'MetaInlines', 'c': [{'t': 'RawInline', 'c': ['tex', '\\today']}]}, 'title': {'t': 'MetaInlines', 'c': [{'t': 'Str', 'c': 'Example'}, {'t': 'Space'}, {'t': 'Str', 'c': 'test'}, {'t': 'Space'}, {'t': 'Str', 'c': '-'}, {'t': 'Space'}, {'t': 'Str', 'c': 'hello'}, {'t': 'Space'}, {'t': 'Str', 'c': 'world'}]}}

Also, re-checked the results while specifying absolute paths for the python script (even if it does not work for my use case):

# ...
#- ${.}/../scripts/pandoc/python-test.py
#- /c/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py # nowork in neither MSYS2 nor MINGW64
- C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py # works in MINGW64, nowork in MSYS2

... or in other words:

Specifying an absolute path as Unix path (starts with forward slash, lowercase letters for drives) does not work - works with neither MSYS2 nor MINGW64 Pythons
Specifying an absolute path as mixed (starts with uppercase drive letter and colon, uses forward slashes) works with MINGW64 Python, but does not work with MSYS2 Python

So, the general workaround for now is: if you want to run a Python filter script (specified in .yaml file via ${.} relative path) in Pandoc under MSYS2 -- then use the MINGW64 shell of MSYS2 (and its corresponding Python; it seems differences in handling Windows vs Unix vs mixed absolute paths is among the differences between MINGW64 and MSYS2 Pythons).

1 reply

jgm Jan 11, 2024
Maintainer

Thanks for figuring this out!

sdbbs · 2024-01-11T20:26:19Z

sdbbs
Jan 11, 2024
Author

Since this problem was irritating, I found also a workaround that makes a Python Pandoc filter script work in both MSYS2 and MINGW64 shells. In brief - we create a "passthrough" Lua script, which then calculates the correct Python filter script path depending on the calling shell, and calls python with the right script path, to process the entire Pandoc document.

Since our Python script was called python-test.py, the passthrough script will be called python-test.lua - save it in the same directory where python-test.py is (note, since I'm not good at Lua or Pandoc API, I have left quite a lot of errors as comments):

function dump(o)
  if type(o) == 'table' then
    local s = '{ '
    for k,v in pairs(o) do
      if type(k) ~= 'number' then k = '"'..k..'"' end
      s = s .. '['..type(k)..'] = ' .. dump(v) .. ',\n'
    end
    return s .. '} '
  else
    return tostring(o) .. "//" .. type(o)
  end
end

function getPath(str)
  return str:match("(.*[/\\])")
end

-- https://stackoverflow.com/questions/49702708/using-a-lua-filter-how-do-you-convert-a-table-into-json-or-native-text
local utils = require 'pandoc.utils'

local current_dir=io.popen"cd":read'*l' -- SO:6032268, works in MSYS2/Windows; the cwd (where we call the pandoc command from)!
local fullpath = debug.getinfo(1,"S").source:sub(2) -- http://lua-users.org/lists/lua-l/2020-01/msg00345.html ; full path to this .lua script
print("current_dir", current_dir)
print("fullpath", fullpath)

--current_dir     C:\msys64\tmp\pandoc_test\docs\test_example
--fullpath        C:\msys64\tmp\pandoc_test\docs/../scripts/pandoc/python-test.lua

-- print(debug.getinfo(1,"S")) -- table: 0000018cf08bc050
--print(dump(debug.getinfo(1,"S")))
--
--{ [string] = @C:\msys64\tmp\pandoc_test\docs/../scripts/pandoc/python-test.lua//string,
--[string] = 0//number,
--[string] = 0//number,
--[string] = main//string,
--[string] = ...4\tmp\pandoc_test\docs/../scripts/pandoc/python-test.lua//string,
--}

-- https://www.tutorialspoint.com/io-popen-function-in-lua-programming
-- https://www.gammon.com.au/scripts/doc.php?lua=io.popen
-- https://www.gammon.com.au/scripts/doc.php?lua=f:read

-- apparently, cmdline gets executed by cmd.exe - even if we run pandoc from a MSYS2 (or MINGW64) bash shell!
-- so, we first want to execute cd to target directory, then just run `cd` without arguments to print out current path (in Linux, `cd` without arguments does not print anything, and instead changes current working directory to the user home directory) - and since cmd.exe executes this, we use & as command separator
local cmdline = "cd " .. getPath(fullpath) .. " & cd"
print("cmdline", cmdline)
local handle = io.popen(cmdline)
-- use "*l" to get a single line; if we use "*a" for all, we also get extra (CR)LF!
local result_winpath_scriptdir = handle:read("*l")
handle:close()
print("result_winpath_scriptdir", result_winpath_scriptdir)

-- cmdline cd C:\msys64\tmp\pandoc_test\docs/../scripts/pandoc/ & cd
-- result_winpath_scriptdir  C:\msys64\tmp\pandoc_test\scripts\pandoc

-- so result_winpath_scriptdir is currently a Windows path;
-- convert it to Unix/MSYS2 one with string processing:
-- 1. replace backslash with forward slash:
local mixed_scriptdir = result_winpath_scriptdir:gsub("\\","/")
--print("unixpath", unixpath) -- " C:/msys64/tmp/pandoc_test/scripts/pandoc"
-- 2. Replace the starting "DRIVELETTER:" with "/DRIVELETTER"
-- seemingly, msys can resolve drive letters with wither uppercase (e.g. `/C/`) or lowercase (e.g. `/c/`) letter; so here we can just remove the colon first:
local unixpath = mixed_scriptdir:gsub(":","")
-- ... and then just prepend a forward slash:
unixpath = "/" .. unixpath
print("unixpath", unixpath) -- "/C/msys64/tmp/pandoc_test/scripts/pandoc"

-- calculate the unix path to Python script
local unixpypath = unixpath .. "/" .. "python-test.py"
print("unixpypath", unixpypath) -- "/C/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py"
local mixedpypath = mixed_scriptdir .. "/" .. "python-test.py"
print("mixedpypath", mixedpypath) -- "C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py"

-- NOTE: the "/C/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py" path works in MSYS2, but MINGW64 fails with: "python: can't open file 'C:/C/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py': [Errno 2] No such file or directory";
-- so MINGW64 ends up prepending DRIVELETTER: to the Unix path;
-- therefore, determine the calling shell, and supply the python script path accordingly

-- https://stackoverflow.com/questions/55559604/how-can-i-find-out-the-version-of-msys2-from-its-bash-shell
-- `cat /proc/version` might produce:
-- MSYS_NT-10.0-19045 version 3.4.10.x86_64 (runneradmin@fv-az1495-832) (gcc version 13.2.0 (GCC) ) 2023-12-22 10:06 UTC
-- MINGW64_NT-10.0-19045 version 3.4.10.x86_64 (runneradmin@fv-az1495-832) (gcc version 13.2.0 (GCC) ) 2023-12-22 10:06 UTC

handle = io.popen("cat /proc/version")
local shell_version = handle:read("*l")
handle:close()
print("shell_version", shell_version)

-- finally, allow for calling the Python filter script on the entire Pandoc document
-- https://pandoc.org/lua-filters.html#pandoc.utils.run_json_filter
function Pandoc(blocks)
  --local proc_doc = utils.run_json_filter(blocks, unixpypath) -- Could not find executable /C/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py
  --local proc_doc = utils.run_json_filter(blocks, "python " .. unixpypath) -- Could not find executable python /C/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py
  local pypathargs = {unixpypath} -- for MSYS2
  if shell_version:match "^MINGW64" then -- SO:30150096
    pypathargs = {mixedpypath}
  end
  print("pypathargs", pypathargs) -- table: 000001d008ff16f0
  print( dump(pypathargs) )
  local proc_doc = utils.run_json_filter(blocks, "python", pypathargs)
  --return proc_doc.blocks -- Error running filter Pandoc expected, got Blocks
  return proc_doc
end

So, with this script, the structure in this example looks like:

$ tree /tmp/pandoc_test/
/tmp/pandoc_test/
├── docs
│   ├── my-pandoc.yaml
│   └── test_example
│       └── test_example.md
└── scripts
    └── pandoc
        ├── python-test.lua
        ├── python-test.py
        └── span-test.lua

Also, replace my-pandoc.yaml to use the passthrough script instead:

pdf-engine: xelatex
# NOTE https://github.com/jgm/pandoc/issues/6152 - resource path should be colon separated?
resource-path: [".:../scripts/pandoc:${.}/../scripts/pandoc"]
filters:
- ${.}/../scripts/pandoc/span-test.lua
#- ${.}/../scripts/pandoc/python-test.py
#- /c/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py # nowork in neither MSYS2 nor MINGW64
#- C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py # works in MINGW64, nowork in MSYS2
- ${.}/../scripts/pandoc/python-test.lua

Now, we can call pandoc in either of the MSYS2 or MINGW64 shells, and have the build in this example complete - here is the output from the MSYS2 shell (since that one was causing trouble up until now):

MSYS /tmp/pandoc_test/docs/test_example
$ pandoc --defaults=../my-pandoc.yaml --toc test_example.md -o test_example.pdf
Hey, got a span:        Span ("01",[],[]) [Str "Hello"]
Hey, got a span:        Span ("02",[],[]) [Str "World"]
current_dir     C:\msys64\tmp\pandoc_test\docs\test_example
fullpath        C:\msys64\tmp\pandoc_test\docs/../scripts/pandoc/python-test.lua
cmdline cd C:\msys64\tmp\pandoc_test\docs/../scripts/pandoc/ & cd
result_winpath_scriptdir        C:\msys64\tmp\pandoc_test\scripts\pandoc
unixpath        /C/msys64/tmp/pandoc_test/scripts/pandoc
unixpypath      /C/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py
mixedpypath     C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py
shell_version   MSYS_NT-10.0-19045 version 3.4.10.x86_64 (runneradmin@fv-az1495-832) (gcc version 13.2.0 (GCC) ) 2023-12-22 10:06 UTC
pypathargs      table: 000001f51f3f32a0
{ [number] = /C/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py//string,
}
main python (stderr)
Hey, python got a span, key: Span value: [['01', [], []], [{'t': 'Str', 'c': 'Hello'}]] fmt:  meta: {'author': {'t': 'MetaInlines', 'c': [{'t': 'Str', 'c': 'Me'}]}, 'date': {'t': 'MetaInlines', 'c': [{'t': 'RawInline', 'c': ['tex', '\\today']}]}, 'title': {'t': 'MetaInlines', 'c': [{'t': 'Str', 'c': 'Example'}, {'t': 'Space'}, {'t': 'Str', 'c': 'test'}, {'t': 'Space'}, {'t': 'Str', 'c': '-'}, {'t': 'Space'}, {'t': 'Str', 'c': 'hello'}, {'t': 'Space'}, {'t': 'Str', 'c': 'world'}]}}
Hey, python got a span, key: Span value: [['02', [], []], [{'t': 'Str', 'c': 'World'}]] fmt:  meta: {'author': {'t': 'MetaInlines', 'c': [{'t': 'Str', 'c': 'Me'}]}, 'date': {'t': 'MetaInlines', 'c': [{'t': 'RawInline', 'c': ['tex', '\\today']}]}, 'title': {'t': 'MetaInlines', 'c': [{'t': 'Str', 'c': 'Example'}, {'t': 'Space'}, {'t': 'Str', 'c': 'test'}, {'t': 'Space'}, {'t': 'Str', 'c': '-'}, {'t': 'Space'}, {'t': 'Str', 'c': 'hello'}, {'t': 'Space'}, {'t': 'Str', 'c': 'world'}]}}

The only difference in the above output, when pandoc is called from a MINGW64 shell, is:

...
shell_version   MINGW64_NT-10.0-19045 version 3.4.10.x86_64 (runneradmin@fv-az1495-832) (gcc version 13.2.0 (GCC) ) 2023-12-22 10:06 UTC
pypathargs      table: 000001d0bb7e1900
{ [number] = C:/msys64/tmp/pandoc_test/scripts/pandoc/python-test.py//string,
}
...

Hope that was it with MSYS2/MINGW64 Python filter scripts for pandoc :)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Relative path problem when calling Pandoc Python filter in subfolders (MSYS2 on Windows) #9328

{{title}}

Replies: 7 comments 1 reply

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Relative path problem when calling Pandoc Python filter in subfolders (MSYS2 on Windows) #9328

sdbbs Jan 10, 2024

Replies: 7 comments · 1 reply

jgm Jan 10, 2024 Maintainer

sdbbs Jan 10, 2024 Author

jgm Jan 11, 2024 Maintainer

jgm Jan 11, 2024 Maintainer

sdbbs Jan 11, 2024 Author

sdbbs Jan 11, 2024 Author

jgm Jan 11, 2024 Maintainer

sdbbs Jan 11, 2024 Author

sdbbs
Jan 10, 2024

Replies: 7 comments 1 reply

jgm
Jan 10, 2024
Maintainer

sdbbs
Jan 10, 2024
Author

jgm
Jan 11, 2024
Maintainer

jgm
Jan 11, 2024
Maintainer

sdbbs
Jan 11, 2024
Author

sdbbs
Jan 11, 2024
Author

jgm Jan 11, 2024
Maintainer

sdbbs
Jan 11, 2024
Author