Use Khronos asciidoctor-spec Docker image in CI (#1196)

* Use Khronos asciidoctor-spec Docker image in CI

Per discussion with $bashbaug

N.b. at present the CI script has less parallelism than it could, at
least as I understand Actions. Some of the 'steps' could be split off
into 'jobs'. Might try that next once the basic build is working. Net
performance is still somewhat faster than current CI since it's
generally faster to load the container than to add needed packages at
each invocation, and the spec build is pretty fast, so there's not much
to be gained.

There was odd error behavior from shifting to the container which I have
never seen in Vulkan CI, having to do with mixed ownership of files in
the checked-out repository. I inserted a brute-force workaround right
after the checkout action.

* Update image (SHA changed, though not contents)

* Update to 20240702 Docker image which sets HOME=/tmp to avoid asciidoctor-pdf permission problems with tmpfiles in home directory /

* Fix SHA and remove fixed parallel job limit on manhtmlpages build

* Switch to '#!/usr/bin/env python3' shebang lines

Since the Docker build image runs a python virtual environment now.

Also added 'scripts/runDocker' script which will invoke docker locally
with the same image used in Github CI, for testing.

Note this script will pull over a GB of Docker stuff onto the machine
it's invoked on, if the image is not already cached.

* Empty commit to try and re-trigger the 'fatal' message...

... which appears sporadic, not easily replicable.

* Try to bulletproof the git invocations in Makefile

* Remove diagnostic job stage after bulletproofing (hopefully) the Makefile

For future reference, some of the git operations in CI and the Makefile
appear to *sporadically* fail in CI because of different checked-out
repo configurations.

I modified the 'git symbolic-ref' and 'git log' operations invoked from
the Makefile to detect errors and substitute a placeholder message,
based on similar changes to the Vulkan Makefile a while back. This
(appears) to eliminate the sporadic 'fatal' messages. We may need to do
that to the 'git describe' as well.

None of this reads on the generated artifacts, except that they may or
may not contain accurate tag / commit comments.

.github/workflows/presubmit.yml

@@ -3,47 +3,54 @@ name: Presubmit
 permissions:
  contents: read
 
-on: [push, pull_request]
+# Controls when the action will run.
+on:
+ push:
+ workflow_dispatch:
+ pull_request:
 
+# These jobs are potentially parallelizeable
 jobs:
  build:
- name: Build all specs
+ name: Build spec artifacts
  runs-on: ubuntu-latest
+ # Refer to the build container by its SHA instead of the name, to
+ # prevent caching problems when updating the image.
+ # container: khronosgroup/docker-images:asciidoctor-spec.20240702
+ container: khronosgroup/docker-images@sha256:4aab96a03ef292439c9bd0f972adfa29cdf838d0909b1cb4ec2a6d7b2d14a37f
 
  steps:
  - uses: actions/checkout@v4
  with:
- fetch-depth: 0
  submodules: recursive
+ # If fetch-depth: 0 is not specified, then
+ # git describe --tags --dirty
+ # below fails.
+ # This could also affect SPECREVISION in the Makefile.
+ fetch-depth: 0
 
- - name: Install required packages
+ # Ownerships in the working directory are odd.
+ # . is owned by UID 1001, while repo files are owned by root.
+ # This leads to many odd messages like
+ # fatal: detected dubious ownership in repository at '/__w/OpenCL-Docs/OpenCL-Docs'
+ # The 'git config' is a brute-force workaround.
+ - name: Git safe directory workaround
  run: |
- sudo apt-get install -y libpango1.0-dev libwebp-dev ghostscript fonts-lyx jing libavalon-framework-java libbatik-java python3-pyparsing
- sudo gem install asciidoctor -v 2.0.16
- sudo gem install coderay -v 1.1.1
- sudo gem install rouge -v 3.19.0
- sudo gem install ttfunk -v 1.7.0
- sudo gem install hexapdf -v 0.27.0
- sudo gem install asciidoctor-pdf -v 2.3.4
- sudo gem install asciidoctor-mathematical -v 0.3.5
- sudo pip install pyparsing
-
- - name: List git tag
+ git config --global --add safe.directory '*'
+ ls -lda . .. .git Makefile
+
+ - name: Validate XML
  run: |
- git describe --tags --dirty
+ make -C xml validate
 
  - name: Generate core specs (HTML and PDF)
  run: |
- python3 makeSpec -clean -spec core OUTDIR=out.core -j 5 api c env ext cxx4opencl
- 
+ python3 makeSpec -clean -spec core OUTDIR=out.core -j 5 -O api c env ext cxx4opencl
+
  - name: Generate core + extension specs (HTML)
  run: |
- python3 makeSpec -clean -spec khr OUTDIR=out.khr -j 12 html
+ python3 makeSpec -clean -spec khr OUTDIR=out.khr -j -O html
 
  - name: Generate reference pages
  run: |
- python3 makeSpec -spec khr OUTDIR=out.refpages -j 12 manhtmlpages
-
- - name: Validate XML
- run: |
- make -C xml validate
+ python3 makeSpec -spec khr OUTDIR=out.refpages -j -O manhtmlpages

Makefile

@@ -14,6 +14,7 @@ EXTOPTIONS := $(foreach ext,$(EXTS),-extension $(ext))
 
 QUIET ?=
 VERYQUIET ?= @
+PYTHON ?= python3
 ASCIIDOCTOR ?= asciidoctor
 RM = rm -f
 RMRF = rm -rf
@@ -72,8 +73,8 @@ SPECREVISION = $(shell echo `git describe --tags --dirty`)
 # This used to be a dependency in the spec html/pdf targets,
 # but that's likely to lead to merge conflicts. Just regenerate
 # when pushing a new spec for review to the sandbox.
-SPECREMARK = from git branch: $(shell echo `git symbolic-ref --short HEAD`) \
- commit: $(shell echo `git log -1 --format="%H"`)
+SPECREMARK = from git branch: $(shell echo `git symbolic-ref --short HEAD 2> /dev/null || echo Git branch not available`) \
+ commit: $(shell echo `git log -1 --format="%H" 2> /dev/null || echo Git commit not available`)
 endif
 # The C++ for OpenCL document revision scheme is aligned with its release date.
 # Revision naming scheme is as follows:
@@ -545,6 +546,7 @@ $(METADEPEND): $(APIXML) $(GENSCRIPT)
 attribs: $(ATTRIBFILE)
 
 $(ATTRIBFILE):
+ $(QUIET)$(MKDIR) $(dir $@)
  for attrib in $(EXTS) ; do \
  echo ":$${attrib}:" ; \
  done > $@

scripts/apiconventions.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2021-2024 The Khronos Group Inc.
 # SPDX-License-Identifier: Apache-2.0

scripts/cgenerator.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 #

scripts/checklinks.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 # SPDX-License-Identifier: Apache-2.0

scripts/clconventions.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 # SPDX-License-Identifier: Apache-2.0

scripts/docgenerator.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 #

scripts/extensionmetadocgenerator.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 #

scripts/genRef.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 #
 # Copyright 2016-2024 The Khronos Group Inc.
 #

scripts/gen_dictionaries.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 
 # Copyright 2019-2024 The Khronos Group Inc.
 # SPDX-License-Identifier: Apache-2.0

scripts/gen_version_notes.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 
 # Copyright 2019-2024 The Khronos Group Inc.
 # SPDX-License-Identifier: Apache-2.0

scripts/gencl.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 #

scripts/generator.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 #

scripts/parse_dependency.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 
 # Copyright 2022-2024 The Khronos Group Inc.
 # Copyright 2003-2019 Paul McGuire

scripts/pygenerator.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 #

scripts/realign.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 # SPDX-License-Identifier: Apache-2.0

scripts/reflib.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 #
 # Copyright 2016-2024 The Khronos Group Inc.
 #

scripts/reg.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 #

scripts/runDocker

@@ -0,0 +1,30 @@
+#!/bin/bash
+# Copyright 2022-2024 The Khronos Group Inc.
+# SPDX-License-Identifier: Apache-2.0
+
+# runDocker - run the Khronos `asciidoctor-spec` Docker image with a local
+# clone of the specification repository.
+# The following command-line tools are required to run this script:
+# awk dirname docker grep id realpath
+# These are all normal Linux developer tools except for 'docker' itself.
+
+# Determine path to repository root directory
+scriptpath=`dirname $0`
+repopath=`realpath $scriptpath/..`
+
+# Get SHA256 of the asciidoctor-spec image build used by CI.
+image=`grep -m 1 khronosgroup/docker-images@sha256: $repopath/.github/workflows/presubmit.yml | \
+ awk '{print $2}'`
+
+uid=`id -u`
+gid=`id -g`
+echo "Executing Docker with spec build image and mounted spec repository root:"
+
+# --user causes Docker to run as the specified UID:GID instead of as root
+# -it runs interactively and uses a pseudotty
+# --rm removes the container on exit
+# -v mounts the repository clone as /vulkan in the container
+# $image is image to run
+# /bin/bash drops into a shell in the container
+set -x
+docker run --network=host --user ${uid}:${gid} -it --rm -v ${repopath}:/opencl $image /bin/bash

scripts/scriptgenerator.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 #

scripts/spec_tools/conventions.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3 -i
+#!/usr/bin/env python3 -i
 #
 # Copyright 2013-2024 The Khronos Group Inc.
 #