Skip to content

Commit

Permalink
Debugging: scripts and documentation (#21)
Browse files Browse the repository at this point in the history 
Co-authored-by: urso <steffen.siering at gmail.com>
  • Loading branch information
@urso
urso authored Mar 14, 2024
1 parent ca3c327 commit e71c4e5
Show file tree
Hide file tree
Showing 7 changed files with 490 additions and 34 deletions.
135 changes: 135 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,141 @@ ok 1 - char_count_test 10 ms
# All 1 tests passed.
```

### Debugging the unit tests

Because Postgres manages the actual processes and starts a new process for each client we must attach our debugger to an existing session.

To debug an extension that exposes a function, like the unit tests, first start a SQL session:

```
$ psql -U postgres
```

In order to attach our debugger to the session we need the PID for your current process:

```
postgres=# select pg_backend_pid();
pg_backend_pid
----------------
14985
(1 row)
```

If we want to set breakpoints we must also ensure that our extensions library has been loaded. You might have to drop and re-create an the test function in case you can't set a breakpoint (this will force Postgres to load the library):

```
postgres=# DROP FUNCTION run_tests;
CREATE FUNCTION run_tests() RETURNS INTEGER AS '$libdir/pgzx_unit' LANGUAGE C IMMUTABLE;
```

Now we can attach our debugger and set a breakpoint (See your debuggers documentation on how to attach):

```
$ lldb -p 14985
(lldb) b hsearch.zig:117
...
(lldb) c
```

You can set breakpoints in your Zig based extension in C sources given you have all debug symbols available.

With our breakpoints set we want to start the testsuite:

```
postgres=# SELECT run_tests();
```


### Postgres debug build

The default development shell and scripts relocated the Nix postgres installation into the `out` folder only. When debugging extension in isolation this is normally all you need. But in case you need to debug and step into the Postgres sources as well it is helpful to have a debug build available.

This project provides a second development shell type that provides tooling to fetch and build Postgres in debug mode.

Select the `debug` shell to start a shell with required development tools:

```
nix develop '.#debug'
```

The `pgbuild` script will fetch the Postgres sources and build a local debug build that you can use for development, testing, and debugging:

```
$ pgbuild
```

This will checkout Postgres into the `out/postgresql_src` directory. The build files will be stored in `out/postgresql_src/build`. We use [meson](https://mesonbuild.com/) and [Ninja](https://ninja-build.org/) to build Postgres. Ninja speeds up the compilation by parallelizing compilation tasks as much as possible. Compilation via Ninja also emits a `compile_commands.json` file that you can use with your editors LSP to improve navigating the Postgres source code if you wish to do so.

Optionally symlink the `compile_commands.json` file:

```
$ ln -s ./out/postgresql_src/build/compile_commands.json ./out/postgresql_src/compile_commands.json
```

The local build will be installed in `out/local`. To switch to the local Postgres build and ensure that your extension is build against it use:

```
$ pguse local
```

Note: Delete the `zig-cache` folder when switching to another Postgres installation to ensure that you extension is rebuild properly against the new versions.


### Debugging Zig standard library and build script support

To debug Zig build scripts or the standard library all you need is the original sources. No additional build step is required. Anyways, it is recommended to use the same library version as is the zig compiler ships with. You can query the current version or Git commit of a nightly build using the `zig` tool:

```
$ zig version
0.12.0-dev.3154+0b744da84
```

The version shown here for example indicates that we use a nightly build. The commit ID of that build is `0b744da84`.

You can clone and checkout the repository by yourself. We also have a small script `ziglocal` to checkout and even build the compiler. You can use the script to just checkout the correct version into your development environment:

```
$ ziglocal clone --commit 0b744da84
```

This command clones the master branch only into the `./out/zig` directory.

Now when building the test extensions you can use the `--zig-lib-dir` CLI flag to tell the compiler to use an alternative library:

```
$ zig build unit -p $PG_HOME --zig-lib-dir $PRJ_ROOT/out/zig/lib
```

The zig compiler will now use the local checkout to build the `build.zig` file and your project.

### Debugging Zig compiler/linker

As Zig is still in development, you might have the need to build the Zig toolchain yourself, maybe in debug mode.

The `debug` shell installs the additional dependencies that you need to build Postgres or the Zig compiler yourself.

```
nix develop '.#debug'
```

Optionally we might want to debug the actual version that we normally use:

```
$ zig version
0.12.0-dev.3154+0b744da84
```

Next we checkout and compile the toolchain (Note: the `--commit` option is optional):

```
$ ziglocal --commit 0b744da84
```

This step will take a while. You will find the compiler and library of your local debug build in the `out/zig/build/stage3` directory.



[docs_Log]: https://xataio.github.io/pgzx/#A;pgzx:elog.Log
[docs_Info]: https://xataio.github.io/pgzx/#A;pgzx:elog.Info
[docs_Notice]: https://xataio.github.io/pgzx/#A;pgzx:elog.Notice
Expand Down
40 changes: 27 additions & 13 deletions build.zig
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,15 @@ pub fn build(b: *std.Build) void {
.optimize = optimize,
});
module.addIncludePath(.{
.path = b.pathFromRoot("./src/pgzx/c/include/"),
.path = "./src/pgzx/c/include/",
});
module.addIncludePath(.{
.cwd_relative = pgbuild.getIncludeServerDir(),
});
// libpq support
module.addCSourceFiles(.{
.files = &[_][]const u8{
b.pathFromRoot("./src/pgzx/c/libpqsrv.c"),
"./src/pgzx/c/libpqsrv.c",
},
.flags = &[_][]const u8{
"-I", pgbuild.getIncludeDir(),
Expand All @@ -53,29 +53,43 @@ pub fn build(b: *std.Build) void {
};

// Unit test extension
{
const psql_run_tests = pgbuild.addRunTests(.{
.name = "pgzx_unit",
.db_user = "postgres",
.db_port = 5432,
});

const test_ext = blk: {
const test_options = b.addOptions();
test_options.addOption(bool, "testfn", true);

const test_ext = pgbuild.addInstallExtension(.{
const tests = pgbuild.addInstallExtension(.{
.name = "pgzx_unit",
.version = .{ .major = 0, .minor = 1 },
.root_source_file = .{
.path = "src/testing.zig",
},
.root_dir = "src/testing",
.link_libc = true,
.link_allow_shlib_undefined = true,
});
test_ext.lib.root_module.addIncludePath(.{
tests.lib.root_module.addIncludePath(.{
.path = b.pathFromRoot("./src/pgzx/c/include/"),
});
test_ext.lib.root_module.addImport("pgzx", pgzx);
test_ext.lib.root_module.addOptions("build_options", test_options);
tests.lib.root_module.addImport("pgzx", pgzx);
tests.lib.root_module.addOptions("build_options", test_options);

break :blk tests;
};

// Unit tests installer target
// Optionally build and install the extension, so we can hook up with t a debugger and run tests manually.
{
var install_unit = b.step("install-unit", "Install unit tests extension (for manual testing)");
install_unit.dependOn(&test_ext.step);
}

// Unit test runner
{
const psql_run_tests = pgbuild.addRunTests(.{
.name = "pgzx_unit",
.db_user = "postgres",
.db_port = 5432,
});

psql_run_tests.step.dependOn(&test_ext.step);

Expand Down
163 changes: 163 additions & 0 deletions dev/bin/pgbuild
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
#!/usr/bin/env bash

# Script to fetch and build postgres from source.
#
# We use meson and ninja to build postgres. Thanks to this we get a
# compile_commands.json file that can be used with LSP to navigate the postgres
# source code and parallel compilation out of the box, which improves
# compilation times quite a bit.

#set -x
set -e

USAGE="
Usage: $0 <options>
-h: print this help message
-r|--repo <repo>: git repository to clone postgres from.
-b|--branch <branch>: branch to clone from the repository.
Environment variables:
WORKDIR: directory to store the files and installation.
the root of the project.
PG_SRC_DIR: directory to store the postgres source code. Defaults to \$WORKDIR/postgresql_src.
PG_BUILD_DIR: directory to store the build files. Defaults to \$PG_SRC_DIR/build.
PG_INSTALL_DIR: directory to install postgres to. Defaults to \$WORKDIR/local.
"

ROOTDIR=${PRJ_ROOT:-$(git rev-parse --show-toplevel)}
WORKDIR=${WORKDIR:-"${ROOTDIR}/out"}
PG_SRC_DIR=${PG_SRC_DIR:-"$WORKDIR/postgresql_src"}
PG_BUILD_DIR=${PG_BUILD_DIR:-"$PG_SRC_DIR/build"}
PG_INSTALL_DIR=${PG_INSTALL_DIR:-"$WORKDIR/local"}

# Use the github mirror by default
POSTGRES_REPO=${POSTGRES_REPO:-https://github.com/postgres/postgres.git}
POSTGRES_BRANCH=${POSTGRES_BRANCH:-"REL_16_STABLE"}

POSITIONAL_ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
-h)
echo "$USAGE"
exit 0
;;
--repo | -r)
POSTGRES_REPO="$2"
shift 2
;;
-b | --branch)
POSTGRES_BRANCH="$2"
shift 2
;;
*)
POSITIONAL_ARGS+=("$1") # save positional arg
shift # past argument
;;
esac
done
set -- "${POSITIONAL_ARGS[@]}" # restore positional parameters

clone() {
echo "Downloading postgres source from $POSTGRES_REPO"

if [ -d "$PG_SRC_DIR" ] && [ -n "$(ls -A $PG_SRC_DIR)" ]; then
echo "The directory $PG_SRC_DIR already exists and is not empty. Aborting."
return 0
fi

mkdir -p $(dirname $PG_SRC_DIR)
git clone --single-branch -b ${POSTGRES_BRANCH} ${POSTGRES_REPO} $PG_SRC_DIR
}

patch() {
# The meson scripts assume that clang is part of the LLVM suite, which is used when
# enabling the jit build (which we do).
# Unfortunately the is not always the case. We work around this by patching the build scripts:
git -C "$PG_SRC_DIR" apply - <<EOF
diff --git a/meson.build b/meson.build
index 6804f941be..d62142c8f7 100644
--- a/meson.build
+++ b/meson.build
@@ -755,7 +755,7 @@ if add_languages('cpp', required: llvmopt, native: false)
llvm_binpath = llvm.get_variable(configtool: 'bindir')
ccache = find_program('ccache', native: true, required: false)
- clang = find_program(llvm_binpath / 'clang', required: true)
+ clang = find_program(llvm_binpath / 'clang', 'clang', required: true)
endif
elif llvmopt.auto()
message('llvm requires a C++ compiler')
EOF
}

configure() {
echo "Configuring postgres build"

patch

meson setup "$PG_BUILD_DIR" "$PG_SRC_DIR" \
--debug \
-Dprefix=${PG_INSTALL_DIR} \
-Dbindir=${PG_INSTALL_DIR}/bin \
-Ddatadir=${PG_INSTALL_DIR}/share \
-Dincludedir=${PG_INSTALL_DIR}/include \
-Dlibdir=${PG_INSTALL_DIR}/lib \
-Dsysconfdir=${PG_INSTALL_DIR}/etc \
-Dplperl=auto \
-Dplpython=auto \
-Dssl=openssl \
-Dpam=auto \
-Dldap=auto \
-Dlibxml=enabled \
-Dlibxslt=enabled \
-Dllvm=auto \
-Duuid=none \
-Dzstd=auto \
-Dlz4=enabled \
-Dgssapi=auto \
-Dsystemd=auto \
-Dicu=auto \
-Dsystem_tzdata=/usr/share/zoneinfo
}

build() {
echo "Building postgres"

ninja -C "$PG_BUILD_DIR"
}

install() {
echo "Installing postgres to $PG_INSTALL_DIR"

mkdir -p ${PG_INSTALL_DIR}
ninja -C ${PG_BUILD_DIR} install
}

# The first argument determines the function to run. If no argument is given we run all commands in order:

if [ -z "$1" ]; then
clone && configure && build && install
exit 0
fi

case $1 in
clone)
clone
;;
patch)
patch
;;
configure)
configure
;;
build)
build
;;
install)
install
;;
*)
echo "Usage: $0 {clone|configure|build|install}"
exit 1
;;
esac
8 changes: 4 additions & 4 deletions dev/bin/pguse
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
set -e
set -x

rootdir=${PRJ_ROOT:-$(git rev-parse --show-toplevel)}
outdir=${1:-$rootdir/out}
ROOTDIR=${PRJ_ROOT:-$(git rev-parse --show-toplevel)}
WORKDIR=${WORKDIR:-"$ROOTDIR/out"}
version=${1:-"16"}

rm -f $outdir/default
ln -s $outdir/$version $outdir/default
rm -f $WORKDIR/default
ln -s $WORKDIR/$version $WORKDIR/default
Loading

0 comments on commit e71c4e5

Please sign in to comment.