Skip to content

Commit

Permalink
Release 1.1.0 (-loops, -time, doc and examples adapted)
Browse files Browse the repository at this point in the history
  • Loading branch information
Jean-Paul Argudo committed Jan 20, 2024
1 parent 2d70747 commit bcff015
Show file tree
Hide file tree
Showing 21 changed files with 393 additions and 61 deletions.
14 changes: 7 additions & 7 deletions bin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ You'll find here binaries for your convenience.

```
$ md5sum pgSimload*
371037d969fa1976573da79f276471ea pgSimload
f80007ed4a454c6559abe8a09f3c8206 pgSimload_mac
7349ab68d8d0128d03b2ff13f64a8e92 pgSimload_static
2b592fc16bc380d031b5705b5fedc86e pgSimload_win.exe
0676f622b8fc213c3a0707b42308b3e0 pgSimload
bee525f38cdbefcd64c58fe0722b26a7 pgSimload_mac
53365de34e4062b70f605aeb98aefe34 pgSimload_static
44a8b6b911fe29576bcb863119c5aa46 pgSimload_win.exe
```

# More information
Expand All @@ -29,9 +29,9 @@ If you're a Linux user, let me warn you that the binary has dependencies:

```
$ ldd bin/pgSimload
linux-vdso.so.1 (0x00007ffc9110f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb69c800000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb69ca6e000)
linux-vdso.so.1 (0x00007ffcad3a2000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc564800000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc564b18000)
```

Those depedencies would be OK if as an example you're using an up-to-date
Expand Down
Binary file modified bin/pgSimload
Binary file not shown.
Binary file modified bin/pgSimload_mac
Binary file not shown.
Binary file modified bin/pgSimload_static
Binary file not shown.
Binary file modified bin/pgSimload_win.exe
Binary file not shown.
19 changes: 14 additions & 5 deletions doc/01_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,6 @@

Welcome to pgSimload !

The actual version of the program is:

**pgSimload version 1.0.3 - January, 15th 2024**

pgSimload is a tool written in Go, and accepts 2 different modes of execution:

- **SQL-Loop mode** to execute a script infintely on a given schema of a
Expand Down Expand Up @@ -69,7 +65,8 @@ This tool can be used in different infrastructures:
- on any PostgreSQL stand-alone PostgreSQL or PostgreSQL cluster
running in a Kubernetes environment.

This tool can be used in different scenarios:
This tool as different usages, and you probably think of some that I haven't
listed here:

- insert dummy data (mostly randomly if you know about, mostly,
`generate_series()` and `random()` PostgreSQL functions) any DB with the
Expand Down Expand Up @@ -106,6 +103,18 @@ This tool can be used in different scenarios:
write work on a PostgreSQL server (standalone or the primary of a
PostgreSQL cluster with a(some) replica(s).

- as per version 1.1.0, the SQL-Loop mode execution can be limitated to:

- a number of loop exections you define thanks to the `-loops <int64>`
parameter and/or

- a given execution time of your choice you can define thanks to the
`-time duration` parameter, where that duration is expressed with
or without simple or double-quotes like in "10s", 1m30s or '1h40m'

- if both parameters are used at the same time, the SQL-Loop will end
whenever one or the other condition is satisfied

- test failovers, or what happens when a DB is down: pgSimLoad handles those
errors. Give it a try: simply shuting down your PostgreSQL server while it
runs... You'll see it throwing errors, then restarting to load once the
Expand Down
75 changes: 75 additions & 0 deletions doc/03_overview_of_flags_and_parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ All flags are optional and intended to run alone.
| `create` | | **X** | JSON file | Sets the SQL DDL to run once prior main loop |
| `script` | **X** | | SQL text file | Sets the script to run inside the loop |
| `session_parameters` | | **X** | JSON file | Sets special session configuration parameters |
| `loops` | | **X** | integer | Sets the number of loops to execute before exiting |
| `time` | | **X** | duration | Sets the total execution time before exiting |


## Patroni-Watcher mode : parameters

Expand Down Expand Up @@ -236,6 +239,78 @@ pgSimload to have such keyword there. It is also expected that your JSON file
here is valid, like given in the example file given in
`examples/testdb/session_parameters.json`

### **loops** (integer64) [OPTIONAL]

This parameter was added in version 1.1.0.

It allows one to limit the number of times the SQL-Loop will be run.

As an example, passing the `-loops 10` parameter to your `pgSimload` command
line will give you the following (extract of) output:

```code
[...]
Now entering the main loop, executing script "light.sql"
Number of loops will be limited:
10 executions
Script executions succeeded : 10
=========================================================================
Summary
=========================================================================
Script executions succeeded : 10 (9.561 scripts/second)
Total exec time : 1.045s
=========================================================================
```

This parameter can be used in conjuction with the `-time` parameter below.
Whichever is satisfied first will end the SQL-Loop.

### **time** (duration) [OPTIONAL]

This parameter was added in version 1.1.0.

It allows one to limit the execution time of the SQL-Loop.

The value has to be one of "duration", that is expressed with or without
simple or double quotes, so all the following values are valid:

- 10s for ten seconds
- "1m30s" for one minute and thirty seconds
- '1h15m4s" for one hour fifteen minutes and four seconds

Note that GoLang's Time package limits duration units to the following list,
you can use to pass the duration you want:

- "ns" for nanoseconds
- "us" (or "µs") for microseconds
- "ms" for milliseconds
- "s" for seconds
- "m" for minutes
- "h" for hours

I hardly can believe you'd ever want pgSimload to run for days, months, year..
Don't you?

As an example passing the `-time 10s` parameter to your `pgSimload` command
line will give you the following (extract of) output:

```code
Now entering the main loop, executing script "light.sql"
Number of loops will be limited:
"10s" maximum duration
Script executions succeeded : 90
=========================================================================
Summary
=========================================================================
Script executions succeeded : 90 (8.907 scripts/second)
Total exec time : 10.104s
=========================================================================
```

This parameter can be used in conjuction with the `-loops` parameter seen in
the previous paragraph. Whichever is satisfied first will end the SQL-Loop.

## Patroni-Watcher mode flag and parameters

Expand Down
35 changes: 35 additions & 0 deletions doc/04_release_notes.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,40 @@
# Release notes

## Version 1.1.0 (January 20th 2024)

### Major changes

- Added 2 new parameters to pgSimload command line to be used in SQL-Loop
mode:

- `-loops <int64>` will limit the SQL-Loop execution to that exact number
of loops. This can be used to avoid running SQL-Loop endlessly, and/or
in comparisons scenarios when one wants to compare effects of various
configurations parameters, including using different values when a
session parameters files is used (see `session_parameters <JSON.file>`
in docs)

- `-time time.Duration` (where Duration is a duration, without or with
double or sigle quotes, like "10s" or 1m10s or '1h15m30s'...). This option
will limit SQL-Loop execution to that amount of time. It can be used in
various scenarios too

- when both are used at the same time, the SQL-Loop ends when any one of
those conditions is satisfied

- documentation update to describe the new `-loops` and `-time` parameters to
be used in SQL-Loop mode

### Minor changes

- updated Crunchy copyright ranges to include 2024 (patch by @youattd)

- updated `examples/simple` examples and README file

- updated `examples/patroni_monitoring/README.md` doc to mention the

- added scripts in `examples/patroni_monitoring/ha_test_tools`

## Version 1.0.3 (January 15th 2024)

### Major changes
Expand Down
Binary file modified doc/pgSimload.doc.pdf
Binary file not shown.
3 changes: 2 additions & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@ They are all described in the documentation you can read at `doc/README.md`.

Feel free to modify/adapt to your needs.

Please read README.md in any of the subdirectories here, where there's one.
Please read README.md in any of the subdirectories here, when there's one,
writing docs is a long process :'-)
24 changes: 24 additions & 0 deletions examples/patroni_monitoring/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ table, then you'll need also a configuration file for the connexion to the
database. This connexion has to be executed with a superuser in PostgreSQL,
aka `postgres` user (or any superuser in PostgreSQL you defined).

If you want to test your PostgreSQL HA you can use the tools provided see
bottom of this file.


## Usage in localhost/baremetal/VMs...

Expand Down Expand Up @@ -188,3 +191,24 @@ So the `config.json` should then be like:
"ApplicationName" : "pgSimload"
}
```

## Some tools to test HA

So you want to test HA?

On bare metal or VMs, that could be as simple as issuing an `rm -rf` in your
$PGDATA directory, and see what happens.

On Kubernetes, same thing can be done by deleting the StatefulSet of the
primary PostgreSQL pod.

You can achieve that by looking at the `ha_test_tools/kill_on_k8s.sh` bash
script. You may have to edit it to configure it to suit your needs.

Those operations can be done while pgSimload runs in 2 terminals, one in the
SQL-Loop mode and the other one in the Patroni-mode. It will help you
understand (or demo) how your PostgreSQL HA cluster behaves.

There is also a script `ha_test_tools/kill_pgSimload_sessions.sh` that allows
you to kill the PostgreSQL sessions of the pgSimload application so you can
test the fail/retry mechanism built in pgSimload.
58 changes: 58 additions & 0 deletions examples/patroni_monitoring/ha_test_tools/kill_on_k8s.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
#!/bin/bash

#YOUR CONFIG
PRIMARY_POD=$(kubectl -n postgres-operator get pods \
--selector=postgres-operator.crunchydata.com/role=master \
-o jsonpath='{.items[*].metadata.labels.postgres-operator\.crunchydata\.com/instance}')

NAMESPACE="postgres-operator"

usage() {
echo "Usage: $0"
echo ""
echo " Will delete the StatefulSet of the PostgreSQL primary pod"
echo " Adapt the script in #YOUR_CONFIG section to your environment"
echo " kubectl must be present in your PATH"
echo " Current NAMESPACE is set to \"${NAMESPACE}\""
}

delete_primary_pod_sts () {
echo "Deleting sts on ${NAMESPACE} for pod ${PRIMARY_POD}"
kubectl delete sts -n ${NAMESPACE} "${PRIMARY_POD}"
}

check_kubectl_is_present () {
if ! [ -x "$(command -v kubectl)" ]
then
echo "kubebctl could not be found on this system"
echo "install it prior executing this script"
exit 1
fi
}

# check presence of kubectl
check_kubectl_is_present

# WARNING MESSAGE
echo "************************************************************************"
echo "WARNING!"
echo "========"
echo " You're about to delete the follwing StafefulSet of the PG primary pod:"
echo " - namespace \"${NAMESPACE}\""
echo " - pod \"${PRIMARY_POD}\""
echo " This action has no return back, unless you have a working HA in place"
echo "************************************************************************"
echo "Abort this if your namespace is different from the default"
echo ""
read -p "Are you sure you want to continue [y/N]: " SURETHING
SURETHING=`echo ${SURETHING:-N} | tr 'a-z' 'A-Z'`

if [[ "${SURETHING}" == "N" ]]
then
echo "Aborting!"
usage
exit 0
else
delete_primary_pod_sts
fi

Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#!/bin/bash

#YOUR CONFIG
SUPERUSER='postgres'
POSTGRESDB='postgres'
POSTGRESPORT=5432

usage() {
echo "Usage: $0 -h (hostname|ip)"
echo ""
echo " Will pg_terminate_backend() all pids matching pgSimload application."
echo " It assumes your superuser is '${SUPERUSER}' on database '${POSTGRESDB}'"
echo " running on port ${POSTGRESPORT}. If that's not the case, please"
echo " edit this tool ${0} in the section #YOUR CONFIG"
echo " and that you know the superuser password, prefabilty have it in ~/.pgpass"
}

exit_abnormal () {
usage
exit 0
}

execute_kills () {
echo "Killing pgSimload process(es) on ${HOSTNAME}"
psql -h ${HOSTNAME} -U ${SUPERUSER} -p ${POSTGRESPORT} ${SUPERUSERDB} \
-c "select pg_terminate_backend(pids.pid)
from
(select pid
from pg_stat_activity
where application_name='pgSimload') pids"
}

if [ -z "$1" ]; then
exit_abnormal
fi

while getopts ":h:" flag; do
case "${flag}" in
h)
HOSTNAME=${OPTARG}
execute_kills
;;
:)
echo "Error : -${OPTARG} requires an argument."
exit_abnormal
;;
*)
exit_abnormal
;;
esac
done
Loading

0 comments on commit bcff015

Please sign in to comment.