- All notable changes to this project will be documented in this file.
- The format is based on Keep a Changelog.
- We do not follow semantic versioning.
- All changes are solely tracked by date and have a git tag available (from 2021-10-19 onwards):
- Git tags are formatted like
go-starter-YYYY-MM-DD
. See GitHub tags for all available go-starter git tags. - The latest
master
is considered stable and should be periodically merged into our customer projects.
- Git tags are formatted like
- Please follow the update process in I just want to update / upgrade my project!.
- Update to golang:1.22.4-bookworm (requires
./docker-helper.sh --rebuild
) containing the bump from Debianbullseye
(11) tobookworm
(12) - Extended and fixed the password reset handling by a debounce and reuse duration. This can for example be leveraged to mitigate email flooding. The token reuse fixes an existing solution that was not working due to searching for tokens created on minute in the future instead of the past using
models.PasswordResetTokenWhere.CreatedAt.GT(time.Now().Add(time.Minute*1)),
. The new default behaviour is to debounce the password reset by 60 seconds and not to reuse reset tokens:PasswordResetTokenDebounceDuration
/SERVER_AUTH_PASSWORD_RESET_TOKEN_DEBOUNCE_DURATION_SECONDS
: if a password reset token has been created in this duration, no password reset is initialized (default: 60 seconds)PasswordResetTokenReuseDuration
/SERVER_AUTH_PASSWORD_RESET_TOKEN_REUSE_DURATION_SECONDS
: if a password reset token has been created in this duration and is still valid, it is reused instead of re-created (default-value: 0 seconds->no reuse)
- Added test helper to simplify assertion of
httperrors.HTTPError
verifying the http status code and the returned error, example usage:
res := test.PerformRequest(t, s, "POST", "/api/v1/auth/forgot-password/complete", payload, nil)
response := test.RequireHTTPError(t, res, httperrors.ErrNotFoundTokenNotFound)
- Added helpers to get last sent emails from mock transport using
mail := test.GetLastSentMail(t, s.Mailer)
orsentMails := test.GetSentMails(t, s.Mailer)
- Bumped Github Actions
github/codeql-action
from v1 to v3 and set go version ingo.mod
to1.22.4
due to github/codeql#15647
- Fixes the
LogErrorFuncWithRequestInfo
to return the error in order to pass the error to the global error handling mechanism
- Switch from Go 1.21.6 Go 1.21.10 (requires
./docker-helper.sh --rebuild
). - Extended the snapshot and request helper to improve the test experience when snapshotting raw JSON (
.SaveJSON
) and raw bytes like images (.SaveBytes
) - Improved custom replace function in snapshot helper to only redact explicit matches
- Added custom LogErrorFunc for recover middleware to attach request info the recover log messages
- Adjusted order of validation error matching to correctly return list of errors wrapped in single error
- Added vulnerability scanning to dev container (trivy and govulncheck)
- Persist bash history in development container (requires
./docker-helper.sh --rebuild
).- Your commands are now persisted between your development container restarts / rebuilds, making it easier to re-run specific commands you've previously executed (e.g. that one go command you cannot remember).
- Hotfix types.NullDecimal error by downgrading indirect
github.com/ericlagergren/decimal@v0.0.0-20190420051523-6335edbaa640
.- Note that we do not pin it in direct dependencies, as this downgrade is already in SQLBoilers master anyways.
- Migration to Docker Compose V2 (Docker Compose Docs), thx @eklatzer
- Upgrade to IntegreSQL v1.1.0
- Switch from Go 1.20.3 Go 1.21.6 (requires
./docker-helper.sh --rebuild
). - Fix premature optimization in
make swagger
->make swagger-generate
(rmrsync
with--size-only
), thx @eklatzer - Dockerfile deps upgrade:
- Upgrade pgFormatter from v5.3 to v5.5
- Upgrade gotestsum from 1.9.0 to 1.11.0
- Upgrade golangci-lint from 1.52.2 to 1.55.2
- Upgrade watchexec from 1.20.6 to 1.25.1
go.mod
upgrades- Minor: Bump github.com/BurntSushi/toml from v1.2.1 to v1.3.2
- Minor: Bump github.com/davecgh/go-spew from v1.1.1 to v1.1.2-0.20180830191138-d8f796af33cc
- Minor: Bump github.com/gabriel-vasile/mimetype from v1.4.1 to v1.4.3
- Minor: Bump github.com/go-openapi/errors from v0.20.3 to v0.21.0
- Minor: Bump github.com/go-openapi/runtime from v0.25.0 to v0.27.1
- Minor: Bump github.com/go-openapi/strfmt from v0.21.3 to v0.22.0
- Minor: Bump github.com/go-openapi/swag from v0.22.3 to v0.22.9
- Minor: Bump github.com/go-openapi/validate from v0.22.0 to v0.22.6
- Minor: Bump github.com/labstack/echo/v4 from v4.9.1 to v4.11.4
- Minor: Bump github.com/lib/pq from v1.10.7 to v1.10.9
- Minor: Bump github.com/nicksnyder/go-i18n/v2 from v2.2.1 to v2.4.0
- Minor: Bump github.com/pmezard/go-difflib from v1.0.0 to v1.0.1-0.20181226105442-5d4384ee4fb2 (deprecated)
- Minor: Bump github.com/rs/zerolog from v1.28.0 to v1.31.0
- Minor: Bump github.com/rubenv/sql-migrate from v1.2.0 to v1.6.1
- Minor: Bump github.com/spf13/cobra from v1.6.1 to v1.8.0
- Minor: Bump github.com/spf13/viper from v1.14.0 to v1.18.2
- Minor: Bump github.com/stretchr/testify from v1.8.1 to v1.8.4
- Minor: Bump github.com/subosito/gotenv from v1.4.1 to v1.6.0
- Minor: Bump github.com/volatiletech/sqlboiler/v4 from v4.13.0 to v4.16.1
- Minor: Bump github.com/volatiletech/strmangle from v0.0.4 to v0.0.6
- Minor: Bump golang.org/x/crypto from v0.3.0 to v0.18.0
- Minor: Bump golang.org/x/sys from v0.5.0 to v0.16.0
- Minor: Bump golang.org/x/text from v0.7.0 to v0.14.0
- Minor: Bump google.golang.org/api from v0.103.0 to v0.161.0
- Minor: Bump xxxx from yyy to zzz
- Replace: github.com/rogpeppe/go-internal v1.9.0 with golang.org/x/mod v0.14.0
- Switch from Go 1.19.3 to Go 1.20.3 (requires
./docker-helper.sh --rebuild
). - Add new log configuration:
- optional
output
param ofLoggerWithConfig
to redirect the log output - optional caller info switched on with
SERVER_LOGGER_LOG_CALLER
- optional
- Minor: rename unused function parameters to fix linter errors
- Minor: update devcontainer.json syntax to remove deprecation warning
- Minor: add
GetFieldsImplementing
to utils and use it to easier add new fixture fields. go.mod
changes:- Minor: Bump github.com/golangci/golangci-lint from 1.50.1 to 1.52.2
- Minor: Bump golang.org/x/net from 0.2.0 to 0.7.0 (Fixing CVE-2022-41723)
- Switch from Go 1.17.9 to Go 1.19.3 (requires
./docker-helper.sh --rebuild
).- Major: Update base docker image from debian buster to bullseye
- Minor: Bump github.com/darold/pgFormatter from 5.2 to 5.3
- Minor: Bump github.com/gotestyourself/gotestsum from 1.8.0 to 1.9.0
- Minor: Bump github.com/golangci/golangci-lint from 1.45.2 to 1.50.1
- Minor: Bump github.com/uw-labs/lichen from 0.1.5 to 0.1.7
- Minor: Bump github.com/watchexec/watchexec from 1.18.11 to 1.20.6
- Minor: Bump github.com/mikefarah/yq from 4.24.2 to 4.30.5
- Major: Upgrade distroless app image from base-debian10 to base-debian11
- Major: Dockerfile is now build to support amd64 and arm64 architecture
- Improve speed of
make swagger
when dealing with many files in/api
by generating to a docker volume instead of the host filesystem, rsyncing only to changes into/internal/types
. Furthermore split our swagger type generation and validation into two separate make targets, that can run concurrently (requires./docker-helper.sh --rebuild
).- Note that
/app/api/tmp
,/app/tmp
and/app/bin
are now baked by proper docker volumes when using ourdocker-compose.yml
/./docker-helper.sh --up
. You cannot remove these directories directly inside the container (but its contents) and you can also no longer see its files on your host machine directly!
- Note that
- Fix
make check-gen-dirs
false positives hidden files. - Allow to trace/benchmark
Makefile
targets execution by using a custom shell wrapper for make execution. SeeSHELL
and.SHELLFLAGS
withinMakefile
and the customrksh
script in the root working directory. Usage:MAKE_TRACE_TIME=true make <target>
go.mod
changes:- Minor: Bump github.com/BurntSushi/toml from 1.1.0 to 1.2.1
- Minor: Bump github.com/gabriel-vasile/mimetype from 1.4.0 to 1.4.1
- Minor: Bump github.com/go-openapi/errors from 0.20.2 to 0.20.3
- Minor: Bump github.com/go-openapi/runtime from 0.23.3 to 0.25.0
- Minor: Bump github.com/go-openapi/strfmt from 0.21.2 to 0.21.3
- Minor: Bump github.com/go-openapi/swag from 0.21.1 to 0.22.3
- Minor: Bump github.com/go-openapi/validate from 0.21.0 to 0.22.0
- Minor: Bump github.com/labstack/echo/v4 from 4.7.2 to 4.9.1 (Fixing CVE-2022-40083)
- Minor: Bump github.com/lib/pq from 1.10.5 to 1.10.7
- Minor: Bump github.com/nicksnyder/go-i18n/v2 from 2.2.0 to 2.2.1
- Minor: Bump github.com/rogpeppe/go-internal from 1.8.1 to 1.9.0
- Minor: Bump github.com/rs/zerolog from 1.26.1 to 1.28.0
- Minor: Bump github.com/rubenv/sql-migrate from 1.1.1 to 1.2.0
- Minor: Bump github.com/spf13/cobra from 1.4.0 to 1.6.1
- Minor: Bump github.com/spf13/viper from 1.10.1 to 1.14.0
- Minor: Bump github.com/stretchr/testify from 1.7.1 to 1.8.1
- Minor: Bump github.com/subosito/gotenv from 1.2.0 to 1.4.1
- Minor: Bump github.com/volatiletech/sqlboiler/v4 from 4.9.2 to v4.13.0
- Minor: Bump github.com/volatiletech/strmangle from 0.0.2 to 0.0.4 (changes in enum generation might require manual changes, minor changes)
- Minor: Bump golang.org/x/crypto from v0.0.0-20220411220226-7b82a4e95df4 to 0.3.0
- Minor: Bump golang.org/x/sys from v0.0.0-20220412211240-33da011f77ad to 0.2.0
- Minor: Bump golang.org/x/text from 0.3.7 to 0.4.0 (Fixing CVE-2022-32149)
- Minor: Bump google.golang.org/api from 0.74.0 to 0.103.0
- Hotfix: Previously there was a chance of recursive error wrapping within our
internal/api/router/error_handler.go
in combination with*echo.HTTPError
. We currently disable this wrapping (as not used anyways) and will schedule a cleaner update regarding this error augmentation approach.
- Switch from Go 1.17.1 to Go 1.17.9 (requires
./docker-helper.sh --rebuild
). - BREAKING Add
tenv
anderrorlint
linter to our default.golangci.yml
configuration.- We switch from
os.Setenv
tot.Setenv
within our own test code. - NOTE: If you have used
os.Setenv
within your*_test.go
code previously, simply replace those calls byt.Setenv
. - NOTE: The go-starter base code now properly uses
errors.Is
anderrors.As
for comparisons (and%w
wrapping where really needed). For a good overview regarding error handling see Effective Error Handling in Golang. For example, if you receive linting errors, you'll need to change your code like this:- Wrong:
if err == sql.ErrNoRows {
- Valid:
if errors.Is(err, sql.ErrNoRows) {
- Valid:
- Wrong:
if err != sql.ErrConnDone {
- Valid:
if !errors.Is(err, sql.ErrConnDone) {
- Valid:
- Wrong:
gErr := err.(*googleapi.Error)
, Valid:var gErr *googleapi.Error
ok := errors.As(err, &gErr)
- Wrong:
- We switch from
Dockerfile
development stage changes (requires./docker-helper.sh --rebuild
):- Bump golang base image from
golang:1.17.1-buster
togolang:1.17.8-buster
. - Bump pgFormatter from v5.0 to v5.2
- Bump golangci-lint from v1.42.1 to v1.45.2
- Bump lichen from v0.1.4 to v0.1.5
- Bump watchexec from v1.17.0 to v1.18.11 (+ switch from gnu to musl)
- Bump yq from v4.16.2 to v4.24.2
- Bump gotestsum from v1.7.0 to v1.8.0
- Adds tmux (debian apt managed)
- Bump golang base image from
go.mod
changes:- Major: Bump
github.com/rubenv/sql-migrate
from v0.0.0-20210614095031-55d5740dbbcc to v1.1.1 (though this should not lead to any major changes) - Minor: Bump github.com/volatiletech/sqlboiler/v4 from 4.6.0 to v4.9.2 (your generated model might slightly change, minor changes).
- Note that v5 will prefer wrapping errors (e.g.
sql.ErrNoRows
) to retain the stack trace, thus it's about time for us to start to enforce propererrors.Is
checks in our codebase (see above).
- Note that v5 will prefer wrapping errors (e.g.
- Minor: #178: Bump github.com/labstack/echo/v4 from 4.6.1 to 4.7.2 (support for HEAD method query params binding, minor changes).
- Minor: #160: Bump github.com/rs/zerolog from 1.25.0 to 1.26.1 (minor changes).
- Minor: #179: Bump github.com/nicksnyder/go-i18n/v2 from 2.1.2 to 2.2.0 (minor changes).
- Minor: Bump
github.com/gabriel-vasile/mimetype
from v1.3.1 to v1.4.0 - Minor: Bump
github.com/go-openapi/runtime
from v0.22.0 to v0.23.3 - Patch: Bump
github.com/go-openapi/strfmt
from v0.21.1 to v0.21.2 - Patch: Bump
github.com/go-openapi/validate
from v0.20.3 to v0.21.0 - Patch: Bump
github.com/lib/pq
from v1.10.3 to v1.10.5 - Patch: Bump
github.com/rogpeppe/go-internal
from v1.8.0 to v1.8.1 - Patch: Bump
github.com/stretchr/testify
from v1.7.0 to v1.7.1 - Patch: Bump
github.com/volatiletech/strmangle
from v0.0.1 to v0.0.2 - Minor: Bump
google.golang.org/api
from v0.63.0 to v0.74.0 - Minor: Bump
github.com/BurntSushi/toml
from v1.0.0 to v1.1.0 - Bump
golang.org/x/crypto
from v0.0.0-20211215165025-cf75a172585e to v0.0.0-20220411220226-7b82a4e95df4 - Bump
golang.org/x/sys
from v0.0.0-20211210111614-af8b64212486 to v0.0.0-20220412211240-33da011f77ad
- Major: Bump
- We now support overriding
ENV
variables during local development through a.env.local
dotenv file.- This does not require a development container restart.
- We override the env within the app process through
config.DefaultServiceConfigFromEnv()
, so this does not mess with the actual container ENV. - See
.env.local.sample
for further instructions to use this. - Note that
.env.local
is NEVER automatically applied during test runs. If you really need that, use the specializedtest.DotEnvLoadLocalOrSkipTest
helper before loading up your server within that very test! This ensures that this test is automatically skipped if the.env.local
file is no longer available.
- VSCode windows closes now explicitly stop Docker containers via
shutdownAction: "stopCompose"
within.devcontainer.json
.- Use
./docker-helper --halt
or otherdocker
ordocker-compose
management commands to do this explicitly instead.
- Use
- Drone CI specific (minor): Fix multiline ENV variables were messing up our
.hostenv
fordocker run
command testing of the final image.
- Merged #165: Allow use of db.join* methods more than once, thx danut007ro.
- Merged #169: Switch to standalone cobra-cli dependency, thx liggitt (requires
./docker-helper.sh --rebuild
).github.com/spf13/cobra@v1.4.0
split intocobra
(the lib) andgithub.com/spf13/cobra-cli
(the generator / scaffolding tool)- We'll now depend on
cobra-cli
directly in ourDockerfile
, while the corecobra
dependency stays unchanged within ourgo.mod
. - Bumps
github.com/spf13/cobra
from v1.3.0 to v1.4.0
- Fixed
test.ApplyMigrations
when combined with the import SQL dump mechanics in the testing context.- Previously, we did still use the default sql-migrate
gorp_migrations
table to track applied migrations in our test databases, not our typicalmigrations
table used everywhere else. - This especially lead to problems when importing (production / live) SQL dumps via
test.WithTestDatabaseFromDump*
,test.WithTestServerFromDump*
ortest.WithTestServerConfigurableFromDump
as our implementation tried to apply all migrations every time, regardless if a partial migration set was already applied previously (as the already applied migrations were not tracked within themigrations
table (but withingorp_migrations
) we did not notice). - We now initialize this pipeline correctly in the test context (similar to our usage within
cmd/db_migrate.go
orapp db migrate
) and explicitly set these globals throughconfig.DatabaseMigrationTable
andconfig.DatabaseMigrationFolder
. - If you encounter problems after the upgrade, please execute
make sql-drop-all
in your local environment to reset the IntegreSQL test databases, then runmake sql-reset && make sql-spec-reset && make sql-spec-migrate && make all
to rebuild and test.
- Previously, we did still use the default sql-migrate
- BREAKING Username format change in auth handlers
- Added the
util.ToUsernameFormat
helper function, which will lowercase and trim whitespaces. We use it to format usernames in the login, register, and forgot-password handlers.- This prevents user duplication (e.g. two accounts registered with the same email address with different casing) and
- cases where users would inadvertently register with specific casing or a trailing whitespace after their username, and subsequently struggle to log into their account.
- This effectively locks existing users whose username contains uppercase characters and/or whitespaces out of their accounts.
- Before rolling out this change, check whether any existing users are affected and migrate their usernames to a format that is compatible with this change.
- Be aware that this may cause conflicts in regard to the uniqueness constraint of usernames and therefore need to be resolved manually, which is why we are not including a database migration to automatically migrate existing usernames to the new format.
- For more information and a possible manual database migration flow please see this special WIKI page: https://github.com/allaboutapps/go-starter/wiki/2022-02-28
- Added the
- Changed order of make targets in the
make swagger
pipeline.make swagger-lint-ref-siblings
will now run aftermake swagger-concat
, always linting the current version of our swagger file. This helps avoid errors regarding an invalidswagger.yml
when resolving merge conflicts as those are often resolved by runningmake swagger
and generating a freshswagger.yml
.
- Upgrades to go-swagger from to v0.26.1 to v0.29.0 (development stage only, requires
./docker-helper.sh --rebuild
). Includes the followinggo.mod
upgrades:- github.com/go-openapi/runtime from v0.19.31 to v0.22.0
- github.com/go-openapi/strfmt from v0.20.2 to v0.21.1
- github.com/go-openapi/validate from v0.20.2 to v0.20.3
- github.com/go-openapi/errors from v0.20.1 to v0.20.2
- github.com/go-openapi/swag from v0.19.15 to v0.21.1
- Adds
yq
(yq: a lightweight and portable command-line YAML processor) to ourDockerfile
(development stage only, requires./docker-helper.sh --rebuild
). - Adds
make swagger-lint-ref-siblings
which is now executed as part of themake build
(andmake swagger
) pipeline.- Any sibling elements of a Swagger
$ref
are ignored. - We have seen several misuses of
$ref
in our projects causing weird merge/flatten behaviors, thus we now lint for this case explicitly. - Having
$ref
and sibling elements (e.g.required
,example
, ...) is unsupported by OpenAPI v2: $ref and Sibling Elements itself and the JSON Reference specification itself. - To mitigate these errors, either expand the referenced element (fully remove
$ref
) or create a new element including your custom siblings elements and$ref
this new one.
- Any sibling elements of a Swagger
- Fix schema visualization generation guide in
docs/schemacrawler/README.md
- Add i18n service wrapping
go-i18n
package by nicksnyder.- Allows parsing of Accept-Language header and language string.
- Support for templating using go templating language in message values.
- Support for CLDR plural keys
- Added environment variables to configure i18n service
SERVER_I18N_DEFAULT_LANGUAGE
- set default language for i18n serviceSERVER_I18N_BUNDLE_DIR_ABS
- set directory of i81n messages, available languages are automatically configured by the files present in the folder
- The
integresql
service previously bound its port (5000
) to the host machine. As this conflicts with newer macOS releases and is not necessary for the development workflow, the port is now only exposed to the linked services.
- Fixes minor
Makefile
typos. - New go-starter releases are now git tagged (starting from the previous release
go-starter-2021-10-19
onwards). See FAQ: What's the process of a new go-starter release? - You may now specify a specific tag/branch/commit from the upstream go-starter project while running
make git-fetch-go-starter
,make git-compare-go-starter
andmake git-merge-go-starter
. This will especially come in handy if you want to do a multi-phased merge (for projects that haven't been updated in a long time):- Merge with the latest:
make git-merge-go-starter
- Merge with a specific tag, e.g. the tag
go-starter-2021-10-19
:GIT_GO_STARTER_TARGET=go-starter-2021-10-19 make git-merge-go-starter
- Merge with a specific branch, e.g. the branch
mr/housekeeping
:GIT_GO_STARTER_TARGET=go-starter/mr/housekeeping make git-merge-go-starter
(heads up! it'sgo-starter/<branchname>
) - Merge with a specific commit, e.g. the commit
e85bedb94c3562602bc23d2bfd09fca3b13d1e02
:GIT_GO_STARTER_TARGET=e85bedb94c3562602bc23d2bfd09fca3b13d1e02 make git-merge-go-starter
- Merge with the latest:
- The primary GitHub Action pipeline
.github/workflows/build-test.yml
has been synced to include most validation tasks from our internal.drone.yml
pipeline. Furthermore:- Avoid
Build & Test
GitHub Action running twice (onpush
and onpull_request
). - Add trivy scan to our base Build & Test pipeline (as we know also build and test the
app
target docker image). - Our GitHub Action pipeline will no longer attempt to cache the previously built Docker images by other pipelines, as extracting/restoring from cache (docker buildx) typically takes longer than fully rebuilding the whole image. We will reinvestigate caching mechanisms in the future if GitHub Actions provides a speedier and official integration for Docker images.
- Avoid
- BREAKING Upgrades to Go 1.17.1
golang:1.17.1-buster
- Switch to
//go:build <tag>
from// +build <tag>
. - Migrates
go.mod
viago mod tidy -go=1.17
(pruned module graphs). - Do the following to upgrade:
make git-merge-go-starter
./docker-helper --rebuild
- Manually remove the new second
require
block (with all the// indirect
modules) within yourgo.mod
- Execute
go mod tidy -go=1.17
once so the secondrequire
block appears again. - Find
// +build <tag>
and replace it with//go:build <tag>
. make all
.- Recheck your
go.mod
that the newly added// indirect
transitive dependencies are the proper version as you were previously using (e.g. via the output frommake get-licenses
andmake get-embedded-modules
). Feel free to move any// indirect
tagged dependencies in your firstrequire
block to the second block. This is where they should live.
- Switch to
- BREAKING You now need to take special care when it comes to parsing semicolons (
;
) in query strings vianet/url
andnet/http
from Go >1.17!- Anything before the semicolon will now be stripped. e.g.
example?a=1;b=2&c=3
would have returnedmap[a:[1] b:[2] c:[3]]
, while now it returnsmap[c:[3]]
- See Go 1.17 URL query parsing.
- You may need to manually migrate your handlers/tests regarding this new default handling.
- Anything before the semicolon will now be stripped. e.g.
- Added
make test-update-golden
for easily refreshing all golden files / snapshot tests (y + ENTER
confirmation). - Upgrades golangci-lint from
v1.41.1
tov1.42.1
(for referencev1.42.0
). - Bump github.com/go-openapi/strfmt from 0.20.1 to 0.20.2
- Bump github.com/go-openapi/errors from 0.20.0 to 0.20.1
- Bump github.com/go-openapi/runtime from 0.19.29 to 0.19.31
- Bump github.com/rs/zerolog from 1.23.0 to 1.25.0
- Bump google.golang.org/api from 0.52.0 to 0.57.0
- Bump github.com/lib/pq from v1.10.2 to v1.10.3
- Bump github.com/spf13/viper from 1.8.1 to v1.9.0
- Bump github.com/labstack/echo from 4.5.0 to v4.6.1
- Update golang.org/x/crypto and golang.org/x/sys
- Hotfix: We will pin the
Dockerfile
development and builder stage togolang:1.16.7-buster
(+-buster
) for now, as currently the new debian bullseye release within the go official docker images breaks some tooling. The upgrade to debian bullseye and Go 1.17 will happensimultaneouslyseparately within go-starter in the following weeks.
- remove ioutil (https://golang.org/doc/go1.16#ioutil)
- Bump golang from 1.16.6 to 1.16.7 (requires
./docker-helper.sh --rebuild
). - Adds
util.GetEnvAsStringArrTrimmed
and minorutil
test coverage upgrades.
README.md
badges for go-starter.- Fix some misspellings of English words within
internal/test/*.go
comments. - Upgrades
- Bump
github.com/labstack/echo/v4
from 4.4.0 to 4.5.0:- Switch from
github.com/dgrijalva/jwt-go
togithub.com/golang-jwt/jwt
to mitigate CVE-2020-26160. - Note that it might take some time until the former dep fully leaves our dependency graph, as it is also a transitive dependency of various versions of
github.com/spf13/viper
. - However, even though this functionality was never used by go-starter, this change fixes an important part: The original
github.com/dgrijalva/jwt-go
is no longer included in the finalapp
binary, it is fully replaced bygithub.com/golang-jwt/jwt
. - Our
.trivyignore
still excludes CVE-2020-26160 as trivy cannot skip checking transitive dependencies. - Breaking: If you have actually directly depended upon
github.com/dgrijalva/jwt-go
, please switch togithub.com/golang-jwt/jwt
via the following command:find -type f -name "*.go" -exec sed -i "s/dgrijalva\/jwt-go/golang-jwt\/jwt/g" {} \;
- Switch from
- Bump
- Upgrades:
- Bump golang from 1.16.5 to 1.16.6
- Bump github.com/labstack/echo/v4 from 4.3.0 to 4.4.0 (adds
binder.BindHeaders
support, not affecting our goswaggerruntime.Validatable
bind helpers) - Bump github.com/gabriel-vasile/mimetype from 1.3.0 to 1.3.1
- Bump github.com/spf13/cobra from 1.1.3 to 1.2.1 (and see all the big completion upgrades in 1.2.0)
- Bump google.golang.org/api from 0.49.0 to 0.52.0
- Bump gotestsum to 1.7.0 (adds handy keybindings while you are in
make watch-tests
mode, see While in watch mode, pressing some keys will perform an action) - Bump watchexec to 1.17.0
- Bump golang.org/x/crypto to
v0.0.0-20210711020723-a769d52b0f97
- Fixed
Makefile
has disregardedpipefail
s in executed targets (e.g.make sql-spec-migrate
previously returned exit code0
even if there were migration errors as its output was piped internally). We now set-cEeuo pipefail
for make's shell args, preventing these issues.
- BREAKING Switched from
golint
torevive
golint
is deprecated.revive
is considered to be a drop-in replacement forgolint
, however this change still might lead to breaking changes in your codebase.
- BREAKING
make lint
no longer uses--fast
when callinggolangci-lint
- Up until now,
make lint
also rangolangci-lint
using the--fast
flag to remain consistent with the linting performed by VSCode automatically. - As running only fast linters in both steps meant skipping quite a few validations (only 4/13 enabled linters are actually active), a decision has been made to break consistency between the two lint-steps and perform "full" linting during the build pipeline.
- This change could potentially bring up additional warnings and thus fail your build until fixed.
- Up until now,
- BREAKING
gosec
is now also applied to test packages- All linters are now applied to every source code file in this project, removing the previous exclusion of
gosec
from test files/packages - As
gosec
might (incorrectly) detect some hardcoded credentials in your tests (variable names such aspasswordResetLink
get flagged), this change might require some fixes after merging.
- All linters are now applied to every source code file in this project, removing the previous exclusion of
- Extended auth middleware to allow for multiple auth token sources
- Default token validator uses access token table, maintaining previous behavior without any changes required.
- Token validator can be changed to e.g. use separate API keys for specific endpoints, allowing for more flexibility if so desired.
- Changed
util.LogFromContext
to always return a valid logger- Helper no longer returns a disabled logger if context provided did not have an associated logger set (e.g. by middleware). If you still need to disable the logger for a certain context/function, use
util.DisableLogger(ctx, true)
to force-disable it. - Added request ID to context in logger middleware.
- Helper no longer returns a disabled logger if context provided did not have an associated logger set (e.g. by middleware). If you still need to disable the logger for a certain context/function, use
- Extended DB query helpers
- Fixed TSQuery escaping, should now properly handle all type of user input.
- Implemented helper for JSONB queries (see
ExampleWhereJSON
for implementation details). - Added
LeftOuterJoin
helper, similar to already existingLeftJoin
variants. - Managed transactions (via
WithTransaction
) can now have their options configured viaWithConfiguredTransaction
. - Added util to combine query mods with
OR
expression.
- Implemented middleware for parsing
Cache-Control
header- Allows for cache handling in relevant services, parsed directive is stored in request context.
- New middleware is enabled by default, can be disabled via env var (
SERVER_ECHO_ENABLE_CACHE_CONTROL_MIDDLEWARE
).
- Added extra misc. helpers
- Extra helpers for slice handling and generating random strings from a given character set have been included (
util.ContainsAllString
,util.UniqueString
,util.GenerateRandomString
). - Added util to check whether current execution runs inside a test environment (
util.RunningInTest
).
- Extra helpers for slice handling and generating random strings from a given character set have been included (
- Test and snapshot util improvements
- Added
snapshoter.SaveU
as a shorthand for updating a single test - Implemented
GenericArrayPayload
with respective request helpers for array payloads in tests - Added VScode launch task for updating all snapshots in a single test file
- Added
- We now directly bake the
gsdev
cli "bridge" (it actually just runsgo run -tags scripts /app/scripts/main.go "$@"
) into thedevelopment
stage of ourDockerfile
and create it at/usr/bin/gsdev
(requires./docker-helper.sh --rebuild
).gsdev
was previously symlinked to/app/bin
from/app/scripts/gsdev
(within the projects' workspace) andchmod +x
via theMakefile
duringinit
.- However this lead to problems with WSL2 VSCode related development setups (always dirty git workspaces as WSL2 tries to prevent
+x
flags). - BREAKING encountered at 2021-06-30: Upgrading your project via
make git-merge-go-starter
if you already have installed our previousgsdev
approach from 2021-06-22 may require additional steps:- It might be necessary to unlink the current
gsdev
symlink residing at/app/bin/gsdev
before merging up (as this symlinked file will no longer exist)! - Do this by issuing
rm -f /app/bin/gsdev
which will remove the symlink which pointed to the previous (now gone bash script) at/app/scripts/gsdev
. - It might also be handy to install the newer variant directly into your container (without requiring a image rebuild). Do this by:
sudo su
to become root in the container,- issuing the following command:
printf '#!/bin/bash\nset -Eeo pipefail\ncd /app && go run -tags scripts ./scripts/main.go "$@"' > /usr/bin/gsdev && chmod 755 /usr/bin/gsdev
(in sync with what we do in ourDockerfile
) and [CTRL + c]
to return to being thedevelopment
user within your container.
- It might be necessary to unlink the current
- Introduces GitHub Actions docker layer caching via docker buildx. For details see
.github/workflows/build-test.yml
. - Upgrades:
- Bump golang from 1.16.4 to 1.16.5
- golangci-lint@v1.41.1
- Bump github.com/rs/zerolog from 1.22.0 to 1.23.0
- Bump github.com/go-openapi/runtime from 0.19.28 to 0.19.29
- Bump github.com/volatiletech/sqlboiler/v4 from 4.5.0 to 4.6.0
- Bump github.com/rubenv/sql-migrate v0.0.0-20210408115534-a32ed26c37ea to v0.0.0-20210614095031-55d5740dbbcc
- Bump github.com/spf13/viper v1.7.1 to v1.8.0
- Bump golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a to v0.0.0-20210616213533-5ff15b29337e
- Bump golang.org/x/sys v0.0.0-20210525143221-35b2ab0089ea to v0.0.0-20210616094352-59db8d763f22
- Bump google.golang.org/api v0.47.0 to v0.49.0
- Fixes linting within
/scripts/**/*.go
, now activated by default.
- Development scripts are no longer called via
go run [script]
but viagsdev
:- The
gsdev
cli is our new entrypoint for development workflow specific scripts, these scripts are not available in the finalapp
binary. - All previous
go run
scripts have been moved to their respective/scripts/cmd
cli entrypoint + internal implementation within/scripts/internal/**
. - Please use
gsdev --help
to get an overview of available development specific commands. gsdev
relys on a tiny helper bash scriptscripts/gsdev
which gets symlinked to/app/bin
onmake init
.- Use
make test-scripts
to run tests regarding these internal scripts within/scripts/**/*_test.go
. - We now enforce that all
/scripts/**/*.go
files set the// +build scripts
build tag. We do this to ensure these files are not directly depended upon from the actualapp
source-code within/internal
.
- The
- VSCode's
.devcontainer/devcontainer.json
now defines that the go tooling must use thescripts
build tag for its IntelliSense. This is neccessary to still get proper code-completion when modifying resources at/scripts/**/*.go
. You may need to reattach VSCode and/or run./docker-helper.sh --rebuild
.
- Scaffolding tool to quickly generate generic CRUD endpoint stubs. Usage:
gsdev scaffold [resource name] [flags]
, also seegsdev scaffold --help
.
- Scans for CVE-2020-26160 also match for our final
app
binary, however, we do not usegithub.com/dgrijalva/jwt-go
as part of our auth logic. This dependency is mostly here because of child dependencies, that yet need to upgrade to>=v4.0.0
. Therefore, we currently disable this CVE for scans in this project (via.trivyignore
). - Upgrades
Dockerfile
:watchexec@v1.16.1
,lichen@v0.1.4
(requires./docker-helper.sh --rebuild
).
- Upgraded
Dockerfile
togolang:1.16.4
,gotestsum@v1.6.4
,golangci-lint@v1.40.1
,watchexec@v1.16.0
(requires./docker-helper.sh --rebuild
). - Upgraded
go.mod
:- github.com/labstack/echo/v4@v4.3.0
- github.com/lib/pq@v1.10.2
- github.com/gabriel-vasile/mimetype@v1.3.0
github.com/go-openapi/runtime@v0.19.28
- github.com/rs/zerolog@v1.22.0
github.com/rubenv/sql-migrate@v0.0.0-20210408115534-a32ed26c37ea
golang.org/x/crypto@v0.0.0-20210513164829-c07d793c2f9a
golang.org/x/sys@v0.0.0-20210514084401-e8d321eab015
- google.golang.org/api@v0.46.0
- GitHub Actions:
- Pin to
actions/checkout@v2.3.4
. - Remove unnecessary
git checkout HEAD^2
in CodeQL step (Code Scanning recommends analyzing the merge commit for best results). - Limit trivy and codeQL actions to
push
againstmaster
andpull_request
againstmaster
to overcome read-only access workflow errors.
- Pin to
- Adds
test.WithTestDatabaseFromDump*
,test.WithTestServerFromDump
methods for writing tests based on a database dump file that needs to be imported first:- We dynamically setup IntegreSQL pools for all combinations passed through a
test.DatabaseDumpConfig{}
object:DumpFile string
is required, absolute path to dump fileApplyMigrations bool
optional, defaultfalse
, automigrate after installing the dumpApplyTestFixtures bool
optional, defaultfalse
, import fixtures after (migrating) installing the dump
test.ApplyDump(ctx context.Context, t *testing.T, db *sql.DB, dumpFile string) error
may be used to apply a dump to an existing database connection.- As we have dedicated IntegreSQL pools for each combination, testing performance should be on par with the default IntegreSQL database pool.
- We dynamically setup IntegreSQL pools for all combinations passed through a
- Adds
test.WithTestDatabaseEmpty*
methods for writing tests based on an empty database (also a dedicated IntegreSQL pool). - Adds context aware
test.WithTest*Context
methods reusing the providedcontext.Context
(first arg). - Adds
make sql-dump
command to easily create a dump of the localdevelopment
database to/app/dumps/development_YYYY-MM-DD-hh-mm-ss.sql
(.gitignored).
test.ApplyMigrations(t *testing.T, db *sql.DB) (countMigrations int, err error)
is now public (e.g. for usage withtest.WithTestDatabaseEmpty*
ortest.WithTestDatabaseFromDump*
)test.ApplyTestFixtures(ctx context.Context, t *testing.T, db *sql.DB) (countFixtures int, err error)
is now public (e.g. for usage withtest.WithTestDatabaseEmpty*
ortest.WithTestDatabaseFromDump*
)internal/test/test_database_test.go
and/app/internal/test/test_server_test.go
were massively refactored to allow for better extensibility later on (non breaking, all method signatures are backward-compatible).
- Adds echo
NoCache
middleware: Usemiddleware.NoCache()
andmiddleware.NoCacheWithConfig(Skipper)
to explicitly force browsers to never cache calls to these handlers/groups.
/swagger.yml
and/-/*
now explicity set no-cache headers by default, forcing browsers to re-execute calls each and every time.- Upgrade watchexec@v1.15.0 (requires
./docker-helper.sh --rebuild
).
- Live-Reload for our swagger-ui is now available out of the box:
- allaboutapps/browser-sync acts as proxy at localhost:8081.
- Requires
./docker-helper.sh --up
. - Best used in combination with
make watch-swagger
(still refreshesmake all
ormake swagger
of course).
- Upgrades to swaggerapi/swagger-ui:v3.46.0 from swaggerapi/swagger-ui:v3.28.0
- Upgrades to github.com/labstack/echo@v4.2.2
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2
- Upgrades to google.golang.org/api@v0.44.0
- Moved
/api/main.yml
to/api/config/main.yml
to overcome path resolve issues (../definitions
) with the VSCode 42crunch.vscode-openapi extension (auto-included in our devContainer) and our go-swagger concat behaviour. - Updated api/README.md information about
/api/swagger.yml
generation logic and changedmake swagger-concat
accordingly
- Bump golang from v1.16.2 to v1.16.3 (requires
./docker-helper.sh --rebuild
).
- Bump golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2
- Bump golang.org/x/sys@v0.0.0-20210331175145-43e1dd70ce54
- Bump github.com/go-openapi/swag@v0.19.15
- Bump github.com/go-openapi/strfmt@v0.20.1
- Bump github.com/gotestyourself/gotestsum@v1.6.3 (requires
./docker-helper.sh --rebuild
).
- Bump golangci-lint@v1.39.0 (requires
./docker-helper.sh --rebuild
).
- Bump github.com/rs/zerolog from 1.20.0 to 1.21.0
- Bump google.golang.org/api from 0.42.0 to 0.43.0
- We no longer do explicit calls to
t.Parallel()
in our go-starter tests (except autogenerated code). For the reasons why see FAQ: Should I uset.Parallel()
in my tests?. - Switched to github.com/uw-labs/lichen for getting license information of embedded dependencies in our final
./bin/app
binary. - The following make targets are no longer flagged as
(opt)
and thus move into the mainmake help
target (usemake help-all
to see all targets):make lint
: Runs golangci-lint and make check-*.make go-test-print-slowest
: Print slowest running tests (must be done after running tests).make get-licenses
: Prints licenses of embedded modules in the compiled bin/app.make get-embedded-modules
: Prints embedded modules in the compiled bin/app.make clean
: Cleans ./tmp and ./api/tmp folder.make get-module-name
: Prints current go module-name (pipeable).
make check-gen-dirs
now ignores.DS_Store
within/internal/models/**/*
and/internal/types/**/*
and echo an errors detailing what happened.- Upgrade to
github.com/go-openapi/runtime@v0.19.27
make all
no longer executesmake info
as part of its targets chain.- It's very common to use
make all
multiple times per day during development and thats fine! However, the output ofmake info
is typically ignored by our engineers (if they explicitly want this information, they usemake info
). Somake all
was just too spammy in it's previous form. make info
does network calls and typically takes around 5sec to execute. This slowdown is not acceptable when runningmake all
, especially if the information it provides isn't used anyways.- Thus: Just trigger
make info
manually if you need the information of the[spec DB]
structure, current[handlers]
and[go.mod]
information. Furthermore you may also visittmp/.info-db
,tmp/.info-handlers
andtmp/.info-go
after triggeringmake info
as we store this information there after a run.
- It's very common to use
- Upgrades
go.mod
:github.com/volatiletech/sqlboiler/v4@v4.5.0
github.com/rogpeppe/go-internal@v1.8.0
golang.org/x/crypto@v0.0.0-20210314154223-e6e6c4f2bb5b
golang.org/x/sys@v0.0.0-20210314195730-07df6a141424
golang.org/x/sys@v0.0.0-20210315160823-c6e025ad8005
google.golang.org/api@v0.42.0
make help
no longer reports(opt)
flagged targets, usemake help-all
instead.make tools
now executesgo install {}
in parallelmake info
now fetches information in parallel- Seeding: Switch to
db|dbUtil.WithTransaction
instead of manually managing the db transaction. Note: We will enforce usingWithTransaction
instead of manually managing the life-cycle of db transactions through a custom linter in an upcoming change. It's way safer and manually managing db transactions only makes sense in very very special cases (where you will be able to opt-out via linter excludes). Also see What'sWithTransaction
, shouldn't I usedb.BeginTx
,db.Commit
, anddb.Rollback
?.
- The correct implementation of
(util|scripts).GetProjectRootDir() string
now gets automatically selected based on thescripts
build tag.- We currently have 2 different
GetProjectRootDir()
implementations and each one is useful on its own:util.GetProjectRootDir()
gets used whileapp
orgo test
runs and resolves in the following way: usePROJECT_ROOT_DIR
(if set), else default to the resolved path to the executable unless we can't resolve that, then panic!scripts.GetProjectRootDir()
gets used while generation time (make go-generate
) and resolves in the following way: usePROJECT_ROOT_DIR
(if set), otherwise default to/app
(baked, as we can assume we are in thedevelopment
container).
/internal/util/(get_project_root_dir.go|get_project_root_dir_scripts.go)
is now introduced to automatically switch to the proper implementation based on the// +build !scripts
or// +build scripts
build tag, thus it's now consistent to importutil.GetProjectRootDir()
, especially while handler generation time (make go-generate
).
- We currently have 2 different
- Upgrades to
golang@v1.16.2
(use./docker-helper.sh --rebuild
). - Silence resolve of
GO_MODULE_NAME
ifgo
was not found in path (typically host env related).
make build
(make go-build
) now setsinternal/config.ModuleName
,internal/config.Commit
andinternal/config.BuildDate
via-ldflags
./-/version
(mgmt key auth) endpoint is now available, prints the same asapp -v
.app -v
is now available and prints out buildDate and commit. Sample:
app -v
github.com/danielmoisa/trip-planner @ 19c4cdd0da151df432cd5ab33c35c8987b594cac (2021-03-11T15:42:27+00:00)
- Upgrades to
golang@v1.16.1
(use./docker-helper.sh --rebuild
). - Updates
google.golang.org/api@v0.41.0
,github.com/gabriel-vasile/mimetype@v1.2.0
(new supported formats),golang.org/x/sys
- Removed
**/.git
from.dockerignore
(builder
stage) as we want the local git repo available while runningmake go-build
. app --help
now prominently includes the module name of the project.- Prominently recommend
make force-module-name
after runningmake git-merge-go-starter
to fix all import paths.
- Introduces
CHANGELOG.md
make git-merge-go-starter
now uses--allow-unrelated-histories
by default.README.md
and FAQ now mention that it's recommended to executemake git-merge-go-starter
during project setup (especially for single commit generated from template project project setups).- See FAQ: I want to compare or update my project/fork to the latest go-starter master.
- Various typos in
README.md
andMakefile
. - Upgrade to
golangci-lint@v1.38.0
allaboutapps/nullable
is now included by default. See #58, FAQ: I need an optional Swagger payload property that is nullable!
- Upgrade to
labstack/echo@v4.2.1
,lib/pq@v1.10.0
util.BindAndValidate
is now marked as deprecated aslabstack/echo@v4.2.0
exposes a more granular binding through itsDefaultBinder
.
- The more specialized variants
util.BindAndValidatePathAndQueryParams
andutil.BindAndValidateBody
are now available. See/internal/util/http.go
.
golang@v1.16.0
labstack/echo@v4.2.0
- Upgrades to
pgFormatter@v5.0.0
+ forces VSCode to use that version within the devcontainer through it's extension.
golang@v1.15.8
,go-swagger@v0.26.1
- Dockerfile updates:
- golang@1.15.7
- apt add icu-devtools (VSCode live sharing)
- gotestsum@1.6.1
- golangci-lint@v1.36.0
- goswagger@v0.26.0
- go.mod:
- sqlboiler@4.4.0
- swag@0.19.3
- strfmt@0.20.0
- testify@1.7.0
- go-openapi/runtime@v0.19.26
- go-openapi/swag@v0.19.13
- go-openapi/validate@v0.20.1
- jordan-wright/email
- rogpeppe/go-internal@v1.7.0
- golang.org/x/crypto
- golang.org/x/sys
- google.golang.org/api@v0.38.0
- disabled goswagger generate server flag
--keep-spec-order
as relative resolution of its temporal created yml file is broken - see go-swagger/go-swagger#2216
make watch-swagger
andmake watch-sql
- sqlboiler@4.3.0
make watch-tests
: Watches .go files and runs package tests on modifications.
pprof
handlers, see FAQ: I need to (remotely) pprof my running service!
make git-merge-go-starter
, see FAQ: I want to compare or update my project/fork to the latest go-starter master.
app probe readiness
andapp probe liveness
sub-commands./-/ready
and/-/healthy
handlers.
- Force VSCode to use our installed version of golang-cilint
- All
*.go
files in/scripts
now use the build tagscripts
so we can ensure they are not compiled into the finalapp
binary.
go.not
file to ensure certain generation- / test-only dependencies don't end up in the finalapp
binary. Automatically checked thoughmake
(sub-targetmake check-embedded-modules-go-not
).
- Switch to
distroless
as final app stage, see FAQ: Should I use distroless/base or debian:buster-slim in the Dockerfile app stage?