Hi! Updates on rolling new versions?

[shawncatz]

Hi, I really love this lib. Use it in a bunch of projects and I'm curious if there's any updates on rolling new releases?

I've been digging into some older code and cleaning it up and noticed that there hasn't been a new tag since 2019. I saw in some of the Issue conversations that you were working on getting things back on track, just wondering if there's a timeline for that.

I am more than happy to help contribute if you need more bodies to get things over the line :)

Edit: Apologies, I meant to add more context but I'm trying to do too many things at once... I was going to ask if there's another way to get newer versions of the lib without pining to master or a specific sha. Thanks again!

[philippgille]

Hello 👋 , Thank you for using and liking the project!

Yes the recent updates and improvements are all in preparation for a new release! I don't have a specific timeline, but several open tasks I'd like to include:

• We recently changed CI to not wait 10s for a storage server to be up, but use Docker health checks to reduce the CI duration. But there seem to be issues and most take 30s (probably some timeout in Docker health checks, it's unclear from the docs). I'd like to investigate and fix this, otherwise CI takes longer instead of shorter.
• Some store tests are skipped because the connection between client and server doesn't work. For example MongoDB, probably due to the dependency not being maintained and the server having changed its protocol slightly, or so. I'd like to investigate and fix this.
• Some store's dependencies have a new major version, like BadgerDB. I'm thinking of updating these for the next version. (Initially I only wanted to bump minor versions to get the next gokv release out with as few potential breaking changes as possible, but there have been some major version updates already (Redis, BigCache), so it might be more consistent to update everything.
• Maybe rename JSONcodec to JSONCodec, and proto package to protobuf? Or better in a future version so that the upcoming one has few breaking changes, and next one might have more (e.g. context parameter as first parameter in all methods)?

And there's also one big question for the next release, where I'm interested in your opinion:

Almost all gokv packages are their own independent module, so that when people use for example redis, they don't get transitive dependencies on any of the other stores or their dependencies. Same goes for the root level gokv module, which people can use when they implement some library that depends just on the interface, but not a specific store implementation.
Because of this, there are different ways to do a new release, with different pros/cons:

1. Tag all packages on the same commit. For example the latest commit is be99209, and I can create a tag v0.7.0, and redis/v0.7.0 etc (each module requires their own tag).
   • Pros: It's nice that all tags point to the same commit, and I can create a GitHub release for the root gokv tag, describing all the changes (see "vNext" section in CHANGELOG)
   • Cons: The store implementations' go.mod file would point to gokv package v0.6.0, while it would be nicer to have them point to v0.7.0
2. Like 1., but before the tag, update the store implementations to use gokv/sql/test/util/encoding's master version
   • Pros: Up-to-date version of gokv/sql/... dependency instead of v0.6.0
   • Cons: For people working on a project that doesn't have a direct dependency on gokv root module, but only on a store implementation, they end up with a gokv transitive dependency with that dirty master version instead of a clean tag
3. Tag gokv root module and sql, test, util and encoding first. Then add commits that update the dependencies in the store implementations so they point to v0.7.0, then add the redis/v0.7.0 etc tags.
   • Pros: When someone just imports redis v0.7.0, they'll have a nice transitive dependency on gokv v0.7.0
   • Cons: On which tag should the GitHub release go? It can't be the plain v0.7.0 because at that commit the stores don't have the v0.7.0 tag yet. Or technically it could be, because module versions are independent from GitHub Releases, but it might be confusing and isn't nice/clean.
4. Tag all modules completely separately and instead of doing a single GitHub release and CHANGELOG, have one for each separate module
   • Pros: This fits the Go module versioning where each module requires its own tag (e.g. redis/v0.7.0). Most users might only care about a specific store implementation and its changelog so it's cleaner for them to only see that and not others.
   • Cons: Might be a mess with dozens of GitHub releases. Also, for stores where there was no change, should it stay at v0.6.0? Could it become a mess to have various different versions across the repo?

When I asked the same question here there was one user preferring a single release for the next version, and I'm leaning towards that as well, but it still leaves the above mentioned downsides.

=> Which variant do you prefer? Or do you see another alternative? Other pros/cons not mentioned yet?

[shawncatz]

Thanks for getting back to me! Really appreciate the thorough response...

My thoughts around your question really boil down to:

Short term

do the simplest thing possible to turn around a release

There's obviously a lot of moving parts in the work that's happening to roll the next release and the last thing you want to do is further complicate that process by changing entirely how you do releases. Iterative approaches are generally better in these cases.

Longer term

I see the pro's and con's on all of the different approaches and I tend to lean towards thinking of each of these libs as a discrete, separate thing. To be fair: I prefer microservice approaches over monolith approaches. That being said, I don't know that it's worth complicating the release process just to fit within a particular model. Overall, I feel like introducing separate release streams for every lib would end up being an administrative nightmare, one that could be solved with solid automation, but still a nightmare.

Bottom line

So after giving my opinions and thinking this through as I write, I think in the longer term I would prefer approach 3 above, where you update the core first, then update the libs with corresponding tags. However, I'm not really sure how to work around the Github Releases issues that you mention.

Hope that helps :)

In addition to be a coder, I'm also very opsy. just let me know if there's anything I can help with.

[philippgille]

Thanks a lot for offering your point of view! I also lean towards 3. I'll think more about how to handle the GitHub release when getting closer to it.

Status update: Just fixed the test container waiting time in CI (#162), as well as some other minor improvements in other PRs. Now of the above list the minimum is to fix the tests of the stores like MongoDB with connection issues. But the other two points are also still on the plan, but in the worst case (when I don't make progress on them) they could be dropped and not go into the v0.7.0 release.

[glimchb]

@philippgille maybe good idea to cleanup old branches https://github.com/philippgille/gokv/branches/all ?

[philippgille]

@philippgille maybe good idea to cleanup old branches https://github.com/philippgille/gokv/branches/all ?

Done ✔️

[philippgille]

Status update: Of the two stores where the unit tests in CI failed, MongoDB is now fixed with a switch from the unmaintained github.com/globalsign/mgo to the official go.mongodb.org/mongo-driver (PR #165).

That leaves on the list for the v0.7.0 release:

• Must: Fixing MySQL tests/CI
• Ideally: Check which dependencies have newer major versions (minor/patch versions are already up-to-date). For example BadgerDB. And update them.
• Optionally: Rename JSONcodec struct and proto package

[philippgille]

Status update:

• The MySQL tests always succeed locally, but in CI the connection can't be established. So it might just be an issue in the CI environment, not the tests or the store implementation being broken. I'm willing to do the new release first, and look into this afterwards.
• For the major version updates, BadgerDB has one (v4, instead of v1), but I'm afraid it leads to breaking changes for existing BadgerDB files on disk, so I'll first try to think about ways to mitigate issues. Maybe by having two versions existing side by side. But that can come after the release as well.
• I renamed proto to protobuf, as there hasn't been a release with the proto naming yet, so no breaking change. For JSONcodec it would be a breaking change, so I'll postpone to another time for now.

I also did some other improvements, like removing some test skipping on connection errors for many store implementations, as the test containers have worked very reliably.

=> So all good for preparing the v0.7.0 release. What this includes is a temporary branch where instead of referencing the v0.6.0 packages in the store's go.mod files, we use for example replace github.com/philippgille/gokv/test => ../test to make sure all packages work together with the very latest (latest on master, not latest release) code.

I'll report back when I have results from that.

[philippgille]

Finally!

The test where all store implementations used the latest versions of all gokv dependencies went well.
I added a documentation around the release process.
Followed the process to tag the modules without gokv dependencies first, then the helper modules that relied on them, and then all store implementations.

And finally we have the new release: v0.7.0

[glimchb]

amazing !

[shawncatz]

Woo! Great job!

[philippgille]

I'm closing this discussion, but feel free to open another one when you feel there should be another release before we create one on our own.