Heartbeat writer can always generate on-demand leased heartbeats, even if not at all configured #16014

shlomi-noach · 2024-05-27T07:47:57Z

Description

Fixes #15303 : the throttler wishes to RequestHeartbeats() and be awarded with a leased heartbeat injection.

With this change, the heartbeats writer (as part of repl tracker) is always enabled. It will open() and close() irrespective of heartbeats configuration (specifically irrespective --heartbeat_enable). We now identify three different configuration types for the heartbeat writer:

No configuration: the heartbeat writer is generally disabled and does not produce heartbeats (but see below).
On-demand: on-demand heartbeat interval (--heartbeat_on_demand_duration) was specified.
Always: the heartbeat writer is always on and produces heartbeats at a regular interval (--heartbeat_enable is set, and heartbeat_on_demand_duration is not set)

The way heartbeats are produced now depend on said configuration:

HeartbeatConfigTypeAlways: while open, the writer produces heartbeats at a regular interval.
RequetHeartbeats() is meaningless in this mode.
HeartbeatConfigTypeOnDemand: when opened, the writer produces heartbeats for the configured lease.
The heartbeats then expire. Lease can be renewed (after expired) or extended (while running) via
RequetHeartbeats().
HeartbeatConfigTypeNone: the writer does not initiate any heartbeats. However, RequetHeartbeats()
can be called to request a heartbeat lease. The lease duration is predetermined as defaultOnDemandDuration.

To clarify, if the heartbeat writer is not configured, then:

It still open()s and close()s
It still creates pool connections when open()ed
It does activate a ticker (via RateLimiter) which is then unused.

And so an unconfigured heartbeat writer now does consume some resources, but we can argue those are negligible. The flip side is that a RequestHeartbeats() is then always answered.

Docs PR: vitessio/website#1762

Related Issue(s)

Checklist

"Backport to:" labels have been added if this change should be back-ported to release branches
If this change is to be back-ported to previous releases, a justification is included in the PR description
Tests were added or are not required
Did the new or modified tests pass consistently locally and on CI?
Documentation was added or is not required

Deployment Notes

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…ill deliver on-demand heartbeats when requested even if configured as disabled Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

vitess-bot · 2024-05-27T07:48:00Z

shlomi-noach · 2024-05-27T08:03:35Z

Momentarily converting to Draft to work on proper goroutine cleanup.

EDIT: done.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…ectively. We use golang atomic LoadPointer/StorePointer to avoid using mutexes in RequestHeartbeats Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

codecov · 2024-05-27T10:04:30Z

Codecov Report

Attention: Patch coverage is 93.97590% with 5 lines in your changes missing coverage. Please review.

Project coverage is 68.72%. Comparing base (780abdd) to head (81b9d53).
Report is 2 commits behind head on main.

Files	Patch %	Lines
...t/vttablet/tabletserver/repltracker/repltracker.go	60.00%	4 Missing ⚠️
go/vt/vttablet/tabletserver/repltracker/writer.go	98.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16014      +/-   ##
==========================================
+ Coverage   68.69%   68.72%   +0.02%     
==========================================
  Files        1547     1547              
  Lines      198260   198270      +10     
==========================================
+ Hits       136202   136261      +59     
+ Misses      62058    62009      -49

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mattlord

Overall this LGTM. Thank you for working on this! ❤️ This will make things much nicer for users.

I had a few questions/concerns that are beyond simple nits.

I'd personally feel better about this not going in just before RC as it could take some time for subtle bugs and unexpected/unintended behaviors to surface. I'd also feel better if at least one important endtoend test which relies on on-demand heartbeats that was previously setting an explicit duration value no longer set a default so that we're testing this new behavior.

mattlord · 2024-05-31T14:53:08Z

go/vt/vttablet/tabletserver/repltracker/repltracker.go

@@ -52,8 +52,7 @@ var (

 // ReplTracker tracks replication lag.
 type ReplTracker struct {
-	mode           string
-	forceHeartbeat bool


OK, so in effect the old forceHeartbeat behavior is always enabled now rather than ONLY being enabled if the on-demand heartbeat interval is > 0.

Along with the new relevant defaults being:

defaultOnDemandDuration = 10 * time.Second defaultHeartbeatInterval = 1 * time.Second

Yes, that is correct!

go/vt/vttablet/tabletserver/repltracker/repltracker_test.go

go/vt/vttablet/tabletserver/repltracker/writer.go

mattlord · 2024-05-31T15:16:47Z

go/vt/vttablet/tabletserver/repltracker/writer.go

+
+var (
+	// Can be overridden by unit tests
+	defaultOnDemandDuration  = 10 * time.Second


Why wouldn't we make this the default for the --heartbeat_on_demand_duration flag? That way the default is visible to users.

It also simplifies the logic here IMO, as then the two modes are HeartbeatConfigTypeOnDemand and HeartbeatConfigTypeAlways. It feels awkward to me that HeartbeatConfigTypeNone effectively becomes HeartbeatConfigTypeOnDemand and we use a default duration value that is not exposed/visible to users.

Why wouldn't we make this the default for the --heartbeat_on_demand_duration flag?

Because if we made that the default, then that would change behavior to existing users, who perhaps do not want to enable on-demand heartbeat, and want full heartbeat.

That was also why we did not proceed with #15099

OK, then could we make/say that this flag is ignored when --heartbeat_enable is used? I think that's already the case isn't it? So I'm still not sure what we gain by doing things this way. Do you NOT have on demand heartbeats enabled by default as things are in the PR currently? I thought so. And if so, why would we not enforce the default duration via the flag instead of internally?

Maybe the difference is that the PR as-is limits on demand heartbeats to ONLY being used by the tablet throttler specifically whereas changing the flag would affect other components as well?

OK, then could we make/say that this flag is ignored when --heartbeat_enable is used?

Because again that would break existing setups. Think of the current configuration schema as an ancient mistake. Today we would have made the configuration different, but right now we need to stick to the existing rules.

I think that's already the case isn't it?

It is not.

Today, in main:

If non of the flags is defined, there's no heartbeat whatsoever (which is what we're addressing here in this PR, enabling a lease of heartbeats by throttler's demand)

If only --heartbeat_enable is set, then heartbeats are constant and always enabled.

If --heartbeat_on_demand_duration is defined (whether --heartbeat_enable is set or not) then heartbeat is generally speaking enabled, but only producing heartbeats on-deman (and on open).

This means if someone only has --heartbeat_enable in their setup, and I set the flag default to 10s, this would reduce the generated heartbeats to on-demand duration only, even though the user expects them to be constant and continuous.

However, If we take your suggestion:

could we make/say that this flag is ignored when --heartbeat_enable is used?

Then people already using both flags --heartbeat_enable + --heartbeat_on_demand_duration and who are expecting only a limited lease of heartbeats, wil suddenly see an influx of heartbeats and a bloat of binary logs.

I hope this clarifies a bit.

Then people already using both flags --heartbeat_enable + --heartbeat_on_demand_duration and who are expecting only a limited lease of heartbeats, wil suddenly see an influx of heartbeats and a bloat of binary logs.

I thought that they both served the same purpose with regards to the throttler, the only thing --heartbeat_enable got you was sub-second replication lag metrics via the replication reporter vs. instead using the polling based reporter via --enable_replication_reporter. Perhaps I'm wrong on that part?

I thought that using both flags would also potentially lead to failures? #14978

I was thinking that 1) (virtually) nobody is using both 2) if they are, they shouldn't be as that can case failures 3) we could take this opportunity to clean this up a bit. Maybe I misunderstood though?

In any event, you've explained that it's more complicated than I had initially thought and we don't need to solve these issues in this PR. 🙂 Thanks!

we could take this opportunity to clean this up a bit.

I'm in favor. I don't like the existing configuration setup. If we can change things in one shot and without disrupting existing setups, I'm happy to. In which case, as you say, the changes in this PR can be greatly simplified. Let's perhaps make that discussion tomorrow.

go/vt/vttablet/tabletserver/repltracker/writer.go

mattlord · 2024-05-31T15:46:11Z

go/vt/vttablet/tabletserver/repltracker/writer_test.go

+	t.Run("exhaust lease, no heartbeats", func(t *testing.T) {
+		<-time.After(tw.onDemandDuration)
+		currentWrites := writes.Get()
+		<-time.After(3 * time.Second)


IMO we should use the onDemandDuration value again here and everywhere else that we're using the 3 second literal in these tests.

It's actually explicitly using a different value. It would be confusing to reuse onDemandDuration. What we want to say is: after onDemandDuration is expired, no new heartbeats are written, regardless of how many seconds we wait.

If we reused onDemandDuration, that could imply to the reader that there is come mysterious connection, where after the duration expires, you need it to expire again so as to eliminate heartbeats.

Does that make sense?

mattlord · 2024-05-31T15:48:59Z

go/vt/vttablet/tabletserver/repltracker/writer_test.go

+	}
+	tw.Open()
+	defer tw.Close()
+	{


Personal preference, but I don't see the value that these code blocks add and use of them is pretty non-standard within Vitess.

I'll explain why I use this: it's allowing me to redeclare a variable (rateLimiter := in this case) that I do not need to fear is "leaking" elsewhere. Also, I dislike how in unit testing you want to duplicate some tests, but golang requires that the first assignment be := and then you can only use =. But then as you shuffle things, you need to again modify := to = or = to := in places that are unrelated to your change. This causes git diffs that are irrelevant to your change (not unlike trailing comma in a table definition on an unrelated index when you add a new index).
So I feel like these code blocks help me isolate:

actual code/variables

git diffs

It's a personal preference and I recognize this is not standard in Vitess. FWIW I've already spread that around quite a bit.

mattlord · 2024-05-31T15:50:50Z

go/vt/vttablet/tabletserver/repltracker/writer_test.go

+		assert.NotNil(t, rateLimiter)
+	}
+	t.Run("open, initial heartbeats", func(t *testing.T) {
+		ctx, cancel := context.WithTimeout(context.Background(), tw.onDemandDuration-time.Second)


Any reason to use the -time.Second part? That would put us at 2 seconds and it seems like it may be possible that we then only tick with the ticker once before the context is done.

Yes. The reasoning is that I don't want to get past the expiration of the heartbeat lease. This function is checking:

That heartbeats are generated when on-demand is engaged (upon Open() in this case)

And that they keep being generated for the duration of the on-demand lease. The heartbeat interval in this test is 250ms, and we test every 1 second, just shy of the expiration.

If we go past the expiration, then we can wrongly condlude that heartbeats are not being produced when they should (they shouldn't, past the expiration).

go/vt/vttablet/tabletserver/repltracker/writer_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…te we still generate heartbeats when both this flag as well as '--heartbeat_enable' are disabled Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2024-06-02T11:28:46Z

I'd also feel better if at least one important endtoend test which relies on on-demand heartbeats that was previously setting an explicit duration value no longer set a default so that we're testing this new behavior.

Done in 236f074, affecting onlineddl_vrepl test.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

mattlord

Nice work on this! ❤️

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2024-06-03T08:16:42Z

If we're not going to merge it for RC1, then I have further simplification for the concurrency logic, effectively reducing n concurrent goroutines to 1.

shlomi-noach · 2024-06-03T11:51:01Z

This will be the followup to clean up the concurrency logic:

https://github.com/planetscale/vitess/compare/throttler-dynamic-heartbeat...planetscale:throttler-dynamic-heartbeat-2?expand=1

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2024-06-04T11:56:11Z

All right. Since we're still here, I did merge throttler-dynamic-heartbeat-2, see in particular a8673c1.

timvaillancourt · 2024-06-11T09:21:16Z

Nice work on this RFC and PR, I like the idea and code 👍. I'm giving this a stamp but I suggest there's a 3rd reviewer as I'm still new at this 😅

Related and certainly out-of-scope, this reminded me of a hack Slack made to enable the "constant" heartbeat on some keyspaces: slackhq#207. TL;DR: it's difficult to enable the heartbeat on a live keyspace that never had it - if you add --enable-heartbeat on the replicas first lag will grow forever because the primary hasn't enabled the heartbeat yet. I saw mention of deferred refactoring - let me know if I can help with that and resolve this upstream while I'm at it!

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2024-06-16T06:59:00Z

Call for additional reviews 🙏

deepthi

I had only a couple of nits. This is much cleaner than the previous forceHeartbeat implementation 👍

deepthi · 2024-07-02T23:30:10Z

go/timer/rate_limiter_test.go

@@ -54,6 +54,23 @@ func TestRateLimiterShort(t *testing.T) {
 	assert.Less(t, val, 10)
 }

+func TestRateLimiterAllowOne(t *testing.T) {


Always nice to see a new unit test for a new function :)

deepthi · 2024-07-02T23:34:44Z

go/vt/vttablet/tabletserver/repltracker/writer_test.go

@@ -33,12 +33,17 @@ import (
 	"vitess.io/vitess/go/vt/vttablet/tabletserver/tabletenv"
 )

+const (
+	testKeyspaceShard = "test:0"


Two nits:

This constant seems to be used exactly once. What is the value of defining it outside of the function where it is used?

The standard delimiter is / so that this would be test/0. It would be even better to use test/- which we have been moving to.

The constant used to be used elsewhere -- but that has changed. Good catch. Now inside to the function and is test/0.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2024-07-03T05:39:03Z

Doc update is in vitessio/website#1762

shlomi-noach added 3 commits May 27, 2024 10:21

RateLimiter: AllowOne

79a37f4

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

HeartbeatWriter initializes, opens and closes even if disabled, and w…

f89eeb0

…ill deliver on-demand heartbeats when requested even if configured as disabled Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

comments

2d29eda

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach requested a review from a team May 27, 2024 07:47

shlomi-noach requested review from harshit-gangal, systay, rohit-nayak-ps and deepthi as code owners May 27, 2024 07:47

github-actions bot added this to the v20.0.0 milestone May 27, 2024

shlomi-noach marked this pull request as draft May 27, 2024 08:03

shlomi-noach added 2 commits May 27, 2024 11:45

do not stop ticks unless actually running

1116f80

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

open() and close() create and destroy the on-deman rate limiter, resp…

02ca773

…ectively. We use golang atomic LoadPointer/StorePointer to avoid using mutexes in RequestHeartbeats Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach marked this pull request as ready for review May 27, 2024 09:45

shlomi-noach mentioned this pull request May 27, 2024

Docs: Unconfigured heartbeat still serves heartbeat event at throttler requests vitessio/website#1762

Merged

shlomi-noach removed the NeedsWebsiteDocsUpdate What it says label May 27, 2024

shlomi-noach mentioned this pull request May 30, 2024

Release of v20.0.0-RC1 #16010

Closed

49 tasks

deepthi requested a review from timvaillancourt May 30, 2024 15:54

mattlord reviewed May 31, 2024

View reviewed changes

shlomi-noach added 5 commits June 2, 2024 08:42

removd redundant check

ad8950a

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

use atomic.Pointer

58be5f2

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

remove redundant check

1fd2b41

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

defer setting isOpen

0ba8682

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Remove '--heartbeat_on_demand_duration' from endtoend test, to valida…

236f074

…te we still generate heartbeats when both this flag as well as '--heartbeat_enable' are disabled Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

proper testing for pools being open

63ab624

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

mattlord approved these changes Jun 2, 2024

View reviewed changes

shlomi-noach added 2 commits June 3, 2024 09:26

simplify rate limiting logic and reduce concurrent routines to one

a8673c1

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

minimal on demand duration at 4s

2d0fb4a

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

check disabling heartbeats _after_ writing the heartbeat

8d39664

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

timvaillancourt approved these changes Jun 11, 2024

View reviewed changes

empty commit to kick CI

875a2d6

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

frouioui modified the milestones: v20.0.0, v21.0.0 Jun 20, 2024

rohit-nayak-ps approved these changes Jul 2, 2024

View reviewed changes

deepthi approved these changes Jul 2, 2024

View reviewed changes

shlomi-noach added 2 commits July 3, 2024 07:51

Merge branch 'main' into throttler-dynamic-heartbeat

73046cf

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

remove variable

81b9d53

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach merged commit 79c54e5 into vitessio:main Jul 3, 2024
94 checks passed

shlomi-noach deleted the throttler-dynamic-heartbeat branch July 3, 2024 05:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Heartbeat writer can always generate on-demand leased heartbeats, even if not at all configured #16014

Heartbeat writer can always generate on-demand leased heartbeats, even if not at all configured #16014

shlomi-noach commented May 27, 2024 •

edited

Loading

vitess-bot bot commented May 27, 2024

shlomi-noach commented May 27, 2024 •

edited

Loading

codecov bot commented May 27, 2024 •

edited

Loading

mattlord left a comment

mattlord May 31, 2024

shlomi-noach Jun 1, 2024 •

edited

Loading

mattlord May 31, 2024

shlomi-noach Jun 1, 2024

mattlord Jun 1, 2024 •

edited

Loading

shlomi-noach Jun 2, 2024

mattlord Jun 2, 2024

shlomi-noach Jun 2, 2024

mattlord May 31, 2024

shlomi-noach Jun 2, 2024 •

edited

Loading

mattlord May 31, 2024

shlomi-noach Jun 2, 2024 •

edited

Loading

mattlord May 31, 2024 •

edited

Loading

shlomi-noach Jun 2, 2024

shlomi-noach commented Jun 2, 2024

mattlord left a comment

shlomi-noach commented Jun 3, 2024

shlomi-noach commented Jun 3, 2024

shlomi-noach commented Jun 4, 2024

timvaillancourt commented Jun 11, 2024

shlomi-noach commented Jun 16, 2024

deepthi left a comment

deepthi Jul 2, 2024

deepthi Jul 2, 2024

shlomi-noach Jul 3, 2024

shlomi-noach commented Jul 3, 2024

Heartbeat writer can always generate on-demand leased heartbeats, even if not at all configured #16014

Heartbeat writer can always generate on-demand leased heartbeats, even if not at all configured #16014

Conversation

shlomi-noach commented May 27, 2024 • edited Loading

Description

Related Issue(s)

Checklist

Deployment Notes

vitess-bot bot commented May 27, 2024

Review Checklist

General

Tests

Documentation

New flags

If a workflow is added or modified:

Backward compatibility

shlomi-noach commented May 27, 2024 • edited Loading

codecov bot commented May 27, 2024 • edited Loading

Codecov Report

mattlord left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach Jun 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord Jun 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach Jun 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach Jun 2, 2024 • edited Loading

Choose a reason for hiding this comment

mattlord May 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach commented Jun 2, 2024

mattlord left a comment

Choose a reason for hiding this comment

shlomi-noach commented Jun 3, 2024

shlomi-noach commented Jun 3, 2024

shlomi-noach commented Jun 4, 2024

timvaillancourt commented Jun 11, 2024

shlomi-noach commented Jun 16, 2024

deepthi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach commented Jul 3, 2024

shlomi-noach commented May 27, 2024 •

edited

Loading

shlomi-noach commented May 27, 2024 •

edited

Loading

codecov bot commented May 27, 2024 •

edited

Loading

shlomi-noach Jun 1, 2024 •

edited

Loading

mattlord Jun 1, 2024 •

edited

Loading

shlomi-noach Jun 2, 2024 •

edited

Loading

shlomi-noach Jun 2, 2024 •

edited

Loading

mattlord May 31, 2024 •

edited

Loading