db: add TestWALFailoverRandomized

[jbowens]

Add a randomized test of WAL failover and recovery of a DB from failover WAL logs.

Close #3865.

[cockroach-teamcity]

This change is 

[jbowens]

Stressing produces a failure:

    open_test.go:1657: 
                Error Trace:    /Users/jackson/go/src/github.com/cockroachdb/pebble/open_test.go:1657
                                                        /Users/jackson/go/src/github.com/cockroachdb/pebble/open_test.go:1693
                Error:          Received unexpected error:
                                pebble: error when replaying WAL: pebble/record: invalid chunk
                                (1) replaying wal 43, offset (secondary/000043-003.log: 0), 105095 from previous files
                                Wraps: (2) attached stack trace
                                  -- stack trace:
                                  | github.com/cockroachdb/pebble.(*DB).replayWAL
                                  |     /Users/jackson/go/src/github.com/cockroachdb/pebble/open.go:885
                                  | github.com/cockroachdb/pebble.Open
                                  |     /Users/jackson/go/src/github.com/cockroachdb/pebble/open.go:505
                                  | github.com/cockroachdb/pebble.TestWALFailoverRandomized.func4
                                  |     /Users/jackson/go/src/github.com/cockroachdb/pebble/open_test.go:1656
                                  | github.com/cockroachdb/pebble.TestWALFailoverRandomized
                                  |     /Users/jackson/go/src/github.com/cockroachdb/pebble/open_test.go:1693
                                  | [...repeated from below...]
                                Wraps: (3) pebble: error when replaying WAL
                                Wraps: (4) forced error mark
                                  | "pebble: corruption"
                                  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
                                Wraps: (5) attached stack trace
                                  -- stack trace:
                                  | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
                                  |     /Users/jackson/go/src/github.com/cockroachdb/pebble/internal/base/error.go:30
                                  | github.com/cockroachdb/pebble/record.init
                                  |     /Users/jackson/go/src/github.com/cockroachdb/pebble/record/record.go:150
                                  | runtime.doInit1
                                  |     /usr/local/go/src/runtime/proc.go:7176
                                  | runtime.doInit
                                  |     /usr/local/go/src/runtime/proc.go:7143
                                  | runtime.main
                                  |     /usr/local/go/src/runtime/proc.go:253
                                  | runtime.goexit
                                  |     /usr/local/go/src/runtime/asm_arm64.s:1222
                                Wraps: (6) pebble/record: invalid chunk
                                Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *withstack.withStack (6) *errutil.leafError
                Test:           TestWALFailoverRandomized

[RaduBerinde]

Reviewable status: 0 of 1 files reviewed, 1 unresolved discussion (waiting on @jbowens and @sumeerbhola)

---

open_test.go line 1656 at r1 (raw file):

			t.Log("initiating hard crash")
			setIsCrashing(true)
			// Take a strict clone of the filesystem and use that going forward.

[nit] crash-consistent clone

[jbowens]

TFTR!

The failure was resolved by #3896.

Reviewable status: 0 of 1 files reviewed, all discussions resolved (waiting on @sumeerbhola)