SoundWire: stream: improve de-prepare transitions #6838

Job	Run time
Clang build x86_64	4m 18s
GCC build arm64	17m 32s
GCC static -fanalyzer	4m 45s
GCC build arm64 allmodconfig	2m 5s
GCC build i386	15m 21s
GCC build X86_64 with W=1	2m 57s
GCC build X86_64	17m 14s
GCC build X86_64 with W=1 allmodconfig	1m 54s
Sparse check	3m 7s
	1h 9m 13s

Provide feedback