sp_QuickieStore: Add the ability to sort by arbitrary orders, but particularly wait stats and "plan count by hashes".

[ReeceGoding]

This commit is a small monster, but I was stunned by the results. It lets you very quickly find the query hashes that are generating the biggest number of distinct plans. This simultaneously tells you which ad-hoc queries are the most worth worrying about and it also tells you what queries (even non-ad-hoc stuff such as stored procedures) have changed their plans a lot. Even if you think the code is ugly (it is), run it at least once before dismissing it. I think it's going to be very useful. Erik's idea of using the hashes was a very good one.

The design of sp_QuickieStore makes it difficult to add new sort orders that don't come from sys.query_store_runtime_stats. This is the source of most of the ugliness in this PR and it required me adding to the existing architecture. However, I think that I may have solved the problem of adding arbitrary sort orders to sp_QuickieStore. I suspect that this may be immediately helpful for #448 , but I don't know that yet.

This closes most of #446 , but I still need to figure out how to sort by wait stats. It might be best if I used this branch as a base for that, but I don't know yet. Don't wait on merging this just for the sake of the wait stats stuff. It will probably be easier to do the wait stats stuff after this is merged.

[erikdarlingdata]

@ReeceGoding thanks for this. I’m on vacation until the end of July, so please take your time with testing etc. I won’t be able to look at anything until then.

[ReeceGoding]

@erikdarlingdata I've not tested what I'm mentioning in this comment yet, but I think that we've both screwed up by not updating the list of tables to truncate. I've missed #plan_ids_with_query_hashes and I think you've missed #query_text_search_not. I think that we need to make some sort of note somewhere to always test or at least think about @get_all_databases = 1. Maybe just write comments near the tables/variables that are relevant to it?

[ReeceGoding]

It might be best if I used this branch as a base for that, but I don't know yet.

Had some time to think about it. I'll put the wait stats bit in this PR as well. It needs the same architecture and doing two at once gives me a good excuse to make sure that it's tidy and easy to extend.

[ReeceGoding]

I've added a wait stats part on. I intended to add in a sort order for total wait time, but it was so easy to add in @sort_order options for every wait time that we already support with @wait_filter that it was just the natural thing to do. It's arguably an improvement on @wait_filter, since that only filters rather than sorts.

What I find most remarkable about the sort orders that I've added today is that it just worked. I spent only a few minutes removing silly syntax errors and then the code simply worked. It's a good hint that the architecture that I've added in has done its job and we can now add whatever @sort_orders we want to sp_QuickieStore.

I'll test more, of course. Afterwards, I'll add on the promised sort order for total wait time. If my bragging in the previous paragraph is honest, then it will be very easy.

[ReeceGoding]

Added on an @sort_order for 'total waits', so I consider #446 ready to close. As promised, I'll do more testing.

Regarding the question of if the 'total waits' sort order is useful, I've found it handy because it reveals big waits that I'd never think to specifically filter for. It's good when a tool tells you about problems you didn't know you had. The only downside is that the top_waits column filters out some waits, as I mentioned here, so the numbers that I'm using to sort might not agree with what's in the top_waits column. Here is one such case from a machine with hardly any waits. You can see that the column that I needed for the sort on the right doesn't always agree with the sum of the numbers in top_waits.

If my bragging in the previous paragraph is honest, then it will be very easy.

It was even easy than I thought! It took less than half an hour and I don't recall needing to think at any point. I'm completely convinced that the architecture for adding arbitrary sort orders will be useful, particularly for #448.

[ReeceGoding]

@erikdarlingdata I think it all works and my testing is done.

[erikdarlingdata]

@ReeceGoding I noticed you were still tweaking this a little today -- do you have anything left to do or is this good to merge into dev? I'd like to test it with the other changes combined.

[ReeceGoding]

@erikdarlingdata I consider it finished.