-
Notifications
You must be signed in to change notification settings - Fork 6
/
18_binary_insertion_sort.html
501 lines (414 loc) · 18.9 KB
/
18_binary_insertion_sort.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width">
<title>18. Binary insertion sort</title>
<link rel="stylesheet" href="template/style.css">
</head>
<body>
<span style="float: right">
[
<a href="17_adaptive_merge_sort.html">previous</a>,
<a href="index.html">contents</a>,
<a href="19_insertion_sort.html">next</a>
]
</span>
<a name="L18.-Binary-insertion-sort"></a>
<h1>18. Binary insertion sort</h1>
<a name="The-Organ-Grinder"></a>
<h2>The Organ Grinder</h2>
<p>I would like to occasionally introduce three minutes of culture.
I used to tell you stories but right now I decided
I’ll just occasionally share a song or something like that
which would indicate what mood I’m in.
This is a very great song by <a href="https://en.wikipedia.org/wiki/Franz_Schubert">Franz Schubert</a> but it also
perfectly reflects what will go on with the course in how I feel.
The song is called <a href="https://www.britannica.com/topic/Winterreise">“The Organ Grinder”</a> (Der Leiermann).
The singer is <a href="https://en.wikipedia.org/wiki/Dietrich_Fischer-Dieskau">Dietrich Fischer-Dieskau</a> maybe the greatest
leader, or art song singer of the last 50, 60, or 70 years.
He started singing in the late forties.
Let us spend a couple of minutes and listen to it… (<a href="https://www.youtube.com/watch?v=sIIS-UgixGE">Video here</a>)</p>
<a name="Strategy"></a>
<h2>Strategy</h2>
<p>Let’s try our <a href="https://en.wikipedia.org/wiki/Insertion_sort">insertion sort</a> idea.
We are going to learn more about insertion sort then
you ever wanted to know.
First we will review the basic idea of algorithm.
Always start with a picture:</p>
<pre><code>| sorted piece | unsorted piece |
</code></pre>
<p>We start with an empty range on the left which is the sorted portion.
We basically want to grow it, one element at a time,
while ensuring it stays sorted.
By repeating it inductively, eventually the whole range is sorted.
So, the main idea is to pick an element in the unsorted piece,
find where the element goes, and insert it there.</p>
<a name="Insertion-sort-variations"></a>
<h3>Insertion sort variations</h3>
<p>How many algorithmic versions of insertion sort are there?
Finding where it should go could be done with either:</p>
<ol>
<li>Linear search</li>
<li>Binary search</li>
</ol>
<p>There is another version which was invented (as everything else was) by <a href="https://en.wikipedia.org/wiki/Tony_Hoare">Tony Hoare</a>.
He realized that in the inner loop of
insertion sort you have to do two things:</p>
<ol>
<li>Guard that you’re not crossing by size</li>
<li>Guard that you’re not crossing the first</li>
</ol>
<p>This makes the insertion sort do two comparisons per cycle.
You could have an insertion sort with a guard,
assume that somebody puts (by hook or by crook)
the smallest element first.
In other words, if you can guarantee
smallest is up front, then you can simplify the inner loop, just going through the algorithm.
Since we wrote the machinery for binary search in previous lessons,
let’s start by writing that.</p>
<a name="When-is-insertion-sort-useful-3f-"></a>
<h3>When is insertion sort useful?</h3>
<p>When should one use insertion sort?
This is an interesting point we should discuss.</p>
<ol>
<li><p>We already talked about when <code>n</code> is small.
How small? We already proved it was when <code>n = 16</code>.
Is it the exact? No, it’s not.
But it’s a good rule of thumb.</p></li>
<li><p>If we just have a few things
to add to a sorted list that would be good.
In other words, most of the list is sorted
but 16 or so elements are out of order.</p></li>
<li><p>Insertion sort is going to move an element
from where it is to where it should be,
one step at a time.
So another case is when the average distance
from where it is to where it should be
is small.
It’s “nearly sorted”.</p></li>
</ol>
<p>There are some considerations where you want to look at the relative cost but they are not important for asymptotic assessment.
A quadratic algorithm,
regardless of the ratio between move and compare,
is still a quadratic algorithm.</p>
<a name="Naming-insertion-sort-function"></a>
<h3>Naming insertion sort function</h3>
<p>Unfortunately, STL does not have insertion sort.
Should it?
Yes, it should.
But they threw it out from the public library<sup id="fnref:1"><a href="#fn:1" rel="footnote">1</a></sup>.
At least put it in your library.
It might not be called insertion sort.
Maybe we should call it something else.
What’s a good name?
This is not a bogus question.
Finding a good name is important
because we want to lead people to use it when
these three conditions are met.
Maybe, <code>sort_almost_sorted</code>.
Only few of them are out of order or it’s just perturbed
everywhere, but not by much.
However, we will find it works well in that case only when the thing
to sort isn’t enormous.
So instead we will settle on <code>binary_insertion_sort</code>.</p>
<p>Naming is extremely hard, but very important.
The goal is to name components so people can actually understand
what they mean. It helps people.
We have to discuss nomenclature.
Respectable sciences spend most of their time discussing
nomenclature.
Chemists, physicists, they know what to call things.
It’s only computer science that doesn’t.</p>
<p>I have to tell you a story <a href="https://sean-parent.stlab.cc/papers-and-presentations/">Sean Parent</a> shared with me.
When STL was introduced, people at Apple decided to try it.
They tried it and found it absolutely unacceptable because they replaced their <code>list</code> with STL
<code>std::list</code> and everything became extremely slow.
The problem is their list was what is still called a vector.
They didn’t realize linked lists are called “linked lists”.
It sort of works, you know, slowly.</p>
<a name="Binary-insertion-sort"></a>
<h2>Binary insertion sort</h2>
<p>First, let’s see how we will use it,
specifically where we can stick it in our code.
We added a minimum size for which our merge switches
algorithms and uses the extra buffer.
Now, we can add a similar check to our sort to switch
algorithms.</p>
<pre><code>const size_t INSERT_SORT_CUTOFF = 16;
template <typename I, typename N, typename R, typename B>
// I is ForwardIterator
// N is Integral
// R is WeakStrictOrdering on the value type of I
I sort_adaptive_n(I first, N n, R r, B buffer, N buffer_size) {
if (!n) return first;
if (n < INSERT_SORT_CUTOFF) return binary_insertion_sort_n(first, n, r); // HERE
N half = n >> 1;
if (!half) return ++first;
I middle = sort_adaptive_n(first, half, r, buffer, buffer_size);
I last = sort_adaptive_n(middle, n - half, r, buffer, buffer_size);
merge_adaptive_n(first, half, middle, n - half, r, buffer, buffer_size);
return last;
}
</code></pre>
<p>(Recall, that we proved 16 was a good cutoff.)
The standard C convention for old people is that <code>ALL_CAPS</code> means it’s a macro.
We will use this for a constant here<sup id="fnref:2"><a href="#fn:2" rel="footnote">2</a></sup>.</p>
<p>We have a few decisions to make for insertion sort.
Should we use <code>upper_bound</code> or <code>lower_bound</code> for binary search?
If we use <code>lower_bound</code> and we have equal guys at the end,
then it will move too far beyond them.
So we want <code>upper_bound</code>.</p>
<p>Since our sort only requires <code>ForwardIterator</code>,
we should aim to make insertion sort use <code>ForwardIterator</code>.
Using <code>ForwardIterator</code> is actually a piece of cake for the binary search,
but we have to be careful.
You might want to use the <code>upper_bound</code> we wrote together.
But, remember it calls <code>std::distance</code> which is linear for <code>ForwardIterator</code>.
So let’s use <code>upper_bound_n</code>.</p>
<p>What we will first write is a function for finding where an element
goes and placing it there.
Then we will structure our loop<sup id="fnref:3"><a href="#fn:3" rel="footnote">3</a></sup> around that.</p>
<pre><code>template <typename I, typename N, typename R>
// I is ForwardIterator
// N is Integral
// R is WeakStrictOrdering on the value type of I
I binary_insert_n(I first, N n, I current, R r) {
// precondition: is_sorted(first, current, r) && current is a valid iterator
// && std::distance(first, current) == n
I insertion_point = upper_bound_n(first, n, *current, r);
rotate_right_by_one(insertion_point, ++current);
return insertion_point;
}
</code></pre>
<p><code>rotate_right_by_one</code> will be discussed in just a bit.
It’s important to return here, in case someone
else wants to use this function.</p>
<p>To write the loop that calls this function,
I suggest that we carefully write invariants.
We have the range:</p>
<pre><code>[first, i)
</code></pre>
<p>It’s semi-open, so <code>i</code> is past the end.
What we want is:</p>
<pre><code>is_sorted_n(first, i, r) && std::distance(first, current) == i
</code></pre>
<p>That’s the invariant on which we rely.</p>
<pre><code>template <typename I, typename N, typename R>
// I is ForwardIterator
// N is Integral
// R is WeakStrictOrdering on the value type of I
I binary_insertion_sort_n(I first, N n, R r) {
if (n == N(0)) return first;
I current = first;
++current;
N i(1);
while (i < n) {
// invariant: is_sorted_n(first, i, r) && std::distance(first, current) == i
binary_insert_n(first, i++, current++, r);
}
return current;
}
</code></pre>
<a name="Rotate"></a>
<h2>Rotate</h2>
<a name="Rotate-for-bidirectional-iterators"></a>
<h3>Rotate for bidirectional iterators</h3>
<p>Once we find where it goes (the element to insert), how do we make room for it?
We “rotate” to the right by one.
If it is a bidirectional iterator there is a beautiful algorithm, copying from the back.
The algorithm is called: <a href="https://en.cppreference.com/w/cpp/algorithm/copy_backward"><code>std::copy_backward</code></a>.</p>
<pre><code>template <typename I>
// I is BidirectionalIterator
void rotate_right_by_one(I first, I last, std::bidirectional_iterator_tag) {
typedef typename std::iterator_traits<I>::value_type T;
I butlast = last;
--butlast;
T x = *butlast;
std::copy_backward(first, butlast, last);
*first = x;
}
</code></pre>
<p>Note that forward copy is the wrong thing, because it will overwrite everything with the same value (namely the first value).</p>
<a name="Rotate-for-forward-iterators"></a>
<h3>Rotate for forward iterators</h3>
<p>For <code>ForwardIterator</code> we have to shift all the elements up,
we move one out of the way, to make room,
and continue up the array until we find an empty place to put it.</p>
<p>I think the problem is quite instructive not just because
it’s a useful algorithm, which it is, but because of the method for deriving it.
Before coding, let us do a bit of mathematics.
You can always “haircut” code, but remember mathematics?
I used to talk about it before they told me to switch to programming.
Deriving mathematically is a good thing.</p>
<p>We want to rotate a sequence right by one.
If we have an empty sequence. What do we do?
We are done.
If we have a one-element sequence <code>a_0</code> and we want to rotate it,
how do we do it?
Done.
That allows us to consider an inductive solution.
Somehow, by hook or by crook, we have an algorithm which knows how to shift <code>n</code> things,
such as the range:</p>
<pre><code>a_{0} ... a_{n-1}
</code></pre>
<p>Then the question is, how could we get an algorithm for</p>
<pre><code>a_{0} ... a_{n-1} a_{n}
</code></pre>
<p>How do we add one additional element?
After the shift the first <code>n</code> elements (leaving <code>a_{n}</code> fixed) we have:</p>
<pre><code>a_{n-1} a_{0} ... a_{n-2} a_{n}
</code></pre>
<p>What do we need to do to solve the problem?
Just swap <code>a_{n-1}</code> and <code>a_{n}</code>.
In general, swap last and first.</p>
<p>Here is an example with 3 elements:</p>
<pre><code>1 2 3
</code></pre>
<p>We first rotate the one element range <code>1</code>. It’s all done,</p>
<pre><code>[1] 2 3
</code></pre>
<p>To rotate the two element range,
swap the first element, and the one following our range:</p>
<pre><code>[2 1] 3
</code></pre>
<p>Now we have the first two rotated.
To rotate the full sequence
we once again swap the first and last:</p>
<pre><code>[3 1 2]
</code></pre>
<p>Done!
It might not be the fastest, but it is going to be much more elegant.</p>
<pre><code>template <typename I>
// I is ForwardIterator
void rotate_right_by_one(I first, I last, std::forward_iterator_tag) {
if (first == last) return;
I current = first;
while (++current != last) std::swap(*first, *current);
}
</code></pre>
<p>Let’s write a dispatch for both versions,
it will compile to no code<sup id="fnref:4"><a href="#fn:4" rel="footnote">4</a></sup>.</p>
<pre><code>template <typename I>
inline
void rotate_right_by_one(I first, I last) {
rotate_right_by_one(first, last, typename std::iterator_traits<I>::iterator_category());
}
</code></pre>
<a name="Should-we-support-forward-iterator-3f-"></a>
<h3>Should we support forward iterator?</h3>
<p>Someone brought up that <code>ForwardIterator</code> doesn’t
really make sense for this algorithm<sup id="fnref:5"><a href="#fn:5" rel="footnote">5</a></sup>,
because if we have something like a linked list, we don’t need to rotate
or shift elements around,
we can just insert it where it belongs.
That’s a good idea, but maybe some measurements will show us otherwise.
We already implemented optimal linked list sort.
Later we need to compare whether it’s actually
faster to use our list sort, or to use our method
we develop.</p>
<p>Why do you think I say that?
List sort destroys locality.
If at every <code>cdr</code> (next) you get to a different cache
line, that’s a problem.
In our sort we constantly re-link next,
so eventually you get to a point where everything is scattered all over memory.</p>
<p>STL used to have a sentence in the container
section which the standard committee threw out.
Use a vector.
This is a true statement.
Unless you are absolutely positive
you need something else, use a vector or a C array<sup id="fnref:6"><a href="#fn:6" rel="footnote">6</a></sup>.</p>
<a name="Code"></a>
<h2>Code</h2>
<ul>
<li><a href="code/binary_insertion_sort.h">binary_insertion_sort.h</a></li>
<li><a href="code/test_insertion_sort.cpp">test_insertion_sort.cpp</a></li>
</ul>
<div class="footnotes">
<hr/>
<ol>
<li id="fn:1">
<p>Alex: Of course, STL still has insertion sort on the inside.
It has to.
What happened during the standardization process,
is they took something which was in the library and was used by the library and threw it out.
The argument was, “we already have too many sorts”.
Is it a good argument?
No, you need to have as many sorts as people might need
when they do practical things.
How many sorts are in STL?</p>
<ol>
<li><a href="https://en.cppreference.com/w/cpp/algorithm/sort"><code>std::sort</code></a> the fastest sort.</li>
<li><a href="https://en.cppreference.com/w/cpp/algorithm/stable_sort"><code>std::stable_sort</code></a>, this is merge sort, the one we are trying to write.</li>
<li><a href="https://en.cppreference.com/w/cpp/algorithm/partial_sort"><code>std::partial_sort</code></a> sort the first thousand, out of a million (something you frequently do in search engines).</li>
<li><a href="https://en.cppreference.com/w/cpp/algorithm/nth_element"><code>std::nth_element</code></a>.
Not quite a sort, but it’s sort related.
What it does is pin, for example the 30th percentile
element, and put all the smaller before, and then all the larger.
If I want to find another one, I can pin again, and sort inbetween, etc.</li>
</ol>
<a href="#fnref:1" rev="footnote">↩</a></li>
<li id="fn:2">
Alex:
I think you should read Bjarne’s book “Design and Evolution of C++”.
It’s a short book and very instructive.
This is the book I hope he will revise, because it only goes up to like 92.
In any case, the last chapter is dedicated to the preprocessor.
It has an epigraph: “CPP must be destroyed” - Cato.
<a href="https://en.wikipedia.org/wiki/Carthago_delenda_est"><em>Carthago delenda est</em></a> (“Carthage must be destroyed”).
It’s still not destroyed, but maybe one day.<a href="#fnref:2" rev="footnote">↩</a></li>
<li id="fn:3">
Alex: Could we use a <code>for</code> loop instead of a <code>while</code>?
Yes, but I hate <code>for</code> loops.
Why? Because the semantics have changed about 6 times,
since I started C++, <code>while</code> loops never changed.<a href="#fnref:3" rev="footnote">↩</a></li>
<li id="fn:4">
Alex: Someday we will get concepts
in the C++ standard and not have to write these things.
But that will be at least 5 years and I won’t be programming.
I’m like an old man planting an apple tree.<a href="#fnref:4" rev="footnote">↩</a></li>
<li id="fn:5">
<p>Alex:
I actually drop this requirement in STL and require
<code>RandomAccessIterator</code> for all the sorts.
It wasn’t the standard committee’s fault, just me.
I am not sure if I agree with myself.</p>
<p>The reasoning went like so.
Most people of course don’t know anything.
Therefore if you give them things which sort
<code>ForwardIterator</code>s, they will attempt to use
them on things like linked lists.
It’s so much slower that they would be better off copying into a vector,
sort stuff there, and then copy back.</p>
<p>I was a “nanny”.
I was making decisions by saying,
“I know how to do it in the more general case.
But, I will not let programmers do it because
they are immature.”
This nanny control is not necessarily a good thing.
I am of two minds here.
I am trying to not be a nanny here.
I’m trying to show you the spectrum.
Sometimes you have to sort linked lists because you have no extra memory.
It doesn’t happen often.
It probably will never happen in your life.
But, it just might for at least one of you.</p><a href="#fnref:5" rev="footnote">↩</a></li>
<li id="fn:6">
Alex: Because they threw out that instruction, people like
<a href="https://en.wikipedia.org/wiki/Herb_Sutter">Herb Sutter</a> use to recommend to the world
to use <a href="https://en.cppreference.com/w/cpp/container/deque"><code>std::deque</code></a> (see <a href="http://www.gotw.ca/gotw/054.htm">“Using Vector and Deque”</a>).
I’m not making it up.
He thought that it’s better because it supports more operations.
He was wrong and I wrote both <code>std::vector</code> and <code>std::deque</code>.<a href="#fnref:6" rev="footnote">↩</a></li>
</ol>
</div>
<span style="float: right">
[
<a href="17_adaptive_merge_sort.html">previous</a>,
<a href="index.html">contents</a>,
<a href="19_insertion_sort.html">next</a>
]
</span>
</body>
</html>