-
Notifications
You must be signed in to change notification settings - Fork 0
/
2108.10341.tei.xml
627 lines (605 loc) · 63.3 KB
/
2108.10341.tei.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
<?xml version="1.0" encoding="UTF-8"?>
<TEI xml:space="preserve" xmlns="http://www.tei-c.org/ns/1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.tei-c.org/ns/1.0 https://raw.githubusercontent.com/kermitt2/grobid/master/grobid-home/schemas/xsd/Grobid.xsd"
xmlns:xlink="http://www.w3.org/1999/xlink">
<teiHeader xml:lang="en">
<fileDesc>
<titleStmt>
<title level="a" type="main">Query Embedding Pruning for Dense Retrieval</title>
</titleStmt>
<publicationStmt>
<publisher/>
<availability status="unknown"><licence/></availability>
<date type="published" when="2021-08-23">23 Aug 2021</date>
</publicationStmt>
<sourceDesc>
<biblStruct>
<analytic>
<author>
<persName coords="1,141.14,113.22,84.50,11.96"><forename type="first">Nicola</forename><surname>Tonellotto</surname></persName>
<email>nicola.tonellotto@unipi.it</email>
<affiliation key="aff0">
<orgName type="institution">University of Pisa</orgName>
<address>
<country key="IT">Italy</country>
</address>
</affiliation>
</author>
<author>
<persName coords="1,387.77,113.22,83.67,11.96"><forename type="first">Craig</forename><surname>Macdonald</surname></persName>
<email>craig.macdonald@glasgow.ac.uk</email>
<affiliation key="aff1">
<orgName type="institution">University of Glasgow</orgName>
<address>
<country key="GB">UK</country>
</address>
</affiliation>
</author>
<title level="a" type="main">Query Embedding Pruning for Dense Retrieval</title>
</analytic>
<monogr>
<imprint>
<date type="published" when="2021-08-23">23 Aug 2021</date>
</imprint>
</monogr>
<idno type="MD5">60E23CD5D5B1DACAA7480C4498E79FAC</idno>
<idno type="arXiv">arXiv:2108.10341v1[cs.IR]</idno>
</biblStruct>
</sourceDesc>
</fileDesc>
<encodingDesc>
<appInfo>
<application version="0.7.2" ident="GROBID" when="2022-12-25T09:40+0000">
<desc>GROBID - A machine learning software for extracting information from scholarly documents</desc>
<ref target="https://github.com/kermitt2/grobid"/>
</application>
</appInfo>
</encodingDesc>
<profileDesc>
<textClass>
<keywords>
<term>CCS CONCEPTS</term>
<term>Information systems → Information retrieval</term>
<term>Information retrieval query processing</term>
<term>Retrieval models and ranking Query processing</term>
<term>Dynamic pruning</term>
<term>Dense retrieval</term>
</keywords>
</textClass>
<abstract>
<div xmlns="http://www.tei-c.org/ns/1.0"><p>Recent advances in dense retrieval techniques have offered the promise of being able not just to re-rank documents using contextualised language models such as BERT, but also to use such models to identify documents from the collection in the first place. However, when using dense retrieval approaches that use multiple embedded representations for each query, a large number of documents can be retrieved for each query, hindering the efficiency of the method. Hence, this work is the first to consider efficiency improvements in the context of a dense retrieval approach (namely ColBERT), by pruning query term embeddings that are estimated not to be useful for retrieving relevant documents. Our proposed query embeddings pruning reduces the cost of the dense retrieval operation, as well as reducing the number of documents that are retrieved and hence require to be fully scored. Experiments conducted on the MSMARCO passage ranking corpus demonstrate that, when reducing the number of query embeddings used from 32 to 3 based on the collection frequency of the corresponding tokens, query embedding pruning results in no statistically significant differences in effectiveness, while reducing the number of documents retrieved by 70%. In terms of mean response time for the end-to-end to end system, this results in a 2.65× speedup.</p></div>
</abstract>
</profileDesc>
</teiHeader>
<facsimile>
<surface n="1" ulx="0.0" uly="0.0" lrx="612.0" lry="792.0"/>
<surface n="2" ulx="0.0" uly="0.0" lrx="612.0" lry="792.0"/>
<surface n="3" ulx="0.0" uly="0.0" lrx="612.0" lry="792.0"/>
<surface n="4" ulx="0.0" uly="0.0" lrx="612.0" lry="792.0"/>
<surface n="5" ulx="0.0" uly="0.0" lrx="612.0" lry="792.0"/>
</facsimile>
<text xml:lang="en">
<body>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">INTRODUCTION</head><p>Pretrained contextualised language models such as BERT <ref type="bibr" coords="1,269.50,566.36,10.68,8.97" target="#b1">[2]</ref> are able to successfully exploit general language features in order to capture the contextual semantic signals allowing to better estimate the relevance of documents w.r.t. a given query, leading to effective search ranking improvements when re-ranking the documents obtained from a classical inverted index <ref type="bibr" coords="1,190.57,621.16,9.30,8.97" target="#b6">[7]</ref>. Recently, representationfocused models have gained attention due to their ability to capture CIKM '21, November 1-5, 2021, Virtual Event, QLD, Australia © 2021 Association for Computing Machinery. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM '21), November 1-5, 2021, Virtual Event, QLD, Australia, https://doi.org/10.1145/3459637. 3482162. semantic information and because they allow to pre-compute document representations at indexing time, greatly reducing the query processing times <ref type="bibr" coords="1,380.86,181.28,9.29,8.97" target="#b4">[5,</ref><ref type="bibr" coords="1,392.38,181.28,6.12,8.97" target="#b8">9,</ref><ref type="bibr" coords="1,400.73,181.28,10.29,8.97" target="#b9">10,</ref><ref type="bibr" coords="1,413.26,181.28,10.09,8.97" target="#b17">18]</ref>. Representation-focused models aim at learning a function mapping a sequence of tokens, e.g., a query or a document, into one or more real-valued vectors called embeddings. These embeddings are then combined to compute a similarity score between the query and the document. Inspired by distributional word embeddings <ref type="bibr" coords="1,386.82,236.07,13.49,8.97" target="#b13">[14]</ref>, many works have adopted some pooling technique such as max-and mean-pooling to generate a single representation for a sequence of terms. However, the use of a single representation in effect compresses all possible semantic facets of the given text into a single vector. More recently, multiple representations, composed by a list of embeddings, one per term in the sequence, have been investigated <ref type="bibr" coords="1,444.74,301.82,9.44,8.97" target="#b3">[4,</ref><ref type="bibr" coords="1,456.64,301.82,6.18,8.97" target="#b4">5,</ref><ref type="bibr" coords="1,465.27,301.82,6.29,8.97" target="#b7">8]</ref>. In this case, there is a similarity score between every query term embedding and every document term embedding, and the pooling is performed when all similarity scores have been computed. Most neural ranking approaches have been used to by re-rank the documents identified by a classical inverted index using relevance models such as BM25, in a multi-stage ranking architecture <ref type="bibr" coords="1,436.24,367.58,9.23,8.97" target="#b5">[6,</ref><ref type="bibr" coords="1,447.61,367.58,10.05,8.97" target="#b12">13]</ref>. However, lexical matching models relying solely on an inverted index may not identify the contextually related candidate documents that would have been highly scored by an effective neural ranking model. Instead, by utilising documents encoded as vectors at indexing time and queries encoded as vectors at query processing time, dense retrieval approaches <ref type="bibr" coords="1,352.37,433.33,9.25,8.97" target="#b4">[5,</ref><ref type="bibr" coords="1,363.86,433.33,11.48,8.97" target="#b16">17]</ref> are of growing interest. In dense retrieval, the topranked documents for a given query are computed by identifying the most similar document embeddings to the given query embeddings, employing a nearest neighbour search procedure. Nearest neighbour search with single representations has been shown to be efficient, but less effective than multiple representations <ref type="bibr" coords="1,532.58,488.13,9.30,8.97" target="#b6">[7]</ref>. On the other hand, when multiple representations are exploited, as pioneered by Khattab and Zaharia <ref type="bibr" coords="1,442.21,510.04,9.27,8.97" target="#b4">[5]</ref>, a multi-stage dense retrieval approach can be executed, where the first stage conducts an approximate but highly efficient nearest neighbour search, retrieving documents to be exactly scored by the second stage.</p><p>However, using multiple representations for dense retrieval results in a first-stage that retrieves many documents. These documents are then re-ranked in the second stage, and the time spent in re-ranking is directly proportional to the number of documents retrieved by the first stage. Hence, the time taken to score all retrieved documents can be expensive. Instead, this paper proposes the adaptation of dynamic pruning strategies for dense retrieval. The retrieval of the most similar documents through ANN can be seen as a term-at-a-time (TAAT) set retrieval. TAAT retrieval for classical best match weighting models (such as BM25) involves the processing of all the documents in which a specific query term appears, to compute a query-document score contribution stored in each document's score accumulator <ref type="bibr" coords="1,450.31,685.39,13.33,8.97" target="#b15">[16]</ref>. Dynamic pruning strategies have been proposed to reduce the number of accumulators being created or updated. In the case of TAAT Quit dynamic pruning strategies, query processing terminates after a certain number of query terms have been processed and the top 𝑘 documents are selected from the documents processed thus far. In our query embedding pruning approach, we are inspired by TAAT Quit, by proposing to estimate which multiple query embeddings are useful for dense retrieval. Indeed, by pruning out less useful query embeddings, we can conduct faster approximate nearest neighbour search, reduce the number of documents that are retrieved by the first stage dense retrieval, and obtain a faster second-stage scoring.</p><p>In summary, this work contributes (i) a first examination of the usefulness of different query embeddings in multiple representation dense retrieval, and (ii) the novel proposition of dynamic pruning of query embeddings for dense retrieval. In particular, our experiments conducted on the MSMARCO passage ranking corpus demonstrate that, for example, when reducing the number of query embeddings used from 32 to 3, our query embedding pruning approach results in no statistically significant differences in effectiveness, while reducing the number of documents retrieved by 70%.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">DENSE RETRIEVAL</head><p>We assume that queries and documents are sequences of terms from a given vocabulary 𝑉 . Any term is represented by a real-valued vector of dimension 𝑑, called an embedding. More formally, let }. Given two embeddings, their similarity is computed by the dot product. Hence, for a query 𝑞 and a document 𝑑, their final similarity score 𝑠 (𝑞, 𝑑) is obtained by summing up the maximum similarity between the query token embeddings and document token embeddings:</p><formula xml:id="formula_0" coords="2,54.25,353.06,77.40,8.97">𝑓 𝑄 : 𝑉 𝑛 → R 𝑛×𝑑 be</formula><formula xml:id="formula_1" coords="2,124.95,497.89,169.10,27.65">𝑠 (𝑞, 𝑑) = |𝑞 | ∑︁ 𝑖=1 max 𝑗=1,..., |𝑑 | 𝜙 𝑇 𝑖 𝜓 𝑗<label>(1)</label></formula><p>The document embeddings from all documents in the collection are pre-computed through the application of the 𝑓 𝐷 learned function and stored into an index data structure for vectors supporting similarity searches. This can identify the closest vectors to a given input vector leveraging with cosine or dot product vector comparisons. Query token embeddings are computed at runtime leveraging the 𝑓 𝑄 learned function; queries may also be augmented with additional masked tokens to provide "a soft, differentiable mechanism for learning to expand queries with new terms or to re-weigh existing terms based on their importance for matching the query" <ref type="bibr" coords="2,257.45,627.45,9.39,8.97" target="#b4">[5]</ref>. <ref type="foot" coords="2,269.98,625.47,3.38,7.27" target="#foot_0">1</ref>In order to reduce the time required to compute the similarities between query and document embeddings using Eq. (1), it is possible to shift from an exact to an approximate nearest neighbour (ANN) search. With ANN, the document embeddings are stored in a quantised form, suitable for fast searching. However, the approximate similarity scores between these compressed embeddings are inaccurate, and hence are not used for computing the final top documents. Indeed, ANN search computes, for each query embedding </p><formula xml:id="formula_2" coords="2,346.93,138.08,4.50,5.01">= " > A A A B 9 X i c b V C 7 T s N A E F y H V w i v A C X N i Q g p V W S j I C g j 0 V A m g j y k x I r O l 0 0 4 5 f z Q 3 R o U R f k E W q j o E C 3 f Q 8 G / Y B s X k D D V a G Z X O z t e p K Q h 2 / 6 0 C m v r G 5 t b x e 3 S z u 7 e / k H 5 8 K h j w l g L b I t Q h b r n c Y N K B t g m S Q p 7 k U b u e w q 7 3 v Q 6 9 b s P q I 0 M g z u a R e j 6 f B L I s R S c E u l 2 P G w N y x W 7 Z m d g q 8 T J S Q V y N I f l r 8 E o F L G P A Q n F j e k 7 d k T u n G u S Q u G i N I g N R l x M + Q T 7 C Q 2 4 j 8 a d Z 1 E X 7 C w 2 n E I W o W Z S s U z E 3 x t z 7 h s z 8 7 1 k 0 u d 0 b 5 a 9 V P z P 6 8 c 0 v n L n M o h i w k C k h 0 g q z A 4 Z o W X S A b K R 1 E j E 0 + T I Z M A E 1 5 w I t W R c i E S M k 1 J K S R / O 8 v e r p H N e c + q 1 i 1 a 9 0 q j m z R T h B E 6 h C g 5 c Q g N u o A l t E D C B J 3 i G F + v R e r X e r P e f 0 Y K V 7 x z D H 1 g f 3 8 K r k i E = < / l a t e x i t ></formula><p>|q| < l a t e x i t s h a 1 _ b a s e 6 4 = " A o 7 w q G S p v h P P 5 𝜙 𝑖 , the set Ψ(𝜙 𝑖 , 𝑘 ) of the 𝑘 ′ document embeddings most similar to 𝜙 𝑖 according to some approximate distance; then, these document embeddings are mapped back to their documents 𝐷 𝑖 (𝑘 ′ ):</p><formula xml:id="formula_3" coords="2,389.31,106.51,7.61,51.38">t P E M A a T W Y h t d T M = " > A A A B / H i c b V C 7 T s N A E D y H V w i v A C X N i Q i J K r J R E J S R a C i D R B 4 i s a L z Z Z O c c j 6 b u z V S 5 I S v o I W K D t H y L x T 8 C 7 Z x A Q l T j W Z 2 t b P j h V I Y t O 1 P q 7 C y u r a + U d w s b W 3 v 7 O 6 V 9 w 9 a J o g 0 h y Y P Z K A 7 H j M g h Y I m C p T Q C T U w 3 5 P Q 9 i Z X q d 9 + A G 1 E o G 5 x G o L r s 5 E S Q 8 E Z J t J d L x y L f j y 7 n 8 3 7 5 Y p d t T P Q Z e L k p E J y N P r l r 9 4 g 4 J E P C r l k x n Q d O 0 Q 3 Z h o F l z A v 9 S I D I e M T N o J u Q h X z w b h x l n h O T y L D M K A h a C o k z U T 4 v R E z 3 5 i p 7 y W T P s O x W f R S 8 T + v G + H w 0 o 2 F C i M E x d N D K C R k h w z X I q k C 6 E B o Q G R p c q B C U c 4 0 Q w Q t K O M 8 E a O k m 1 L S h 7 P 4 / T J p n V W d W v X 8 p l a p 0 7 y Z I j k i x + S U O O S C 1 M k 1 a Z A m 4 U S R J / J M X q x H 6 9 V 6 s 9 5 / R g t W v n N I / s D 6 + A b w j Z W m < / l a t e x i t > 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " j n 0 I W t C p 5 3 I w W u M D 2 d M P y x s L k L E = " > A A A B + n i c b V C 7 T s N A E D y H V w i v A C X N i Q i J K r J R E J S R a C i D R B 5 S Y k X n y y Y 5 5 X w + 3 a 2 R I p O f o I W K D t H y M x T 8 C 7 Z x A Q l T j W Z 2 t b M T a C k s u u 6 n U 1 p b 3 9 j c K m 9 X d n b 3 9 g + q h 0 c d G 8 W G Q 5 t H M j K 9 g F m Q Q k E b B U r o a Q M s D C R 0 g 9 l N 5 n c f w F g R q X u c a / B D N l F i L D j D V O o N 9 F Q M E 2 8 x r N b c u p u D r h K v I D V S o D W s f g 1 G E Y 9 D U M g l s 7 b v u R r 9 h B k U X M K i M o g t a M Z n b A L 9 l C o W g v W T P O + C n s W W Y U Q 1 G C o k z U X 4 v Z G w 0 N p 5 G K S T I c O p X f Y y 8 T + v H + P 4 2 k + E 0 j G C 4 t k h F B L y Q 5 Y b k R Y B d C Q M I L I s O V C h K G e G I Y I R l H G e i n H</formula><formula xml:id="formula_4" coords="2,390.48,127.45,2.49,4.38">s E R 1 m s K u Y = " > A A A B + n i c b V C 7 T s N A E D z z D O E V o K Q 5 E S F R R X Y U B G U k G s o g k Y e U W N H 5 s k l O O Z 9 P d 2 u k y O Q n a K G i Q 7 T 8 D A X / g m 1 c Q M J U o 5 l d 7 e w E W g q L r v v p r K 1 v b G 5 t l 3 b K u 3 v 7 B 4 e V o + O O j W L D o c 0 j G Z l e w C x I o a C N A i X 0 t A E W B h K 6 w e w m 8 7 s P Y K y I 1 D 3 O N f g h m y g x F p x h K v U G e i q G S X 0 x r F T d m p u D r h K v I F V S o D W s f A 1 G E Y 9 D U M g l s 7 b v u R r 9 h B k U X M K i P I g t a M Z n b A L 9 l C o W g v W T P O + C n s e W Y U Q 1 G C o k z U X 4 v Z G w 0 N p 5 G K S T I c O p X f Y y 8 T + v H + P 4 2 k + E 0 j G C 4 t k h F B L y Q 5 Y b k R Y B d C Q M I L I s O V C h K G e G I Y I R l H G e i n H a T D n t w 1 v + f p V 0 6 j W v U b u 8 a 1 S b t G i m R E 7 J G b k g H r k i T X J L W q R N O J H k i T y T F + f R e X X e n P e f 0 T W n 2 D k h f + B 8 f A O 6 L J R b < / l a t</formula><formula xml:id="formula_5" coords="2,517.51,101.34,18.15,14.37">D1(k 0 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " o B L 5 H Q E O m h w p N O J O v X M k h 9 j 1 b + A = " > A A A B + 3 i c b V C 7 T s N A E D y H V w i v A C X N i Q g R m s h G Q V B G g o I y S O S B E i s 6 X z b h l D v b u l s j R Z a / g h Y q O k T L x 1 D w L 9 j G B Q S m G s 3 s a m f H C 6 U w a N s f V m l p e W V 1</formula><formula xml:id="formula_6" coords="2,511.01,109.46,18.15,17.37">q I w L / F e Q i u Y l N f T A R n m E p 3 V 6 P Y S e q z 4 5 N R t W Y 3 7 B z 0 L 3 E K U i M F 2 q P q 5 3 A c 8 E i B j 1 w y Y w a O H a I b M 4 2 C S 0 g q w 8 h A y P i M T W G Q U p 8 p M G 6 c B 0 7 o U W Q Y B j Q E T Y W k u Q g / N 2 K m j J k r L 5 1 U D O / N o p e J / 3 m D C C c X b i z 8 M E L w e X Y I h Y T 8 k O F a p E 0 A H Q s N i C x L D l T 4 l D P N E E E L y j h P x S i t p p L 2 4 S x + / 5 d 0 T x t O s 3 F 2 0 6 y 1 a N F M m R y Q Q 1 I n D j k n L X J N 2 q R D O F H k k T y R Z y u x X q x X 6 + 1 7 t G Q V O / v k F 6 z 3 L y a U k + 4 = < / l a t e x i t > D2(k 0 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " b H 5 b 1 N E N T s e y f L b k A u p N 1 F z q Z R 0 = " > A A A B + 3 i c b V C 7 T s N A E D y H V w i v A C X N i Q g R m s i O g q C M B A V l k M g D J V F 0 v m z C K X e 2 d b d G i i x / B S 1 U d I i W j 6 H g X 7 C N C 0 i Y a j S z q 5 0 d N 5 D C o G 1 / W o W V 1 b X 1 j e J m a W t 7 Z 3 e v v H / Q M X</formula><formula xml:id="formula_7" coords="2,511.01,120.57,2.43,6.25">C f W Y A j O M s s A x P Q k N Q 5 8 G o K m Q N B P h 9 0 b E l D F z 5 S a T i u G D W f R S 8 T + v H + L k c h g J L w g R P J 4 e Q i E h O 2 S 4 F k k T Q M d C A y J L k w M V H u V M M 0 T Q g j L O E z F M q i k l f T i L 3 y + T T r 3 m N G r n t 4 1 K k + b N F M k R O S Z V 4 p A L 0 i Q 3 p E X</formula><formula xml:id="formula_8" coords="2,518.65,154.45,2.43,6.25">P i a V t I L 4 i L 3 y k o = " > A A A B / X i c b V C 7 T s N A E D y H V w i v A C X N i Q g R m s h G Q V B G g o I y S O Q h O V Z 0 v m z C K e c</formula><formula xml:id="formula_9" coords="2,495.98,213.89,3.92,7.53">K o = " > A A A B 8 3 i c b V C 7 T s N A E D y H V w i v A C X N i Y B E F d k o C M p I N J S J R B 5 S Y k X n y y a c c j 5 b d 3 t I k Z U v o I W K D t H y Q R T 8 C 7 Z x A Q l T j W Z 2 t b M T x F I Y d N 1 P p 7 S 2 v r G 5 V d 6 u 7 O z u 7 R 9 U D 4 + 6 J r K a Q 4 d H M t L 9 g B m Q Q k E H B U r o x x p Y G E j o B b P b z O 8 9 g j Y i U v c 4 j 8 E P 2 V S J i e A M U 6 k 9 G 1 V r b t 3 N Q V e J V 5 A a K d A a V b + G 4 4 j b E B R y y Y w Z e G 6 M f s I 0 C i 5 h U R l a A z H j M z a F Q U o V C 8 H 4 S R 5 0 Q c + t Y R j R G D Q V k u Y i / N 5 I W G j M P A z S y Z D h g 1 n 2 M v E / b 2 B x c u M n Q s U W Q f H s E A o J + S H D t U g b A D o W G h B Z l h y o U J Q z z R B B C 8 o 4 T 0 W b V l J J + / C W v 1 8 l 3 c u 6 1 6 h f t R u 1 5 l n R T J m c k F N y Q T x</formula><formula xml:id="formula_10" coords="2,360.06,272.51,155.72,11.09">𝐷 (𝑘 ′ ) = {𝑑 ∈ 𝐷 : 𝑓 𝐷 (𝑑) ∩ Ψ(𝜙 𝑖 , 𝑘 ′ ) ≠ ∅}</formula><p>(2) and finally the union 𝐷 (𝑘 ′ ) of these sets is returned:</p><formula xml:id="formula_11" coords="2,403.57,300.26,154.64,27.65">𝐷 (𝑘 ′ ) = |𝑞 | 𝑖=1 𝐷 𝑖 (𝑘 ′ )<label>(3)</label></formula><p>Once the approximate nearest documents 𝐷 (𝑘 ′ ) have been identified, they are exploited to compute the final list of top 𝑘 documents to be returned. To this end, the set of documents 𝐷 (𝑘 ′ ) is re-ranked using the query embeddings and the documents' multiple embeddings to produce exact scores that determine the final ranking. Figure <ref type="figure" coords="2,353.64,386.39,4.20,8.97" target="#fig_7">1</ref> summarises the dense retrieval architecture described above. At the time of writing, ColBERT <ref type="bibr" coords="2,470.36,397.35,10.68,8.97" target="#b4">[5]</ref> is the only effective dense retrieval system exploiting the multiple representations for queries and documents proposed thus far, exhibiting higher effectiveness than dense retrieval based on single query and document representations such as ANCE <ref type="bibr" coords="2,431.35,441.18,14.72,8.97" target="#b16">[17]</ref> (see <ref type="bibr" coords="2,464.70,441.18,9.33,8.97" target="#b6">[7,</ref><ref type="bibr" coords="2,476.27,441.18,30.39,8.97">Table 27</ref> </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>vs. Table 28]).</head><p>To illustrate the challenges of using the multiple representationfocused dense retrieval, we analyse the performance of the Col-BERT end-to-end dense retrieval system by varying the number of processed query embeddings, summarising the main limitations identified by our analysis. All experiments are conducted on the MSMARCO Passage Ranking corpus; effectiveness results are reported on the 43 TREC 2019 Deep Learning track queries -our experimental setup is detailed later in Section 4.</p><p>ColBERT uses 32 representations for each query. Assuming that we order them in their order of occurrence, we can measure the resulting effectiveness if only a subset of these embedding were used. Figure <ref type="figure" coords="2,363.52,572.69,4.09,8.97" target="#fig_8">2</ref> demonstrates the resulting effectiveness measures in term of normalised discounted cumulative gain (nDCG@10), mean average precision (MAP), mean number of relevant documents retrieved, and mean number of documents retrieved. In each graph, the x-axis represents the number of query embeddings used, based on their order of occurrence: CLS, query tokens, masked tokens.</p><p>From Figure <ref type="figure" coords="2,375.37,638.44,3.10,8.97" target="#fig_8">2</ref>, it can be seen that effectiveness across all three measures (nDCG@10, MAP and number of relevant retrieved) rises as more query embeddings are deployed. With only one query embedding (i.e. CLS), effectiveness on all measures is low<ref type="foot" coords="2,530.14,669.33,3.38,7.27" target="#foot_1">2</ref> . Effectiveness in terms of nDCG@10 and MAP rises quickly before stabilising, with 6-8 embeddings appearing to be sufficient for nDCG@10 (Figure <ref type="figure" coords="3,80.58,191.03,2.96,8.97" target="#fig_8">2</ref>(a)), and 10-12 being sufficient for MAP (Figure <ref type="figure" coords="3,250.74,191.03,12.21,8.97" target="#fig_8">2(b)</ref>). In contrast, the number of relevant documents continues to rise, but more slowly after 12 embeddings (Figure <ref type="figure" coords="3,178.56,212.95,11.66,8.97" target="#fig_8">2(c)</ref>). This increased recall comes at a cost of retrieving more documents, with an average 1000 more documents being retrieved when moving from 15 embeddings to 32 (Figure <ref type="figure" coords="3,81.30,245.82,12.15,8.97" target="#fig_8">2(d)</ref>). Indeed, while the recall of the retrieved document set is increased, it is apparent that these additional relevant documents are not promoted to the top ranks of the final ranking, and hence there is no benefit to the (top heavy) nDCG@10 or MAP measures.</p><p>In short, our analysis shows that not all query embeddings are needed for effective retrieval -indeed, this implies that some query embeddings can be ignored without negatively impacting the effectiveness of the whole system. Moreover, many documents are retrieved by each query embedding -the ramification is that all encoded document embeddings must be stored in memory, as many documents must be scored by the exact ranker to obtain effective results. Finally, less new documents are retrieved by later query (masked) embeddings, as these embeddings are reduntant w.r.t. to earlier embeddings. In order to address the aforementioned limitations, we aim to use less query embeddings for effective and efficient dense retrieval. Indeed, we desire high effectiveness with less query embeddings. In the next section we discuss how dynamic pruning can be exploited in dense retrieval to increase efficiency without negatively impacting effectiveness.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">QUERY EMBEDDING PRUNING</head><p>As we have shown in Section 2, the query embeddings do not contribute equally to the final effectiveness of the retrieved document set. We argue that not every query embedding will bring useful documents for retrieval, even if each query embedding well represents the context of the query term. Hence, we propose to adapt the TAAT Quit dynamic pruning strategy to Equation (3). TAAT query processing, as well as its dynamic pruning strategies, orders the query terms by relative importance, e.g., inverse document frequency, such that the rarest query term was executed first. This is motivated by the fact that in best match weighting models the rarest query terms appearing in a document contribute most to the final document score, compared to more common terms. Similarly, we postulate that the most important query tokens 3 are more likely to bring relevant documents than non-relevant documents, and therefore we propose to prune (remove) the unimportant query embeddings. Indeed, Formal et al. <ref type="bibr" coords="3,73.80,644.34,10.55,8.97" target="#b2">[3]</ref> noted that exact matches and the more important terms contribute more to the overall ColBERT scores; we argue that these terms are those that should be the focus of the ANN search.</p><p>We propose the following query embedding pruning strategy to compute the results of the ANN search in conjunction with Eq. ( <ref type="formula" coords="3,286.38,688.18,2.90,8.97">2</ref>): 3 While we use the notion of terms and tokens, these could be wordpieces as identified by the BERT tokeniser.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>𝐷 (𝑘</head><formula xml:id="formula_12" coords="3,417.85,202.00,140.35,25.67">′ ) = 𝑝 𝑖=1 𝐷 𝑖 (𝑘 ′ ).<label>(4)</label></formula><p>According to Eq. ( <ref type="formula" coords="3,382.72,231.12,2.86,8.97" target="#formula_12">4</ref>), in query embedding pruning, the ANN search does not compute the set of retrieved documents to be re-ranked over all query embeddings, but it only processes the 𝑝 most important query embeddings, and computes the 𝑘 ′ document embeddings most similar to those only. Note that query embeddings are only pruned for the first ANN stage, and are restored for the exact scoring stage. Our approach ignores the less important query embeddings, thus, as a consequence, lesser query embeddings processed in ANN search will generate lesser documents to be re-ranked using all document embeddings. The identification of the most important query tokens requires a concept of ordering among query embeddings.</p><p>A natural way to rank the query embeddings is to order them by ascending order of frequency in the collection of the corresponding query tokens. This is akin to the ordering of query terms in TAAT by IDF. We denote this ranking of query embeddings as Inverse Collection Frequency (ICF) <ref type="foot" coords="3,418.54,393.51,3.38,7.27" target="#foot_2">4</ref> , and postulate that the frequency in the collection of the query token corresponding to a query embedding is inversely proportional to its importance in identifying relevant documents. Note that special tokens such as CLS and the masked tokens do not correspond to any document token, hence they are placed after the query embeddings corresponding to actual wordpieces, CLS then masked tokens.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">EXPERIMENTS</head><p>Our experiments make use the MSMARCO passage ranking dataset and the PyTerrier IR experimentation platform <ref type="bibr" coords="3,487.52,507.69,13.40,8.97" target="#b10">[11,</ref><ref type="bibr" coords="3,503.13,507.69,10.05,8.97" target="#b11">12]</ref>. We use the ColBERT implementation provided by the authors <ref type="foot" coords="3,496.94,516.66,3.38,7.27" target="#foot_3">5</ref> , which we have extended <ref type="foot" coords="3,350.48,527.62,3.38,7.27" target="#foot_4">6</ref> . We follow <ref type="bibr" coords="3,397.20,529.61,10.42,8.97" target="#b4">[5]</ref> for the settings of ColBERT; the resulting document embeddings index is 176 GB. The FAISS ANN index is trained on a randomly selected 5% sample of the document embeddings. The resulting FAISS index is 16 GB. ANN search is performed on the 10 partitions most similar to the given input embeddings. For evaluating effectiveness, we use the available querysets with relevance assessments: the official small version of the MSMARCO Dev set, consisting of 6,980 queries with on average 1.1 judgements per query, as well as the TREC 2019 queryset, which contains 43 queries with an average of 215.3 judgements per query<ref type="foot" coords="3,487.64,626.25,3.38,7.27" target="#foot_5">7</ref> . To measure effectiveness, we employ MRR@10 for the MSMARCO Dev queryset, and the MRR@10, nDCG@10 and MAP for the TREC queryset. We compare our results to the default setting of dense retrieval of ColBERT, using all 32 query embeddings, and retrieving 𝑘 ′ = 1000 documents for each query embedding. In conducting our experiments addressing the efficiency, we determine the success of our query embeddings pruning strategy based on ICF 8 , compared to the baseline approach, called First, by demonstrating that for a fixed number of embeddings, it attains effectiveness that is not significantly different from that of the default setting, while resulting in less documents being retrieved and re-scored by ColBERT's second-stage. The red and blue curves in Figure <ref type="figure" coords="4,197.82,353.14,4.25,8.97" target="#fig_9">3</ref> correspond to selecting query embeddings based on their order in the query (denoted First) and based on collection frequency (denoted ICF). In general, from each of the figures, we can see that first (red) curve always exhibits lower effectiveness with a similar number of documents retrieved. For instance, when using one query embedding, on MSMARCO Dev, First retrieves on average 824 documents and achieves an MRR@10 of 0.2165; in contrast, ICF retrieves less documents (724) and achieves a higher MRR@10 (0.3229). Similar trends are observed for nDCG@10 and MAP on the TREC 2019 query set, where effectiveness is always increased by using ICF compared to First, while retrieving a similar number of documents (689 for First vs. 880 for ICF). Moreover, ICF exhibits less statistically significant differences in effectiveness compared to First.</p><p>In general, the higher effectiveness of ICF over First is apparent for larger number of query embeddings, and effectiveness saturates at less query embeddings retrieving less documents: for instance, for nDCG@10 and MAP on TREC 2019 (Figures <ref type="figure" coords="4,233.74,539.45,10.31,8.97" target="#fig_9">3(a</ref>) and 3(b)), at 𝑝 = 2, ICF reaches the same values as the full ColBERT retrieval with 32 query embeddings, but re-ranking 1367 documents only on average, while First needs at least 𝑝 = 8 query embeddings and 4441 documents on average. On MSMARCO Dev (Figures <ref type="figure" coords="4,241.01,583.28,8.74,8.97" target="#fig_9">3(c</ref>)), MRR@10 reaches the value of the full ColBERT retrieval with just the 𝑝 = 3 query embeddings with the highest ICF score. When reducing 𝑝 from 32 to 3, we decrease the number of documents retrieved by the first stage by 70%, e.g., from ∼7000 to ∼2000. In terms of mean response time for the end-to-end to end system, this results in a reduction from 461.4ms to 173.7ms , i.e., a 2.65× speedup.</p><p>Note that the mean number of masked tokens in both query sets is 22.2, and, correspondingly, the average number of wordpieces per query is 9.8. In First, the query embeddings of the masked tokens appear last, hence their impact on the average effectiveness across all metrics is negligible. In ICF, all special tokens (e.g. CLS & MASK), 8 We do not report results using IDF since they match those obtained by using ICF. appear last; in this case, the query embedding corresponding to the CLS token (on average the ∼10th query embedding under ICF) does not contribute to the final effectiveness if the original query word tokens are used first. Overall, we conclude from Figure <ref type="figure" coords="4,515.92,298.35,4.09,8.97" target="#fig_9">3</ref> that using the collection frequency of the query tokens can be used as indicator of the importance of the corresponding query embeddings, and our proposed query embedding pruning strategy with 3 embeddings can obtain effectiveness results with no statistical significant differences w.r.t the original system using all query embeddings. Indeed, the lower frequency tokens are more discriminative, and their query embeddings retrieve most of the relevant documents. This conclusion aligns well with the use of measures such as IDF in deciding on the most informative query terms for TAAT dynamic pruning.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">CONCLUSIONS</head><p>In this paper, we identified efficiency challenges concerning the use of multiple embedding representations of queries and documents for dense retrieval. We proposed query embedding pruning, and demonstrated that a subset of the original query embeddings can be used for effective retrieval while reducing the number of document requiring to be exactly scored. For example, when reducing the number of query embeddings used from 32 to 3, our approach results in no statistically significant differences in effectiveness, while reducing the number of documents retrieved and fully scored by 70%. In terms of mean response time for the end-to-end to end system, this results in a 2.65× speedup. The results in this paper give rise to several possible direction of future work. The effectiveness of pruning suggests that adapting static pruning <ref type="bibr" coords="4,503.04,556.81,10.68,8.97" target="#b0">[1]</ref> to work on embedding-based document representations before approximate nearest neighbour search may also have potential. Moreover, query embedding pruning can be applied selectively <ref type="bibr" coords="4,484.21,589.68,13.22,8.97" target="#b14">[15]</ref>, with a different number of embeddings selected for different queries.</p></div><figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_0" coords="2,337.95,84.82,27.97,7.15;2,385.78,86.52,15.68,7.15;2,378.25,94.36,30.73,7.15;2,485.58,92.08,54.79,7.15;2,483.91,208.54,46.72,7.15;2,346.75,130.44,8.01,7.69;2,346.93,138.08,4.50,5.01"><head></head><label></label><figDesc>t e x i t s h a 1 _ b a s e 6 4 = " 9 J s m C k 5 3 V M 2 z R o a 0 s P M i I v y H S H I</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_1" coords="2,390.48,110.33,2.49,4.38;2,394.43,123.62,2.49,4.38;2,390.48,127.45,2.49,4.38"><head></head><label></label><figDesc>a T C X t w 1 v + f p V 0 L u p e o 3 5 5 1 6 g 1 a d F M m Z y Q U 3 J O P H J F m u S W t E i b c C L J E 3 k m L 8 6 j 8 + q 8 O e 8 / o y W n 2 D k m f + B 8 f A O 4 n J R a < / l a t e x i t > 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " M p s a y G L K e g s o y g U j g 0</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_2" coords="2,517.51,109.46,2.43,6.25"><head></head><label></label><figDesc>r b x e 2 d j c 2 t 6 p 7 u 5 1 T R B p D h 0 e y E D 3 P W Z A C h 8 6 K F B C P 9 T A l C e h 5 8 0 u M 7 / 3 A N</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_3" coords="2,511.01,120.57,2.43,6.25"><head></head><label></label><figDesc>6 o O b S 5 L 3 3 d c 5 k B K T x o o 0 A J v U A D U 6 6 E r j u 7 S v 3 u I 2 g j f O 8 O 5 w E M F Z t 6 Y i I 4 w 0 S 6 v x 5 F 9 b g 6 O z 0 b l S t 2 z c 5 A l 4 m T k w r J 0 R q V v w Z j n 4 c K P O S S G d N 3 7 A C H E d M o u I S 4 N A g N B I z P 2 B T 6</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_4" coords="2,511.01,120.57,2.43,6.25;2,518.55,146.05,20.97,11.29;2,518.65,154.45,2.43,6.25"><head></head><label></label><figDesc>a h B N F n s g z e b F i 6 9 V 6 s 9 5 / R g t W v n N I / s D 6 + A Y o K J P v < / l a t e x i t > D |q| (k 0 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " s b m c d m d Z u h K q u</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_5" coords="2,518.65,154.45,2.43,6.25;2,495.85,207.98,3.92,7.53;2,495.98,213.89,3.92,7.53"><head></head><label></label><figDesc>H d 2 u k y I n 4 C l q o 6 B A t 3 0 L B v 2 A b F 5 A w 1 W h m V z s 7 b i i F R t P 8 N A p L y y u r a 8 X 1 0 s b m 1 v Z O e X e v r Y N I c W j x Q A a q 6 z I N U v j Q Q o E S u q E C 5 r k S O u 7 4 M v U 7 D 6 C 0 C P x b n I T g e G z k i 6 H g D B P J v u r H 0 / v p r D o + P u m X K 2 b N z E A X i Z W T C s n R 7 J e / e o O A R x 7 4 y C X T 2 r b M E J 2 Y K R R c w q z U i z S E j I / Z C O y E + s w D 7 c R Z 5 B k 9 i j T D g I a g q J A 0 E + H 3 R s w 8 r S e e m 0 x 6 D O / 0 v J e K / 3 l 2 h M M L J x Z + G C H 4 P D 2 E Q k J 2 S H M l k i 6 A D o Q C R J Y m B y p 8 y p l i i K A E Z Z w n Y p S U U 0 r 6 s O a / X y T t 0 5 p V r 5 3 d 1 C s N m j d T J A f k k F S J R c 5 J g 1 y T J m k R T g L y R J 7 J i / F o v B p v x v v P a M H I d / b J H x g f 3 2 C 3 l T o = < / l a t e x i t > k < l a t e x i t s h a 1 _ b a s e 6 4 = " i 0 x N O 4 K / l y x 6 f w / x S l l G H 7 1 6 +</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_6" coords="2,495.98,213.89,3.92,7.53;2,438.66,129.35,26.30,7.15;2,438.89,137.19,25.85,7.15;2,436.45,145.04,30.73,7.15;2,440.45,167.30,23.06,7.15;2,438.24,179.24,27.14,7.15;2,436.45,187.08,30.73,7.15"><head></head><label></label><figDesc>y T Z r k j r R I h 3 A C 5 I k 8 k x f H O q / O m / P + M 1 p y i p 1 j 8 g f O x z d r 2 J F e < / l a t e x i t ></figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_7" coords="2,361.82,224.75,152.53,7.70;2,418.08,96.24,66.67,66.67"><head>Figure 1 :</head><label>1</label><figDesc>Figure 1: Dense retrieval architecture.</figDesc><graphic coords="2,418.08,96.24,66.67,66.67" type="bitmap" /></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_8" coords="3,87.38,169.41,399.73,7.70;3,488.08,166.64,36.54,10.14"><head>Figure 2 :</head><label>2</label><figDesc>Figure 2: TREC 2019 deep learning track effectiveness for various numbers of query embeddings (𝑘 ′ = 1000).</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_9" coords="4,53.80,205.50,504.40,7.70;4,53.52,216.45,504.93,7.70;4,53.80,226.92,504.70,8.20;4,53.62,237.88,31.81,7.86;4,86.40,235.61,323.66,10.47"><head>Figure 3 :</head><label>3</label><figDesc>Figure3: Effectiveness vs. number of retrieved documents as the number of query embeddings 𝑝 is varied for the baseline (First) and our proposed query embedding pruning (ICF). From left to right, each point on each curve is one additional query embedding, i.e. 𝑝 = 1, 2, .., 32. Hollow points denote a statistically significant difference in effectiveness compared to ColBERT 𝑝 = 32, 𝑘 ′ = 1000, according to a paired t-test with Bonferroni correction (p-value < 0.05).</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" type="table" xml:id="tab_0" coords="2,53.08,353.06,242.48,98.08"><head></head><label></label><figDesc>a learned function mapping a given query term 𝑡 𝑖 in a query of 𝑛 terms to the query embedding 𝜙 𝑖 , i.e., {𝜙 1 , . . . , 𝜙 𝑛 } = 𝑓 𝑄 (𝑡 1 , . . . , 𝑡 𝑛 ). Similarly, let 𝑓 𝐷 : 𝑉 𝑛 → R 𝑛×𝑑 be a (potentially different) learned function mapping a given document term 𝑡 𝑗 in a document of 𝑛 terms to the document embedding 𝜓 𝑗 , i.e., {𝜓 1 , . . . ,𝜓 𝑛 } = 𝑓 𝐷 (𝑡 1 , . . . , 𝑡 𝑛 ). Hence, a query 𝑞 composed by |𝑞| tokens is represented by |𝑞| query embeddings {𝜙 1 , . . . , 𝜙 |𝑞 | }. Analogously, a given document 𝑑 composed by |𝑑 | tokens is represented by |𝑑 | document embeddings {𝜓 1 , . . . ,𝜓 |𝑑 |</figDesc><table /></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" type="table" xml:id="tab_1" coords="2,333.52,101.56,138.88,30.27"><head></head><label></label><figDesc>e x i t ></figDesc><table coords="2,333.52,101.56,138.88,29.51"><row><cell></cell><cell>Approximate</cell></row><row><cell>Learned query representation</cell><cell>nearest neighbour search</cell></row></table></figure>
<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="1" xml:id="foot_0">In current practice<ref type="bibr" coords="2,114.91,734.99,7.31,6.97" target="#b4">[5]</ref>, queries are augmented up to 32 query token embeddings.</note>
<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="2" xml:id="foot_1">This is probably as the CLS embedding encodes less information in ColBERT than in a single-representation dense retrieval approach such as ANCE</note>
<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="4" xml:id="foot_2">Collection frequency is usually correlated with document frequency. Indeed, in our initial experiments we find that ICF and IDF result in almost identical orderings of query terms, and hence only ICF is reported.</note>
<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="5" xml:id="foot_3"><ref type="bibr" coords="3,409.68,704.14,2.55,5.48" target="#b4">5</ref> https://github.com/stanford-futuredata/</note>
<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="6" xml:id="foot_4">ColBERT 6 https://github.com/terrierteam/</note>
<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="7" xml:id="foot_5">pyterrier_colbert<ref type="bibr" coords="3,481.06,713.55,2.55,5.48" target="#b6">7</ref> Additional experiments conducted on TREC 2020 confirmed our results.</note>
</body>
<back>
<div type="acknowledgement">
<div xmlns="http://www.tei-c.org/ns/1.0"><head>ACKNOWLEDGEMENTS</head><p>Nicola Tonellotto was partially supported by the Italian Ministry of Education and Research (MIUR) in the framework of the CrossLab project (Departments of Excellence). Craig Macdonald acknowledges EPSRC grant EP/R018634/1: Closed-Loop Data Science for Complex, Computationally-& Data-Intensive Analytics.</p></div>
</div>
<div type="references">
<listBibl>
<biblStruct coords="4,333.39,716.06,225.58,6.97;4,333.19,724.03,225.01,6.97;5,69.23,88.42,113.56,6.97" xml:id="b0">
<analytic>
<title level="a" type="main">Static index pruning for information retrieval systems</title>
<author>
<persName coords=""><forename type="first">David</forename><surname>Carmel</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Doron</forename><surname>Cohen</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Ronald</forename><surname>Fagin</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Eitan</forename><surname>Farchi</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Michael</forename><surname>Herscovici</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Yoelle</forename><forename type="middle">S</forename><surname>Maarek</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Aya</forename><surname>Soffer</surname></persName>
</author>
<idno type="DOI">10.1145/383952.383958</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '01</title>
<meeting>the 24th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '01</meeting>
<imprint>
<publisher>ACM Press</publisher>
<date type="published" when="2001">2001</date>
<biblScope unit="page" from="43" to="50" />
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,69.23,96.39,225.64,6.97;5,69.23,104.36,224.81,6.97;5,69.23,112.96,38.22,6.23" xml:id="b1">
<analytic>
<title level="a" type="main"></title>
<author>
<persName coords=""><forename type="first">Jacob</forename><surname>Devlin</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Ming-Wei</forename><surname>Chang</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Kenton</forename><surname>Lee</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Kristina</forename><surname>Toutanova</surname></persName>
</author>
<idno type="DOI">10.18653/v1/n19-1423</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the 2019 Conference of the North</title>
<meeting>the 2019 Conference of the North</meeting>
<imprint>
<publisher>Association for Computational Linguistics</publisher>
<date type="published" when="2019">2019</date>
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,69.23,120.30,224.81,6.97;5,69.23,128.27,139.94,6.97" xml:id="b2">
<analytic>
<title level="a" type="main">A White Box Analysis of ColBERT</title>
<author>
<persName coords=""><forename type="first">Thibault</forename><surname>Formal</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Benjamin</forename><surname>Piwowarski</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Stéphane</forename><surname>Clinchant</surname></persName>
</author>
<idno type="DOI">10.1007/978-3-030-72240-1_23</idno>
</analytic>
<monogr>
<title level="m">Lecture Notes in Computer Science</title>
<imprint>
<publisher>Springer International Publishing</publisher>
<date type="published" when="2021">2021</date>
<biblScope unit="page" from="257" to="263" />
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,69.23,136.24,225.99,6.97;5,69.23,144.21,224.81,6.97;5,68.99,152.18,134.53,6.97" xml:id="b3">
<analytic>
<title level="a" type="main">Polyencoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring</title>
<author>
<persName coords=""><forename type="first">Samuel</forename><surname>Humeau</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Kurt</forename><surname>Shuster</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Marie-Anne</forename><surname>Lachaux</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">J</forename><surname>Weston</surname></persName>
</author>
</analytic>
<monogr>
<title level="m">Proc. ICLR</title>
<meeting>ICLR</meeting>
<imprint>
<date type="published" when="2020">2020</date>
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,69.23,160.15,224.81,6.97;5,69.23,168.12,215.95,6.97" xml:id="b4">
<analytic>
<title level="a" type="main">ColBERT</title>
<author>
<persName coords=""><forename type="first">Omar</forename><surname>Khattab</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Matei</forename><surname>Zaharia</surname></persName>
</author>
<idno type="DOI">10.1145/3397271.3401075</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval</title>
<meeting>the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval</meeting>
<imprint>
<publisher>ACM</publisher>
<date type="published" when="2020-07-25">2020</date>
<biblScope unit="page" from="39" to="48" />
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,69.23,176.09,224.81,6.97;5,69.03,184.06,18.66,6.97" xml:id="b5">
<analytic>
<title level="a" type="main">The neural hype, justified!</title>
<author>
<persName coords=""><forename type="first">Jimmy</forename><surname>Lin</surname></persName>
</author>
<idno type="DOI">10.1145/3458553.3458563</idno>
</analytic>
<monogr>
<title level="j">ACM SIGIR Forum</title>
<title level="j" type="abbrev">SIGIR Forum</title>
<idno type="ISSN">0163-5840</idno>
<imprint>
<biblScope unit="volume">53</biblScope>
<biblScope unit="issue">2</biblScope>
<biblScope unit="page" from="88" to="93" />
<date type="published" when="2019-12">2019. 2019</date>
<publisher>Association for Computing Machinery (ACM)</publisher>
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,69.23,192.03,224.81,6.97;5,69.23,200.00,156.02,6.97" xml:id="b6">
<monogr>
<author>
<persName coords=""><forename type="first">Jimmy</forename><surname>Lin</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Rodrigo</forename><surname>Nogueira</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Andrew</forename><surname>Yates</surname></persName>
</author>
<idno type="arXiv">arXiv:2010.06467</idno>
<title level="m">Pretrained Transformers for Text Ranking: BERT and Beyond</title>
<imprint>
<date type="published" when="2020">2020</date>
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,69.23,207.97,225.58,6.97;5,69.23,215.94,222.98,6.97" xml:id="b7">
<analytic>
<title level="a" type="main">Sparse, Dense, and Attentional Representations for Text Retrieval</title>
<author>
<persName coords=""><forename type="first">Yi</forename><surname>Luan</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Jacob</forename><surname>Eisenstein</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Kristina</forename><surname>Toutanova</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Michael</forename><surname>Collins</surname></persName>
</author>
<idno type="DOI">10.1162/tacl_a_00369</idno>
</analytic>
<monogr>
<title level="j">Transactions of the Association for Computational Linguistics</title>
<idno type="ISSNe">2307-387X</idno>
<imprint>
<biblScope unit="volume">9</biblScope>
<biblScope unit="page" from="329" to="345" />
<date type="published" when="2020">2020</date>
<publisher>MIT Press - Journals</publisher>
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,69.23,223.91,224.81,6.97;5,69.23,231.88,225.99,6.97;5,69.23,239.85,200.04,6.97" xml:id="b8">
<analytic>
<title level="a" type="main">Efficient Document Re-Ranking for Transformers by Precomputing Term Representations</title>
<author>
<persName coords=""><forename type="first">Sean</forename><surname>Macavaney</surname></persName>
</author>
<author>
<persName><forename type="first">Franco</forename><forename type="middle">Maria</forename><surname>Nardini</surname></persName>
</author>
<author>
<persName><forename type="first">Raffaele</forename><surname>Perego</surname></persName>
</author>
<author>
<persName><forename type="first">Nicola</forename><surname>Tonellotto</surname></persName>
</author>
<author>
<persName><forename type="first">Nazli</forename><surname>Goharian</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Ophir</forename><surname>Frieder</surname></persName>
</author>
<idno type="DOI">10.1145/3397271.3401093</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval</title>
<meeting>the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval</meeting>
<imprint>
<publisher>ACM</publisher>
<date type="published" when="2020-07-25">2020</date>
<biblScope unit="page" from="49" to="58" />
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,69.23,247.82,224.81,6.97;5,69.23,255.79,224.81,6.97;5,69.23,263.76,126.93,6.97" xml:id="b9">
<analytic>
<title level="a" type="main">Expansion via Prediction of Importance with Contextualization</title>
<author>
<persName coords=""><forename type="first">Sean</forename><surname>Macavaney</surname></persName>
</author>
<author>
<persName><forename type="first">Franco</forename><forename type="middle">Maria</forename><surname>Nardini</surname></persName>
</author>
<author>
<persName><forename type="first">Raffaele</forename><surname>Perego</surname></persName>
</author>
<author>
<persName><forename type="first">Nicola</forename><surname>Tonellotto</surname></persName>
</author>
<author>
<persName><forename type="first">Nazli</forename><surname>Goharian</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Ophir</forename><surname>Frieder</surname></persName>
</author>
<idno type="DOI">10.1145/3397271.3401262</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval</title>
<meeting>the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval</meeting>
<imprint>
<publisher>ACM</publisher>
<date type="published" when="2020-07-25">2020</date>
<biblScope unit="page" from="1573" to="1576" />
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,333.39,88.42,224.81,6.97;5,333.39,96.39,176.68,6.97" xml:id="b10">
<analytic>
<title level="a" type="main">Declarative Experimentation in Information Retrieval using PyTerrier</title>
<author>
<persName coords=""><forename type="first">Craig</forename><surname>Macdonald</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Nicola</forename><surname>Tonellotto</surname></persName>
</author>
<idno type="DOI">10.1145/3409256.3409829</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval</title>
<meeting>the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval</meeting>
<imprint>
<publisher>ACM</publisher>
<date type="published" when="2020-09-14">2020</date>
<biblScope unit="page" from="161" to="168" />
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,333.39,104.36,225.88,6.97;5,333.39,112.33,225.88,6.97;5,333.39,120.30,40.49,6.97" xml:id="b11">
<analytic>
<title level="a" type="main">PyTerrier: Declarative Experimentation in Python from BM25 to Dense Retrieval</title>
<author>
<persName coords=""><forename type="first">Craig</forename><surname>Macdonald</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Nicola</forename><surname>Tonellotto</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Sean</forename><surname>Macavaney</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Iadh</forename><surname>Ounis</surname></persName>
</author>
<idno type="DOI">10.1145/3459637.3482013</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the 30th ACM International Conference on Information & Knowledge Management</title>
<meeting>the 30th ACM International Conference on Information & Knowledge Management</meeting>
<imprint>
<publisher>ACM</publisher>
<date type="published" when="2021-10-26">2021</date>
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,333.39,128.27,225.88,6.97;5,333.39,136.24,223.60,6.97" xml:id="b12">
<analytic>
<title level="a" type="main">High accuracy retrieval with multiple nested ranker</title>
<author>
<persName coords=""><forename type="first">Irina</forename><surname>Matveeva</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Chris</forename><forename type="middle">J C</forename><surname>Burges</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Timo</forename><surname>Burkard</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Andy</forename><surname>Laucius</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Leon</forename><surname>Wong</surname></persName>
</author>
<idno type="DOI">10.1145/1148170.1148246</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '06</title>
<meeting>the 29th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '06</meeting>
<imprint>
<publisher>ACM Press</publisher>
<date type="published" when="2006">2006</date>
<biblScope unit="page" from="437" to="444" />
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,333.39,144.21,225.88,6.97;5,333.39,152.18,225.88,6.97;5,333.39,160.15,70.48,6.97" xml:id="b13">
<analytic>
<title level="a" type="main">Distributed Representations of Words and Phrases and their Compositionality</title>
<author>
<persName coords=""><forename type="first">Tomas</forename><surname>Mikolov</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Ilya</forename><surname>Sutskever</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Kai</forename><surname>Chen</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Greg</forename><forename type="middle">S</forename><surname>Corrado</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Jeff</forename><surname>Dean</surname></persName>
</author>
</analytic>
<monogr>
<title level="m">Proc. NIPS</title>
<meeting>NIPS</meeting>
<imprint>
<date type="published" when="2013">2013</date>
<biblScope unit="page" from="3111" to="3119" />
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,333.39,168.12,224.81,6.97;5,333.39,176.09,162.60,6.97" xml:id="b14">
<analytic>
<title level="a" type="main">Efficient and effective retrieval using selective pruning</title>
<author>
<persName coords=""><forename type="first">Nicola</forename><surname>Tonellotto</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Craig</forename><surname>Macdonald</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Iadh</forename><surname>Ounis</surname></persName>
</author>
<idno type="DOI">10.1145/2433396.2433407</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the sixth ACM international conference on Web search and data mining - WSDM '13</title>
<meeting>the sixth ACM international conference on Web search and data mining - WSDM '13</meeting>
<imprint>
<publisher>ACM Press</publisher>
<date type="published" when="2013">2013</date>
<biblScope unit="page" from="63" to="72" />
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,333.39,184.06,224.99,6.97;5,333.39,192.03,224.81,6.97;5,333.39,200.00,93.59,6.97" xml:id="b15">
<analytic>
<title level="a" type="main">Efficient Query Processing for Scalable Web Search</title>
<author>
<persName coords=""><forename type="first">Nicola</forename><surname>Tonellotto</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Craig</forename><surname>Macdonald</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Iadh</forename><surname>Ounis</surname></persName>
</author>
<idno type="DOI">10.1561/1500000057</idno>
</analytic>
<monogr>
<title level="j">Foundations and Trends® in Information Retrieval</title>
<title level="j" type="abbrev">FNT in Information Retrieval</title>
<idno type="ISSN">1554-0669</idno>
<idno type="ISSNe">1554-0677</idno>
<imprint>
<biblScope unit="volume">12</biblScope>
<biblScope unit="issue">4-5</biblScope>
<biblScope unit="page" from="319" to="500" />
<date type="published" when="2018">2018. 2018</date>
<publisher>Now Publishers</publisher>
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,333.39,207.97,225.58,6.97;5,333.28,215.94,225.06,6.97;5,333.39,223.91,190.48,6.97" xml:id="b16">
<analytic>
<title level="a" type="main">Approximate nearest neighbor negative contrastive learning for dense text retrieval</title>
<author>
<persName coords=""><forename type="first">Lee</forename><surname>Xiong</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Chenyan</forename><surname>Xiong</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Ye</forename><surname>Li</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Kwok-Fung</forename><surname>Tang</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Jialin</forename><surname>Liu</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Paul</forename><surname>Bennett</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Junaid</forename><surname>Ahmed</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Arnold</forename><surname>Overwijk</surname></persName>
</author>
</analytic>
<monogr>
<title level="m">Proc. ICLR</title>
<meeting>ICLR</meeting>
<imprint>
<date type="published" when="2021">2021</date>
</imprint>
</monogr>
</biblStruct>
<biblStruct coords="5,333.39,231.88,224.81,6.97;5,333.39,239.85,224.81,6.97;5,333.39,247.82,175.60,6.97" xml:id="b17">
<analytic>
<title level="a" type="main">From Neural Re-Ranking to Neural Ranking</title>
<author>
<persName coords=""><forename type="first">Hamed</forename><surname>Zamani</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Mostafa</forename><surname>Dehghani</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">W</forename><forename type="middle">Bruce</forename><surname>Croft</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Erik</forename><surname>Learned-Miller</surname></persName>
</author>
<author>
<persName coords=""><forename type="first">Jaap</forename><surname>Kamps</surname></persName>
</author>
<idno type="DOI">10.1145/3269206.3271800</idno>
</analytic>
<monogr>
<title level="m">Proceedings of the 27th ACM International Conference on Information and Knowledge Management</title>
<meeting>the 27th ACM International Conference on Information and Knowledge Management</meeting>
<imprint>
<publisher>ACM</publisher>
<date type="published" when="2018-10-17">2018</date>
<biblScope unit="page" from="497" to="506" />
</imprint>
</monogr>
</biblStruct>
</listBibl>
</div>
</back>
</text>
</TEI>