-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.txt
71 lines (59 loc) · 2.91 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
Evaluator - evaluation of effectiveness measures
------------------------------------------------
This is an implementation of an Evaluation algortihms that calculate the below effectiveness measures:
Precision
Recall
P@K - precision at rank K
MAP - Mean Average Precision
NDCG - Normalized Distributive Cumulative Gain
It works on the given result set of a query and the binary relevance measure of the documents in the corpus for the query in the specified format.
Then it generates the result files for each query with the specifed measures like precision, recall,NDCG etc. and a file with the aggreate measures like MAP.
Build
-----
Developed in Java using jdk1.8.0_05.
Compiled for jre8.
No other libraries used in the implementation
Compile and Run
----------------
For windows:
1. execute executeEval.bat
2. Provide the file path of relevance file
e.g. C:/IR/Assignment/Rajul_Kumar_CS6200_HW5/rel_file/cacm.rel
3. Provide the queryID of the query in the relevance file
e.g. 12
4. Provide the file path of the result file
e.g. C:/IR/Assignment/Rajul_Kumar_CS6200_HW5/hw3_results/q1_pos.eval
5. Provide the queryID and file paths of other queries as well or press 'n' to continue for evaluation
6. Provide the location where the result files will be generated
e.g. C:/IR/Assignment/Rajul_Kumar_CS6200_HW5/results
7. The results will be generated the given location
For linux/unix*:
1. execute executeEval.sh
2. Provide the file path of relevance file
e.g. /IR/Assignment/Rajul_Kumar_CS6200_HW5/rel_file/cacm.rel
3. Provide the queryID of the query in the relevance file
e.g. 12
4. Provide the file path of the result file
e.g. /IR/Assignment/Rajul_Kumar_CS6200_HW5/hw3_results/q1_pos.eval
5. Provide the queryID and file paths of other queries as well or press 'n' to continue for evaluation
6. Provide the location where the result files will be generated
e.g /IR/Assignment/Rajul_Kumar_CS6200_HW5/results
7. The results will be generated the given location
* not tested for linux/unix
Alternatively,
The java files Evaluator.java could be compiled from here by:
javac ./src/com/eval/Evaluator.java
This should be executed from this location by:
java com/eval/Evaluator
Provide the paths required as given above.
* Results for the queries on BM25 engine(HW3) is available at ./hw3_results
and the relevance file for CACM corpus at ./rel_file
Results/response of execution
-----------------------------
1. The result file with rank, documentID, document score, relevance level, precision, recal and NDCG score is generated for each query at the given result location in the given sequence
with the file name <queryID>_<MM-dd-yy_HH:mm:ss>.out
2. The result file with P@20 and Mean Average Precision(MAP) is generated at the same given location with the file name precision_<MM-dd-yy_HH:mm:ss>.out
3. The evaluation results are also avaliable in a .pdf file at this location.
References
-----------
1. JavaSE Documentation: https://docs.oracle.com/javase/8/docs/