A GoP is a systematic series of experiments where (ideally) all the combinations of retrieval methods and components are represented.
GoP produces two positive effects:
We face the problem of studying system variance in order to better understand how much system components contribute to overall performances. We propose a methodology based on General Linear Mixed Model (GLMM) to develop statistical models able to isolate system variance, component effects as well as their interaction.
We apply the proposed methodology to the analysis of TREC Ad-hoc data in order to show how it works and discuss some interesting outcomes of this new kind of analysis. Finally, we extend the analysis to different evaluation measures, showing how they impact on the sources of variance.
We selected a set of alternative implementations of each component and by using the Terrier open source system we created a run for each system defined by combining the available components in all possible ways.
We considered three main components of an IR system: stop list, Lexical Unit Generator (LUG) and IR model. We selected a set of alternative implementations of each component and by using the Terrier open source system we created a run for each system defined by combining the available components in all possible ways. The components we selected are:
stop list: nostop, indri, lucene, smart, terrier
LUG: nolug, weak Porter, Porter, Krovetz, Lovins, 4grams, 5grams;
model: BB2, BM25, DFRBM25, DFRee, DLH, DLH13, DPH, HiemstraLM, IFB2, InL2, InexpB2, InexpC2, LGD, LemurTFIDF, PL2, TFIDF.
Note that some stemmers and the n-grams are not natively implemented by Terrier 4.1. It is possible to download the extensions to Terrier 4.1 here: Get Terrier extensions.
In this paper we run a systematic series of experiments for creating a grid of points where many combinations of retrieval methods and components adopted by MultiLingual Information Access (MLIA) systems are represented. This grid of points has the goal to provide insights about the effectiveness of the different components and their interaction and to identify suitable baselines with respect to which all the comparisons can be made.
Full reference: Nicola Ferro and Gianmaria Silvello (2016). The CLEF Monolingual Grid of Points. In Fuhr, N., Quaresma, P., Gonçalves, T., Larsen, B., Balog, K., Macdonald, C., Cappellato, L., and Ferro, N., editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventh International Conference of the CLEF Association (CLEF 2016), pages 13-24. Lecture Notes in Computer Science (LNCS) 9822, Springer, Heidelberg, Germany