Information Retrieval Evaluation In A Changing World: Lessons Learned From 20 Years Of Clef (the Information Retrieval Series)

"This volume celebrates the twentieth anniversary of CLEF - the Cross-Language Evaluation Forum for the first ten years, and the Conference and Labs of the Evaluation Forum since - and traces its evolution over these first two decades. CLEF's main mission is to promote research, innovation and development of information retrieval (IR) systems by anticipating trends in information management in order to stimulate advances in the field of IR system experimentation and evaluation. The book is divided into six parts. Parts I and II provide background and context, with the first part explaining what is meant by experimental evaluation and the underlying theory, and describing how this has been interpreted in CLEF and in other internationally recognized evaluation initiatives. Part II presents research architectures and infrastructures that have been developed to manage experimental data and to provide evaluation services in CLEF and elsewhere. Parts III, IV and V represent the core of the book, presenting some of the most significant evaluation activities in CLEF, ranging from the early multilingual text processing exercises to the later, more sophisticated experiments on multimodal collections in diverse genres and media. In all cases, the focus is not only on describing "what has been achieved", but above all on "what has been learnt". The final part examines the impact CLEF has had on the research world and discusses current and future challenges, both academic and industrial, including the relevance of IR benchmarking in industrial settings. Mainly intended for researchers in academia and industry, it also offers useful insights and tips for practitioners in industry working on the evaluation and performance issues of IR tools, and graduate students specializing in information retrieval"--Back cover Front Matter ....Pages i-xxii Front Matter ....Pages 1-1 From Multilingual to Multimodal: The Evolution of CLEF over Two Decades (Nicola Ferro, Carol Peters)....Pages 3-44 The Evolution of Cranfield (Ellen M. Voorhees)....Pages 45-69 How to Run an Evaluation Task (Tetsuya Sakai)....Pages 71-102 Front Matter ....Pages 103-103 An Innovative Approach to Data Management and Curation of Experimental Data Generated Through IR Test Collections (Maristella Agosti, Giorgio Maria Di Nunzio, Nicola Ferro, Gianmaria Silvello)....Pages 105-122 TIRA Integrated Research Architecture (Martin Potthast, Tim Gollub, Matti Wiegmann, Benno Stein)....Pages 123-160 EaaS: Evaluation-as-a-Service and Experiences from the VISCERAL Project (Henning Müller, Allan Hanbury)....Pages 161-173 Front Matter ....Pages 175-175 Lessons Learnt from Experiments on the Ad Hoc Multilingual Test Collections at CLEF (Jacques Savoy, Martin Braschler)....Pages 177-200 The Challenges of Language Variation in Information Access (Jussi Karlgren, Turid Hedlund, Kalervo Järvelin, Heikki Keskustalo, Kimmo Kettunen)....Pages 201-216 Multi-Lingual Retrieval of Pictures in ImageCLEF (Paul Clough, Theodora Tsikrika)....Pages 217-230 Experiences from the ImageCLEF Medical Retrieval and Annotation Tasks (Henning Müller, Jayashree Kalpathy-Cramer, Alba García Seco de Herrera)....Pages 231-250 Automatic Image Annotation at ImageCLEF (Josiah Wang, Andrew Gilbert, Bart Thomee, Mauricio Villegas)....Pages 251-273 Image Retrieval Evaluation in Specific Domains (Luca Piras, Barbara Caputo, Duc-Tien Dang-Nguyen, Michael Riegler, Pål Halvorsen)....Pages 275-305 About Sound and Vision: CLEF Beyond Text Retrieval Tasks (Gareth J. F. Jones)....Pages 307-329 Front Matter ....Pages 331-331 The Scholarly Impact and Strategic Intent of CLEF eHealth Labs from 2012 to 2017 (Hanna Suominen, Liadh Kelly, Lorraine Goeuriot)....Pages 333-363 Multilingual Patent Text Retrieval Evaluation: CLEF–IP (Florina Piroi, Allan Hanbury)....Pages 365-387 Biodiversity Information Retrieval Through Large Scale Content-Based Identification: A Long-Term Evaluation (Alexis Joly, Hervé Goëau, Hervé Glotin, Concetto Spampinato, Pierre Bonnet, Willem-Pier Vellinga et al.)....Pages 389-413 From XML Retrieval to Semantic Search and Beyond (Jaap Kamps, Marijn Koolen, Shlomo Geva, Ralf Schenkel, Eric SanJuan, Toine Bogers)....Pages 415-437 Front Matter ....Pages 439-439 Results and Lessons of the Question Answering Track at CLEF (Anselmo Peñas, Álvaro Rodrigo, Bernardo Magnini, Pamela Forner, Eduard Hovy, Richard Sutcliffe et al.)....Pages 441-460 Evolution of the PAN Lab on Digital Text Forensics (Paolo Rosso, Martin Potthast, Benno Stein, Efstathios Stamatatos, Francisco Rangel, Walter Daelemans)....Pages 461-485 RepLab: An Evaluation Campaign for Online Monitoring Systems (Jorge Carrillo-de-Albornoz, Julio Gonzalo, Enrique Amigó)....Pages 487-510 Continuous Evaluation of Large-Scale Information Access Systems: A Case for Living Labs (Frank Hopfgartner, Krisztian Balog, Andreas Lommatzsch, Liadh Kelly, Benjamin Kille, Anne Schuth et al.)....Pages 511-543 Front Matter ....Pages 545-545 The Scholarly Impact of CLEF 2010–2017 (Birger Larsen)....Pages 547-554 Reproducibility and Validity in CLEF (Norbert Fuhr)....Pages 555-564 Visual Analytics and IR Experimental Evaluation (Nicola Ferro, Giuseppe Santucci)....Pages 565-582 Adopting Systematic Evaluation Benchmarks in Operational Settings (Jussi Karlgren)....Pages 583-590 Back Matter ....Pages 591-595