Evaluation info:eu-repo/classification/acm/H.3.4 Systems and Software. Performance evaluation Cross-language information retrieval info:eu-repo/classification/acm/H.3.3 Information Search and Retrieval From CLEF to TrebleCLEF: the evolution of the cross-language evaluation forum 2008-01-01 National Institute of Informatics, Tokyo The Cross-Language Evaluation Forum (CLEF) has been running for nearly ten years now; the aim of this paper is to provide a critical assessment of the results achieved so far. In the first part of the paper, we pro- vide a brief overview of the entire activity and sum- marise the main achievements; in the second part, we focus our attention on the Ad Hoc track with the aim of showing how the results of evaluation can be exploited to increase understanding of the many issues involved in multilingual retrieval system development. In the final part, we outline our main ideas for the future of CLEF. In: NTCIR-7 Meeting (Tokyo, Giappone, 16-19 December 2008). Proceedings, pp. 577 - 593. National Institute of Informatics, Tokyo, 2008. oai:pumaoai.isti.cnr.it:cnr.isti/cnr.isti/2008-A2-123 oai:pumaoai.isti.cnr.it:cnr.isti/cnr.isti/2008-A2-123 false false 0.9 dedup_wf_001::177a370da8dce795e6619dbdcf05e866 2 Peters, Carol dedup_wf_001::eb972c7eda02fc9b4328773c7456903a 1 Ferro, Nicola dedup_wf_001::bc55bac1aef52baa168a1d1422eec109 0.8117433 STANDARD From CLEF to TrebleCLEF: promoting technology transfer for multilingual information retrieval 2007-01-01 DELOS dedup_wf_001::bc55bac1aef52baa168a1d1422eec109 0.8117433 STANDARD From CLEF to TrebleCLEF: promoting technology transfer for multilingual information retrieval 2007-01-01 DELOS corda_______::a4a88c699deafde87220fdb3115e4f4d 215231 TREBLE-CLEF Treble-CLEF: Evaluation, Best Practices and Collaboration for Multilingual Information Access ec__________::EC::FP7 ec__________::EC::FP7::SP1 ec__________::EC::FP7::SP1::ICT http://puma.isti.cnr.it/dfdownloadnew.php?ident=cnr.isti/cnr.isti/2008-A2-123 http://puma.isti.cnr.it/rmydownload.php?filename=cnr.isti/cnr.isti/2008-A2-123/2008-A2-123.pdf Protein Data Bank http://www.rcsb.org/pdb/explore/explore.do?structureId=1the 1the [1] M. Agosti, G. M. Di Nunzio, and N. Ferro. A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns. In T. Sakay, M. Sanderson, and D. K. Evans, editors, Proc. 1st International Work- shop on Evaluating Information Access (EVIA 2007), pages 62–73. National Institute of Informatics, Tokyo, Japan, 2007. [2] M. Braschler. Combination Approaches for Mul- tilingual Text Retrieval. Information Retrieval, 7(1/2):183–204, 2004. [3] M. Braschler and C. Peters. Cross-Language Evalu- ation Forum: Objectives, Results, Achievements. In- formation Retrieval, 7(1–2):7–31, 2004. [4] G. M. Di Nunzio and N. Ferro. DIRECT: a System for Evaluating Information Access Components of Digi- tal Libraries. In A. Rauber, S. Christodoulakis, and A. Min Tjoa, editors, Proc. 9th European Conference on Research and Advanced Technology for Digital Li- braries (ECDL 2005), pages 483–484. Lecture Notes in Computer Science (LNCS) 3652, Springer, Heidel- berg, Germany, 2005. [5] D. Harman and C. Buckley. The NRRC Reliable In- formation Access (RIA) Workshop. In M. Sanderson, K. Ja¨rvelin, J. Allan, and P. Bruza, editors, Proc. 27th Annual International ACM SIGIR Conference on Re- search and Development in Information Retrieval (SI- GIR 2004), pages 528–529. ACM Press, New York, USA, 2004. [6] D. K. Harman, M. Braschler, M. Hess, M. Kluck, C. Peters, P. Schau¨bie, and P. Sheridan. CLIR Eval- uation at TREC. In Peters [8], pages 7–23. [7] H. Mu¨ller, J. Kalpathy-Cramer, . Kahn, C.E., W. Hatt, S. Bedrick, and W. Hersh. Overview of the ImageCLEFmed 2008 Medical image Retrieval Task. In F. Borri, A. Nardi, and C. Pe- ters, editors, Working Notes for the CLEF 2008 Workshop. http://www.clef-campaign. org/2008/working_notes/ GeoCLEF-2008-overview-notebook-paperWNfinal. pdf [last visited 2008, September 10], 2008. [8] C. Peters, editor. Cross-Language Information Re- trieval and Evaluation: Workshop of Cross-Language Evaluation Forum (CLEF 2000). Lecture Notes in Computer Science (LNCS) 2069, Springer, Heidel- berg, Germany, 2001. [9] C. Peters. Introduction. In Cross-Language Infor- mation Retrieval and Evaluation: Workshop of Cross- Language Evaluation Forum (CLEF 2000) [8], pages 1–6. [10] S. E. Robertson. The methodology of information re- trieval experiment. In K. Spa¨rck Jones, editor, In- formation Retrieval Experiment, pages 9–31. Butter- worths, London, United Kingdom, 1981. [11] J. Savoy. Report on CLEF-2001 Experiments: Ef- fective Combined Query-Translation Approach. In C. Peters, M. Braschler, J. Gonzalo, and M. Kluck, editors, Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross– Language Evaluation Forum (CLEF 2001) Revised Papers, pages 27–43. Lecture Notes in Computer Sci- ence (LNCS) 2406, Springer, Heidelberg, Germany, 2002.