Empirical Comparison of Evaluation Methods for Unsupervised Learning of Morphology

Sami Virpioja*, Ville T. Turunen*, Sebastian Spiegler**, Oskar Kohonen* and Mikko Kurimo*
*Department of Information and Computer Science; Aalto University; P.O. Box 15400; FI-00076 Aalto; Finland; sami.virpioja@aalto.fi
**Department of Computer Science; University of Bristol; Woodland Road; Bristol BS8 1UB; UK; spiegler@cs.bris.ac.uk
Résumé (en anglais)
Unsupervised and semi-supervised learning of morphology provide practical solutions for processing morphologically rich languages with less human labor than the traditional rule-based analyzers. Direct evaluation of the learning methods using linguistic reference analyses is important for their development, as evaluation through the final applications is often time consuming. However, even linguistic evaluation is not straightforward for full morphological analysis, because the morpheme labels generated by the learning method can be arbitrary. We review the previous evaluation methods for the learning tasks and propose new variations. In order to compare the methods, we perform an extensive meta-evaluation using the large collection of results from the Morpho Challenge competitions.