Translation Quality Evaluation in the MOLTO Project

The goal of the MOLTO project is to produce automatic tools that will offer high quality, real-time translation in multiple languages with the intended target user of these tools being a producer of information wishing to offer online information in multiple languages. This scenario calls for a high standard of quality: a translation with both high accuracy of content and fluency of form, ready for publication. Furthermore, the MOLTO project aims to provide tools that are extensible to new domains within months or even days rather than the years normally required for building such a system, and without deep technological knowledge. These goals point to the specific aspects of quality that are central to the evaluation of the MOLTO tools: text quality in terms of fidelity and fluency of the translation, coverage of each domain in question and usability in terms of work needed to build and extend the system.

This presentation will discuss both manual and automated evaluation approaches and technology. The first part of the presentation illustrates manual approaches with a specific focus on a machine translation post-editing task carried out by the University of Helsinki and University of Turku. The second part of the presentation discusses automated evaluation technology and its implementation within The Asiya Open Toolkit for Automatic MT (Meta-)Evaluation developed at the Universitat Politècnica de Catalunya. Some preliminary evaluation results using this software on patent translations are given as well.