User Requirements and Evaluation

Use of resources

Node	Budgeted	Period 1	Period 2	Period 3 (est)
UGOT	3	0	OC:0.4	X
UHEL	11	0	X	X
UPC	8	0	2,66	X
Ontotext	6	0	X	X
BI	1		X	X
UZH	2		X	X

Requirements for work

The work will start with collecting user requirements for the grammar development IDE (WP2), translation tools (WP3), and the use cases (WP6-8).

We will define the evaluation criteria and schedule in synchrony with the WP plans (D9.1). We will define and collect corpora including diagnostic and evaluation sets, the former, to improve translation quality on the way, and the latter to evaluate final results.

Corpus definitions

• Each corpus available for MOLTO will be described by the providing project members.
• The corpora will be split for development, diagnostic and evaluation use.
• Contact persons will be named for questions and requests for each corpus.
• Storage places and access protocols to gain this specified corpus data will be defined.

Description of end-user workflow

Translator's new role (parallel to WP3: Translator's tools) will be designed and described in the D9.1 deliverable. Most current translator's workbench software treat the original text as read-only source. The tools to be developed within WP3 (+ 2) will lead towards more mutable role of source text. The translation process will resemble more like structured document editing or multilingual authoring than transformation from a fixed source to a number of target languages.

We will only provide a basic infrastructure API for external translation workbenches and keep an eye on the "new multilingual translator's workflow".

Introduction of WP liaison persons and other contacts

For each work package, the liaison contact information and work progress will be kept up-to-date on the MOLTO web site. Our liaison person Mirka Hyvärinen will be in contact with other project members.

Also possibility to access UHEL's internal working wiki "MOLTO kitwiki" will be granted upon request to other project members.

Evaluation of results

Evaluation aims at both quality and usability aspects. UHEL will develop usability tests for the end-user human translator. The MOLTO-based translation workflow may differ from the traditional translator's workflow. This will be discussed in the D9.1 evaluation plan.

To measure the quality of MOLTO translations, we compare them to (i) statistical and symbolic machine translation (Google, SYSTRAN); and (ii) human professional translation. We will use both automatic metrics (IQmt and BLEU; see section 1.2.8 for details) and TAUS quality criteria (Translation Automation Users Society). As MOLTO is focused on information-faithful grammatically correct translation in special domains, TAUS results will probably be more important.

Given MOLTO's symbolic, grammar-based interlingual approach, scalability, portability and usability are important quality criteria for the translation results. For the translator's tools, user-friendliness will be a major aspect of the evaluation. These criteria are quantified in (D9.1) and reported in the final evaluation (D9.2).

In addition to the WP deliverables, there will be continuous evaluation and monitoring with internal status reports according to the schedule defined in D9.1.