As the technology underlying machine translation (MT) continues to evolve, so does its use by organizations interested in leveraging its benefits. But the growth of MT and its many applications gives rise to a host of questions that can stump both first-time users and experienced localization professionals: is MT right for my content? How can I maximize savings? How do I know if MT quality is good, or even sufficient?
To help answer these and other questions, we’ve simplified the process by identifying the top three questions to ask when evaluating MT for your organization:
1. What is the business problem that needs to be solved?
Although seemingly straightforward, it is essential to start by identifying the business problem that needs to be solved before viewing MT as the solution. This will enable a more informed exploration of its potential benefits and avoid inadequate or ineffective applications of MT.
2. How well can MT solve the business problem?
Once the business problem is identified, it is then necessary to select the criteria that will demonstrate how well MT can actually address it. Instead of measuring how good MT is “in general,” establish specific goals or KPIs that directly represent the main aspects of the business problem that MT proposes to solve.
3. How can we measure the success of MT in this application?
Finally, a standard of measurement is needed to determine the degree to which MT is successful in solving the business problem and achieving the goal or KPI identified previously. For this, metrics such as post-editing distance (PED), BLEU, and COMET can be used alone or alongside other methods such as direct assessment and qualitative linguistic feedback to measure short-term success and give insights into MT’s long-term benefits.
With the excitement around MT’s breakthroughs on the rise, it is more important than ever to use these three considerations as a guide before implementation. It is not uncommon for organizations to deploy MT first and discover only later that its potential isn’t fully realized, or that their expectations aren’t met by the solution that was chosen.
Addressing these core questions will lay the foundation for an evaluation centered on maximizing the ways that MT can add value to your organization, whether you simply want to assess if MT is right for your content or formally compare MT output quality from several providers.
But what about MT for “gisting” or understandability purposes? And how about a business problem that is complex and requires several success criteria? The answers to these and other questions can be found in the latest edition of our MT Spotlight Report, which provides an overview of best practices for MT evaluation and includes a detailed case study.
By: Ty Trainer