Adaptive Heads-up Displays for Simultaneous Interpretation
Interpretation, the task of translating speech from one language to another, is an important tool in facilitating communication in multi-lingual settings such as international meetings, travel, or diplomacy. However, simultaneous interpretation, during which the results must be produced as the speaker is speaking, is an extremely difficult task requiring a high level of experience and training. In particular, simultaneous interpreters often find certain content such as technical terms, names of people and organizations, and numbers particularly hard to translate correctly. This Early Grant for Exploratory Research project aims to create automatic interpretation assistants that will help interpreters with this difficult-to-translate content by recognizing this content in the original language, and displaying translations on a heads-up display (similar to teleprompter) for interpreters to use if they wish. This will make simultaneous interpretation more effective and accessible, making conversations across languages and cultures more natural, more common, and more effective and joining communities and cultures across the world in trade, cooperation, and friendship. Creating these systems is a technically challenging problem and has not previously been attempted. One challenge is that simultaneous interpretation is already a cognitively taxing task, and any interface must not unduly increase the cognitive load on the interpreter by being too intrusive. Reducing this cognitive load requires an interface that can decide when to provide translation suggestions and when to refrain from doing so. To achieve this goal, this project will develop methods that are robust to speech recognition errors, and learn what to display by observing the interpreters’ interpretation results. The utility of the proposed framework will be evaluated with respect to how much it improves the ability of interpreters to produce fluent, accurate interpretation results, as well as the cognitive load the additional interface imposes on them.
Duration:
September 2017 - February 2019
Project Website:
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1748663
Total Award Amount:
$150,000