Machine translation: from rule-based systems to neural sequence models
Machine translation maps text between languages using computational methods. Early systems used hand-crafted rules or statistical phrase models. Modern neural systems employ encoder-decoder networks with attention and sub-word units, improving accuracy and handling rare words.
Definition and rule-based approaches
Machine translation is a natural language processing task for mapping text across languages, as described in IBM's machine translation overview. The same source distinguishes machine translation from computer-assisted translation, where human translators use digital tools, and outlines rule-based machine translation with approaches like direct, transfer, and interlingua based on the level of linguistic analysis.
The complexity of rule-based machine translation depends on the amount of linguistic analysis applied. Direct translation uses a dictionary for word-for-word output then reorders it, while transfer uses part-of-speech tagging and word sense disambiguation. These methods are often illustrated with Vauquois' triangle.
Statistical machine translation
The idea of using computers to translate languages emerged in the mid-twentieth century, but early machines lacked the power and storage, as AWS's machine translation overview notes. By the early twenty-first century, software, data, and hardware enabled basic machine translation using statistical databases of languages, requiring substantial manual labor.
The statistical and neural MT community provides shared resources such as the Moses toolkit, parallel corpora, and scoring tools. These support training and evaluation of translation systems.
Neural sequence-to-sequence models
A general end-to-end sequence learning approach used a multilayered long short-term memory network to encode an input sequence into a fixed-length vector and another LSTM to decode the target sequence, as described in the sequence-to-sequence learning paper. On the WMT'14 English-to-French task, it achieved a BLEU score of 34.8, compared to 33.3 for a phrase-based statistical system.
A later paper proposed an extension with an attention mechanism that lets the model automatically soft-search for source parts relevant to each target word, as presented in the joint alignment paper. This approach avoids the fixed-length vector bottleneck.
Modern neural MT systems
Google's neural machine translation system uses a deep LSTM with eight encoder and eight decoder layers, attention, and residual connections, as described in the GNMT paper. To handle rare words, it splits them into sub-word units called wordpieces.
The system also employs length normalization and coverage penalty, and in a human evaluation on simple sentences, it reduced translation errors by 60% compared to Google's phrase-based system.
| Approach | How it translates | Strengths | Weaknesses |
|---|---|---|---|
| Rule-based | Applies hand-crafted rules and linguistic information to map source to target | Works for specific domains with limited data | Requires expert rule writing and maintenance |
| Statistical | Trains on parallel corpora to learn word and phrase alignments | Handles large volumes and reduces manual rules | Needs large parallel data and computational effort |
| Neural | Uses deep networks with attention to translate sequences end-to-end | Achieves high accuracy and handles rare words | Computationally expensive and data hungry |
Frequently asked questions
What is the difference between machine translation and computer-assisted translation?
Machine translation is performed by a computer, while computer-assisted translation involves human translators using tools like dictionaries or translation memories.
What is a rule-based machine translation system?
It uses a set of hand-crafted rules and stored linguistic information, such as word pairs and part-of-speech tags, to translate text.
What is the main idea of neural machine translation?
It uses a single neural network, typically an encoder-decoder with attention, to translate sequences directly from source to target.
How do modern neural systems handle rare words?
They break words into sub-word units called wordpieces, as implemented in Google's system.
What resources are available for statistical and neural MT research?
The StatMT site lists resources like the Moses toolkit, parallel corpora, and evaluation tools.