Getting the Message In:A Global Company’s Experience with the New Generation
|
|
Abstract. Most large companies are very good at „getting the message out" – publishing reams of announcements and documentation to their employees and customers. More challenging by far is „getting the message in" – ensuring that these messages are read, understood, and acted upon by the recipients. This paper describes NCR Corporation’s experience with the selection and implementation of a machine translation (MT) system in the Global Learning division of Human Resources. The author summarizes NCR‘s vision for the use of MT, the competitive „fly-off" evaluation process he conducted in the spring of 2000, the current MT production environment, and the reactions of the MT users. Although the vision is not yet fulfilled, progress is being made. The author describes NCR’s plans to extend its current MT architecture to provide real-time translation of web pages and other intranet resources. |
1 Introduction
NCR Corporation, headquartered in Dayton, Ohio, is a global technology solutions and services company with 32,000 associates in 80 countries. NCR has five major business units and hundreds of professional and technical job roles. Over 50% of NCR’s workforce reside outside the United States.
Like most modern companies, NCR is very good at „getting the message out" – publishing reams of announcements, brochures, instructions, and other documents to their employees and customers. More challenging by far is „getting the message in" – ensuring that these messages are read, understood, and acted upon by the recipients.
Although English is the official language of the company, many associates are not fluent in English. This impairs not only their ability to read company documents and converse with their English-speaking colleagues; it also makes it more difficult for them to stay abreast of global company developments and even to take advantage of specific opportunities, for example, training programs, that would help them improve their performance. In short, it reduces their productivity.
NCR also generates more than half of its revenue outside the United States. When country-level revenue statistics are combined on the basis of local language, it is clear that effective communication in Japanese, German, French, Spanish, and Italian is very important to the company.
In recent surveys of NCR associates around the world, between 10 and 15 percent of respondents say they would prefer to receive company communications in a language other than English. The actual numbers are probably much higher due to several factors. The most important of these are: 1) since the surveys were in English respondents tend to be those who are comfortable with English, 2) many respondents may not have taken the question seriously since they may have assumed that a vote for anything other than English would have no practical impact, and 3) respondents were disproportionately drawn from the upper echelons of the company who have better training in and more exposure to English.
One verbatim response to the question, "In which language would you prefer to receive your company communications?" was telling.
"Although English is an international language, it should be that the information is agreement to the country. So that it is but understandable the information and we all are speaking of the same communication. It is ridiculous that they worry about the communication without everybody speaks English."
In both style and content this is a great argument for the use of good translation services...
1.1 NCR’s MT Vision
In January 1998, an advanced technology team in NCR’s Organization Development group became aware of the newly released web page translation service offered on the Altavista internet search site using machine translation (MT) technology from SYSTRAN Software, Inc. This team monitored the new developments in MT technology as it migrated from mainframes to minicomputers to PCs and as the software moved from highly customized, bespoke systems to shrink-wrapped, off-the-shelf packages.
Excited by these advances, the author formulated this simple vision.
NCR should have a global company intranet on which associates can navigate in the language of their choice and have the entire contents of that intranet appear to be in that language.
1.2 A Scenario Using Transparent MT on a Global Intranet
To understand the impact of this vision on a practical level, consider this scenario.
- An NCR associate in France launches her web browser and enters the URL of the page on the company intranet (or public internet) she wants to see.
- The first time she does this she is connected automatically to a central NCR translating web server that presents a screen that says (in French), "What language would you like to use?" She chooses French. [This preference would be stored in a web "cookie" on her PC so that it would not be necessary for her to make this selection again.]
- The translating server (acting as a proxy server) then retrieves the requested page, translates it, and sends it to the French user's browser which displays it.
- The French user views the page she requested in French. The page contains a number of links to other information. She clicks on one of these links.
- The user's request is sent back through the translating server; it does not go directly to the server originally specified on the original source (untranslated) web page.
- When the translating server receives the request for a new page, it knows that the request came from a user who wants to see French. So without stopping to ask the user again what language she wants, the translating server retrieves the requested page, translates it, and sends it to the user.
Up to this point the user-system interaction and result are similar to the way some free internet translations services now operate, but the scenario continues.
Now this is different from the services commonly available on the internet. In this scenario, the user has the feeling that she is simply surfing the company intranet and every page is coming up in French. She is using no special software on her PC, just a standard internet browser.
[Note: The scenario described above was created four years ago and is a simplification of the actual implementation that would be used today. It suffers several technical deficiencies, but these do not detract from the purpose of this illustration.]
This technique of transparent translation will ultimately improve productivity by eliminating the mental clutter that would otherwise be introduced by constant translation requests. The key to its success is not only the technical challenge of embedding the translation server invisibly between the user and the source information. Equally important is the quality of the translation.
In the example above, the illusion of a French intranet can only be preserved if the translations are good enough to permit the rapid "scan-click, scan-click" behavior that has become widely known as "web surfing." If the user stumbles over every other sentence or, worse, cannot fathom the meaning of a particularly inept turn of phrase, the spell is broken and the productivity gains are lost. This, then, is the challenge today: to implement the right level of MT technology in the right context, in other words, for delivering the right source texts to the right audiences.
2 Selecting an MT System
In January 2000, NCR invited two MT suppliers – Lernout & Hauspie of Brussels, Belgium, and SYSTRAN Software of Paris and San Diego, California - to participate in an MT "Fly-Off" competition. Modeled loosely on the type of competition sponsored by national airforces to evaluate new military aircraft, NCR’s program called for both suppliers to install prototype systems on NCR servers. These systems were similar in cost and capability.
2.1 Selection Criteria
Once installed on NCR servers, these systems were tested by a team of US and international associates and judged on the basis of: 1) quality of translation, 2) simplicity of user interface and system integration, and 3) total cost of ownership.
In judging the quality of translations provided by the MT systems, NCR's evaluators focused on the clarity and accuracy of meaning and the avoidance of outright errors. Recognizing that MT systems do not yet deliver perfect translations, NCR’s evaluators looked for the system that produced the clearest, easiest to read language that correctly conveyed the gist of the original text. When mistakes were made they assessed whether the mistakes were merely clumsy constructions or outright errors. Instances of the latter detracted more significantly from the scores awarded in this category.
The second criterion - simplicity of the user interface and system integration – reflected the fact that for this technology to be usefully applied, the MT system must be well integrated with the target applications. In the case of the pilot, this was the web server that hosted the NCR University Online Campus (NCRU). NCR wanted the translations to take place behind the scenes and to introduce no significant delay in web information delivery. From the system support standpoint, NCR looked for the MT system to be robust (high availability/high reliability) and to require no extraordinary operator effort or regular intervention.
When the first two criteria were met, the author made a final judgement factoring in the Total Cost of Ownership over three years. Costs included initial license fees, annual maintenance and update charges, and special one-time or on-going support charges, if any.
NCR advised the suppliers that all three criteria had to be met within practical limits in order for NCR to proceed to the production phase of the project, i.e. putting an MT system into continuous use on the NCRU web site.
2.2 The Test Environment
The evaluation took place in May 2000. The test environment consisted of 50 web pages representing typical content from all areas of NCR University. Thirteen NCR associates in eight countries on four continents participated in the evaluation.
The systems were assessed on two levels:
- Page Level: Each of 35 pages from which the suppliers built special NCR dictionaries were evaluated separately. In addition, evaluators were asked to review 15 new pages that contained similar content, but which were not used for dictionary input.
- System Level: When they completed the page level evaluations for both systems, evaluators were asked to complete a short web survey summarizing their impressions and providing a "Yes" or "No" vote on whether either one of the test systems would be practical to implement.
Overall quality on each page was rated on a 1-5 scale (with 5 being best) and serious errors per page were simply counted. The system level evaluation consisted of the survey reproduced in Appendix A. The fourth question (below) addressed the central issue.
Based on this test, what position would you recommend that NCR take on machine translation of web sites for your language?
- Use it now throughout the NCR intranet - it works.
- Test it with a larger audience in selected areas of the NCRU web site only.
- Do not apply it now, but continue to monitor progress in this technology closely.
- Don't waste more time - this is still years away from a practical use.
2.3 Fly-Off Results
In the detailed, page-by-page analyses, the evaluators gave a slight edge, quantitatively and qualitatively, to SYSTRAN. On the overall "Go/No Go" question (above), the results were generally unfavorable. None of the evaluators felt that either system was ready for a large-scale deployment and only a couple thought it would be worthwhile to test them with a larger audience on one web site. Most suggested monitoring the technology further and a few believed that no use would be practical for several years.
In a discussion of this negative result one of the Spanish evaluators, an associate in Argentina, made an insightful observation.
"...I am fluent in English, and can read it effortlesly. (probably this is true with most of the evaluators). So, I surely prefer to read English than bad Spanish. But maybe it is not true for all the people that only reads English with great effort.
Maybe you could find a group of evaluators that need the translations and ask them not is the translation is good (it is not), but wether (sic) they would prefer to read the translated version, however bad, rather than the original."
The project team decided to press on and the results achieved with larger, more randomized audiences support this observation (more on this in Section 5.)
In October 2000, NCR purchased the Enterprise Translation System from SYSTRAN. The order included the "engines" for translations between English and French, German, Italian, and Spanish. As a result of the collaboration between NCR and SYSTRAN during the fly-off project, SYSTRAN’s server software included a newly developed plug-in for Microsoft’s IIS™ web server. This plug-in was designed to monitor web page requests and route selected pages through the translation engines. The basic elements of the configuration are illustrated in Figure 3 in the next section.
3 NCR’s Initial Plan for MT on the Web (Real-Time Translation)
NCR’s original plan called for pages to be automatically translated for all visitors who expressed their preference for a language other than English. If they got something they couldn't understand or that looked odd, they could request the original English page.
Since current MT technology produces only rough or "gist" translations, NCR decided that it would be very important to advise users that the translations they were reading came from a machine. This notice, embedded in a colorful banner at the top of every page (see Figures 1 and 2), also provided a link to the source text to assist users in the event of poor or mis-translations.
Fig. 1. English Text of the MT Advisory Banner.
For pages translated from U.S. English to French, this banner would appear as illustrated in Figure 2 below.
Fig. 2. French Text of the MT Advisory Banner.
The architecture to provide this real-time web page translation service is illustrated in Figure 3 below.
Fig. 3. Architecture for Real-Time Web Page Translation.
Based on the fly-off evaluation, however, none of the NCR evaluators recommended proceeding with a real-time translation system configured in this way. The consensus was that the translations from both suppliers were at best clumsy and at worst wrong and the evaluators felt it would not benefit their local colleagues to have these translations as their default view of the pages on the NCR University web site. Most extreme in this view were the Japanese evaluators who felt that the translations into their language were a long way from being ready for productive use.
One evaluator suggested that NCR take a less ambitious approach. In this scenario users would actively request translations in a method similar to, but more streamlined than, the methods that are offered by many suppliers on the public internet today. All NCR University web pages would be presented in their original English form. If the visitor had trouble understanding the text, he or she could click a button in the header or footer of the page that reads "Translate This Page." Since nearly 2,000 of our 19,000 registered users have indicated which non-English language they prefer, NCR could translate the pages into their languages without having to ask them to supply the language pair each time.
In this second approach the translation would be on demand, rather than automatic. If users felt the translations were not helpful, they would not have to use them. If, on the other hand, NCR discovered through a log analysis that many users were regularly requesting translations, automatic translation could be offered. This would give NCR the quantitative business case needed for a more comprehensive implementation.
Due to technical difficulties connecting the translation engines to the web server, NCR has not put either real-time web page approach into production at this time. The next section describes a batch translation process for pure MT translation that is now in production for HTML formatted newsletters.
4 Publishing a Global Newsletter (Batch Translation)
As a company, NCR’s motto is "turning transactions into relationships." This means that NCR assists its customers in converting terabytes of raw data about transactions with their customers (point-of-sale data in retail stores, financial transactions in banks, etc.) into valuable business intelligence that enables NCR’s customers to serve their millions of customers more personally and effectively.
Applying this same philosophy internally, NCR’s Global Learning division introduced two important personalization services for NCR University – "MyNCRU" web pages and the "MyNCRU Personal Learning News" – a monthly email publication.
 |
|  |
Fig. 4. Layout of the MyNCRU Personal Learning News
The content of the Personal Learning News (PLN) is drawn from an online news and calendar database. The contributors to this database are HR and learning staff members from all over the world. To build a copy of the PLN, the NCRU server compares the keys associated with the most recent news and calendar entries with the set of keys stored in the user’s personal profile. It then generates the email using the items that match, sends the message, and moves on to the next subscriber. The current system builds and sends two such personalized messages per second. Figure 4 illustrates the layout of the newsletter.
Since each of nearly 6,000 copies of the PLN newsletter is individually constructed for its recipient it is not feasible to consider a translation process that would involve human intervention. Using MT software the NCRU server can translate the PLN newsletters from English into French, German, Italian, and Spanish for subscribers who have requested those languages. PLN newsletters are typically 3-4 pages long. These are translated by NCR’s current system at the rate of two newsletters per minute. Figure 5 illustrates how translated copies are created, translated, and sent.
Fig. 5. Process by which the MyNCRU Personal Learning News is created, translated, and sent to subscribers.
The yellow line at the bottom of the diagram pointing toward the subscriber’s PC illustrates how the
subscriber can request the original English version via a hyperlink in the translated message.
The system stores the original English versions of all translated newsletters on the NCRU web site. The preamble to the translated newsletters explains that they are translated entirely by machine and provides a link back to the English originals in the event that subscribers have any questions about or problems with the translation. (The translations of a sample newsletter are available for review at this address: http://www.geocities.com/morlav/pln/translations.htm.)
5 Subscriber Reaction
In the nine months since the first issue the PLN subscriber base has grown 171% to 5,787. (This is a CMGR – compound monthly growth rate – of 13% per month.) Of these 481 (9%) have requested to receive their copies in French, German, Italian, or Spanish. Since the first issue the requests for translations have grown 152% from 191 to 481. (This is an 12% CMGR.) Since the newsletter’s inception NCR has published 31,066 individual copies of which 2,383 (8%) were machine translated. Only 52 subscribers (less than 1%) have cancelled their subscriptions.
Following the first issue of the PLN, 653 (31%) of the 2,133 charter recipients responded to an 8-question online survey. The overall reaction was very favorable, but response to the machine translation of the PLN was been mixed. When asked to list the three things they liked most about the newsletter, several respondents said: "the translation," "was translated in French," and "There is a German version of the Newsletter." One respondent apparently felt it was now OK to express himself in the language he prefers and answered the entire survey in Spanish.
On the other side, when asked to list the three things they liked least about the newsletter, other respondents said: "automatic translation," "translation is sometimes funny," "Translation almost incomprehensible," and "the translation is very poor."
Based on these divergent views, NCR’s current hypothesis on the usefulness of pure machine translation is the following.
Those who speak English well will prefer to read English rather than a clumsy and occasionally inaccurate version of their native language. Those who do not speak English well will prefer to read the machine translation and refer back to the original only when they encounter something that appears to be mistranslated. In the second case, the availability of the "gist" translation can significantly improve reading speed and comprehension, thereby increasing associate productivity.
In quantitative terms, the popularity of the translated editions has kept pace with that of the English newsletter – see Table 1, below.
TRANS-
ISSUE DATE ENGLISH LATED %
============= ======== ======= ====== ======
Vol. 1, No. 1 Jul 2001 2,044 191 8.5%
No. 2 Aug 2001 1,971 195 9.0%
No. 3 Sep 2001 2,172 204 8.6%
No. 4 Oct 2001 2,234 204 8.4%
No. 5 Nov 2001 2,786 277 9.0%
No. 6 Dec 2001 2,948 285 8.8%
Vol. 2, No. 1 Jan 2002 3,075 298 8.8%
No. 2 Feb 2002 3,287 309 8.6%
No. 3 Mar 2002 4,593 420 8.4%
No. 4 Apr 2002 4,727 431 8.4%
====== =====
TOTALS: 29,837 2,814 8.6%
Table 1. Publication Statistics for the English and Translated Copies of the Personal Learning News.
Since the first PLN issue, the requests for translations have grown at a compound rate of 12% per month from 191 to 481. In the coming months NCR plans to survey the recipients of the MT copies to determine more precisely how they are using the newsletter and the extent to which it is enhancing their use of NCR University courses and other services. At this time the growth in overall newsletter requests demonstrates that the message is getting out and the growth in translation requests strongly suggests that the message is also getting in.
Appendix A: Overall MT Evaluation Survey
- How would you rate each system in terms of translation quality for "gist" translations on web pages? (Scale: Excellent, Good, Acceptable, Poor, Unacceptable)
- How would you rate each system in terms of ease of use? (Scale: Very easy, Easy, Acceptable, Difficult, Very difficult)
- How would you describe the time delay that the real-time translation caused in the delivery web pages? (Scale: No noticeable delay, Minor delays, Acceptable delays, Significant delays, Unacceptable)
- Based on this test, what position would you recommend NCR take on machine translation of web sites for your language?
- Use it now throughout the NCR intranet - it works.
- Test it with a larger audience in selected areas of the NCRU web site only.
- Do not apply it now, but continue to monitor progress in this technology closely.
- Don't waste more time - this is still years away from a practical use.
- Please provide any final comments you may have on either or both systems. (You may explain your previous answers if you wish.)