Setting the Scene:

NCR Corporation, headquartered in Dayton, Ohio, is a global technology solutions and services company with 32,000 associates in 80 countries. We have five major business units and hundreds of job roles. Learning requirements and priorities are very different from region to region, unit to unit, and job to job. In this context it is a major challenge to communicate and promote new learning opportunities that are relevant to each audience and, more importantly, to each individual.

As a company, NCR’s motto is “turning transactions into relationships.” By this we mean that we assist our customers in converting terabytes of raw data about transactions with their customers (point-of-sale data in retail stores, financial transactions in banks, etc.) into valuable business intelligence that enables our customers to serve their millions of customers more personally and effectively.

Applying this same philosophy internally, NCR’s Global Learning division (a practice area in Human Resources) has introduced two important personalization services for our NCR University Online Campus - “MyNCRU” web pages and the “MyNCRU Personal Learning News” – a monthly email publication. This application describes the latter as a best practice in learning marketing and communications.

Global Learning introduced the NCR University Online Campus in September of 1997. Like many corporate university web sites of its day, NCRU provided our internal users with a convenient, single-point-of-access to an extensive online course catalog, a registration system (LMS), and both NCR and third-party courses – but the interface and content were the same for all users.

In the summer of 1999 we enhanced the NCRU web interface by offering a simple system for capturing users’ job responsibilities and learning interests and using this data to provide a personalized “MyNCRU” access page to the web site. This service was well received and in the ensuing 2-1/2 years nearly 19,000 of NCR’s 32,000 associates signed up.

After becoming familiar with the type of focused communications that they were receiving via their MyNCRU web pages, many users requested that we augment the “pull” type communications approach used on our web site with some “push” type announcements of new course offerings and other professional development activities. Such requests could be summarized like this: “Can’t you send me an email when courses that I would be interested in are released?”

In November of 2000 we added a field on our user registration page that allowed new users to tell us if they would like to receive a monthly, personalized email newsletter. By mid-2001 over 2,000 users (about 40% of all new users) had said, “Yes!” In July of 2001 we introduced the “MyNCRU Personal Learning News” or PLN.

As the subsequent sections will show, this service is user-driven, responsive to change, continuously improved, and global in both scope and content. Most important, it has a clear and positive impact on the effectiveness of learning at NCR.

Best Practice Description:

The content of the Personal Learning News is drawn from an online news and calendar database. The contributors to this database are HR and learning staff members from all over the world. At this time, approximately three years after its introduction, 200 authors from all business units and infrastructure groups have contributed over 3,000 news and calendar entries. Each entry is keyed to one or more business units, job categories, and areas of learning interest (e.g., industry certifications). In addition, classroom courses are also keyed to the geographic regions in which they will be held.

To build a copy of the PLN, our server compares the keys associated with the most recent news and calendar entries with the set of keys stored in the user’s personal profile. It then generates the email using the items that match, sends the message, and moves on to the next subscriber. Our current system builds and sends two such personalized messages per second.

Subscribers have a great deal of control over the content they receive. If they feel their newsletter is too long, they can reduce their interest list; if it’s too short, they can expand their interest list. If the content is a little off-target, they can review their interest profile and revise their selections. Instructions to this effect, together with the current contents of the user’s profile, are contained at the bottom of each newsletter and each user’s profile update page is always one click away.

In parallel with this effort, the NCRU Systems Team had also been working with a couple of suppliers to test the practicality of applying state-of-the-art machine language translation (MT) to NCRU. This, too, was a user-driven development since roughly 10% of our 19,000 registered users have indicated that they would prefer to receive learning information in a language other than English.

In October 2000 Global Learning purchased the Enterprise Translation System with four “language pairs” from SYSTRAN Software, one of the pioneers in the MT field. Using this software the NCRU server can translate the PLN newsletters from English into French, German, Italian, and Spanish for subscribers who have requested those languages. PLN newsletters are typically 3-4 pages long. These are translated by our current system at the rate of two newsletters per minute. The diagram below illustrates how translated copies are created.

Since pure machine translation is still far from perfect, we store the original English versions of all translated newsletters on our web site. The preamble to the translated newsletters explains that they are translated by machine and provides a link back to the English originals in the event that subscribers have any questions about or problems with the translation. (You can see the translations of a sample newsletter here.)

Presenting the Evidence:

Following the first issue of the PLN, 653 (31%) of the 2,133 charter recipients responded to an 8-question online survey. The overall reaction was very favorable. Eighty-nine percent rated the Personal Learning News as "good," "very good," or "excellent" and 94% reported that they found it "somewhat useful" or "very useful." Qualitatively associates said they liked the newsletter's "organization," "focused approach," and "functionality" and the fact that it is "personalized," "concise," "customized," "convenient," and that it "comes automatically to me."

The most frequently reported criticism of the newsletter was that it included news and calendar information that was not relevant to the recipient's geographic region. This was corrected in the second edition.

In the nine months since the first issue the PLN subscriber base has grown 171% to 5,787. (This is a CMGR – compound monthly growth rate – of 13% per month.) Of these 481 (9%) have requested to receive their copies in French, German, Italian, or Spanish. Since the first issue the requests for translations have grown 152% from 191 to 481. (This is an 12% CMGR.) Since the newsletter’s inception we have published 31,066 individual copies of which 2,383 (8%) were machine translated. Only 52 subscribers (less than 1%) have cancelled their subscriptions.

The PLN is published in two formats – web page format (HTML) and plain text. The web page format is colorful and attractive with images and advanced formatting to facilitate legibility and functionality. All of the news and calendar items are hyperlinked to pages on NCRU or to course descriptions in our catalog so that subscribers can get to a course description, a registration page, or even start an online course with a single click. (Here is a sample of the full newsletter in web format.)

Since some NCR associates do not yet have a mail reader that supports the web page formatting, we also offer a simple, clean, plain text edition of the PLN. At this time 5,081 subscribers (88%) receive the web page format and 706 (12%) receive plain text.

We have seen that this new form of "push" communication is stimulating participation in our courses and other learning services. For example, in December 2001 we introduced a new, online, development planning feature of our LMS. Immediately following the publication of the December and January PLN messages that featured the new offering, user activity on the gateway page for this service spiked up by hundreds of visits.

We have not yet quantified the increase that the PLN has brought to course registrations and online completions, but we have indirect evidence that it is having a positive effect. We have experienced increases in user feedback questions following PLN publication cycles and we have been copied on internal correspondence in which the PLN was used as the basis for obtaining management approval for course registration requests.

Regarding the machine translation of the PLN, user response is mixed. In our pilot survey when asked to list the three things they LIKED MOST about the newsletter, several respondents said: "the translation," "was translated in French," and "There is a German version of the Newsletter." One respondent apparently felt it was now OK to express himself in the language he prefers and answered the entire survey in Spanish.

On the other side, when asked to list the three things they LIKED LEAST about the newsletter, other respondents said: "automatic translation," "translation is sometimes funny," "Translation almost incomprehensible," and "the translation is very poor."

Based on these divergent views, our current hypothesis on the usefulness of pure machine translation is the following. Those who speak English well will prefer to read English rather than a clumsy and occasionally inaccurate version of their native language. Those who do not speak English well will prefer to read the machine translation and refer back to the original only when they encounter something that appears to be mistranslated. In the second case, the availability of the "gist" translation can significantly improve reading speed and comprehension, thereby increasing associate productivity.

The bottom line on MT is that, as noted above, since the first PLN issue the requests for translations have grown 12% per month from 191 to 481. This would certainly appear to be a clearly and quantitatively positive "voice of the customer."