The value of Post Editing - IBM Case Study.ppt

1、The value of Post Editing - IBM Case Study,Frank X. Rojas, Jian Ming Xu, Santi Pont Nesta, lex Martnez Corri, Salim Roukos, Helena Chapman, Saroj K. Vohra June 2011,2,IBM Case Study MT Post Editing,IntroductionMT InnovationProcess Overview Findings Conclusion / Recommendations,3,IBM World Wide Trans

2、lation Operations,24 Centers World Wide 115 Translation Suppliers,Process 2.8 B Words Translate 0.4 B Words,60 language pairs,One Stop Shop for all Translation Services,Marketing Material,Web,Product Integrated,Information,Publications,Legal/Safety/ Contracts,Machine Translation,Multimedia,Francizat

3、ion,Cultural Consultancy,Centralized DTP,Overall,End to End,Process,Management,4,IBM Professional Translation Services,Professional Memory 72% 85% Re-Use,Unit Cost 50% Reduction,Traditional TechnologyProcess Mgmt,Human Skill,Memory Assets,CAT Editor,Consistent Quality Standards Global Brand Identity

4、 Professional Quality Standards,1,2,3,Future: Ability to reduce cost using conventional methods reaching limits Business pressure for additional cost elimination Looking to MT Technology as next wave to reach business goals,5,Historical Perspective,2006,2007,2008,2009,2011,2012,2010,2010 MT piloting

5、 Pilot: SPA, ITA, FRE, GER - New E2E process Partnership: WWTO/n.Fluent 8.6 M words,2011 MT Training Pilot: GER, BPR, JPN, CHS - MT payment profiles ready 16.0 M words target,eSupport (www) “Translate This Page” JPN pilot / rule engine,Initial n.Fluent/WWTO Spanish MT pilot - Improve efficiency of p

6、rofessional translators,6,MT Critical Success Metrics,Necessary and sufficient condition to measure success 5.0 M words sampled Minimum of 3 languages Net Contribution to ROI by MT Engine: 10% of payable words should be MT No more than 5% adverse impact to Overall Quality Index No more than 5% impac

7、t to Customer Satisfaction Lack of industry metrics and guidance. Active research on MT technology. no guidance on operational impacts A business vacuum existed on how to integrate MT services No operational process had been defined for MT services,7,IBMs Watson Q&A computer Googles autonomous car T

8、echnologies to understand and produce natural human speech Instantaneous, high-quality machine translation Smartphones / App phones in the developing world,*Andrew McAfee is a principal research scientist in the MIT Sloan School of Business,Recent Digital Innovations with Biggest Impact in the Busin

9、ess World*,8,Real-Time Translation Server (RTTS) & n.Fluent,Real Time Translation Server (RTTS) IBMs MT Engine RTTS provides machine translation for n.Fluent & other applications APIs allow other applications to access these translation services. Customization tools Domains, chat-specific models, Co

10、mmercially licensed to IBM partners Language Pairs to/from English:n.Fluent IBMs MT translation application Providing machine translation services for:Text, web pages, and documents (Word, Excel, )Instant Messaging chats (via IM plug-in) Mobile translation application (BlackBerry and others) Enabled

11、 with LEARNING via crowdsourcing (internal 450K IBMers) Deployed for eSupport self serving tech support (external),中文,Deutsch,English,Franais,Italiano,日本語,Portugus,Espaol,9,Historical Perspective,2006,2007,2008,2009,2011,2012,2010,2010 MT piloting Pilot: SPA, ITA, FRE, GER - New E2E process Partners

12、hip: WWTO/n.Fluent 8.6 M words,2011 MT Training Pilot: GER, BPR, JPN, CHS - MT payment profiles ready 16.0 M words target,eSupport (www) “Translate This Page” JPN pilot / rule engine,Initial n.Fluent/WWTO Spanish MT pilot - Improve efficiency of professional translators,10,MT Pre-Process,Editing Ses

13、sion,MT Post Editing End to End Workflow,Upfront & on-going MT tuning via IBM TM professional translations Professional translation = Best context Matching methods Traditional TM breaks down content segment level Machine TM breaks down segments block level using MT models reconstructs segments prese

14、rving formats/mark-up tags MT service level integration,TM Pre-Process,TM Match Analysis,CAT Translation Show best choice vs vs Select best choice (Post Edit rules) Commit language,TESTING QUALITY,MT Model & Trans.,= Localization Kit (NLV Folder),11,18-sept.-08,MT Pre-processing,TM,Build dynamic, do

15、main specific MT model,MT,MT initial corpus,General parallel training corpus,Domain specific parallel training corpus,ALL segment “no match segments”,Translation of no match segments,Initial MT corpus done before start of project,Localization kit,12,18-sept.-08,Xxx xxx xx xxx xxx xxx. La aplicacin d

16、esprotege los archivos antes de exportarlos. Yy yyy yyy,TM Editing Environment,TM Environment,Xxx xxx xx xxx xxx xxx. The application unprotects files before exporting them. Yy yyy yyy,Translation Memory,0 - The application unprotects files before exporting them. 1m La aplicacin desprotege archivos

17、antes de exportarlos. 2f 85% - La aplicacin protege los archivos antes de exportarlos,TM Environment,Ctrl + 1,Typed,Translator options Ignore fuzzy and MT Post edit MT Post edit fuzzy,Two Seconds Rule: Translators are trained on several strategies to make a quick choice,TM,MT,13,Productivity Measure

18、ments,Start segment Choose actionEnd segmentMT productivity evaluation log (MTeval Log) N events Words | Time | Existing Proposal | Used Proposal | . Examine productivity per payment category SUM(Words) / SUM(Time) Use of IBM Business Analytic Tool (SPSS) Trim events that fall into 5% (slowest) and

19、95% (fastest) percentile,accept match 0 time edit match X time reject match manual translation,Each event,EM : Exact RM : Replace FM : Fuzzy MT : Machine NP : No Proposal,A) = “best” Existing Proposal B) = “alternative” Existing Proposal C) = reject all Existing Proposal, 100% human labor,14,Total #

20、 events : 2,309 (377+1,932) Total words: 24,150 Total time: 27,362 3,911 w/ MT match 11,377 w/ MT match 20,239 w/o MT match 15,985 w/o MT matchMT impact to productivity MT : 0.44 words/sec 1777 words / 4071 sec NP 0.21 w/ MT match 0.32 w/o MT match Baseline (placebo)MT Leverage : 71.8% 1777 / (1777+

21、697),Single Shipment EXAMPLE,rate(MT) / rate(NP): 1.37 i.e. Translator can complete 37% more words in the same time.,Key metrics,15,MT Impact on Fuzzy Match : 4Q10 Findings,When FM & MT matches exist simultaneously Productivity: rate(MT) / rate(NP): Case : Translator edits FM FM-MT Combined case Cas

22、e: Translator edits MT,* Findings subject to change with additional sampling.,Overall Machine matches not as good as professional (fuzzy) matchesNo statistical impact to fuzzy productivity to include MT matches. SPA highest sample case,28.6%,4.4%,57.6%,46.9%,FM-MT Pick Rate:,16,MT Key Metrics: 4Q10

23、Findings,8.6 M words sampled in real time translation service. SPA : Qualified MT engine 4Q10 ITA : Qualified MT engine 4Q10 FRA : Qualified MT engine 1Q11 While rate(MT) / rate(NP) is high, the findings were not statistically significant in 4Q. GER : Insufficient productivity from MT engine,* Findi

24、ngs subject to change with additional sampling.,17,Overall Savings Assessment,Overall savings % Word savings due to MT efficiency Convert time savings MT payment factor % MT payment factor X MT % words + NP % words Results in less payable words. MT productivity savings drives a overall savings These

25、 are not the same due to MT % distribution. Supply chain has to consider cost of MT services,* Findings subject to change with additional sampling.,18,Pay for MT Words Translated not MT Matches,We pay for final results (MT payable words) not MT matches MT matches considered “opinion” until chosen by

26、 a human Too many opinions & opinions by immature MT models are less efficient.Actual MT payable words have value beyond the specific project Post Edited words are reused in future and unknown MT contextEngine has to deliver consistent MT payable words Minimum needed to quality an MT engine for comp

27、ensation High MT productivity rate(MT) / rate(NP) High MT leverage % of MT matches used Compensation to be based on MT payment factor,19,Variance across Languages,There is no single maturity path when modeling MT engines across many languages. IBM Pilot: each trained MT engine is a unique asset. Som

28、e languages require more modeling/tuning than others. Language pairs that service “Loose - Structured” languages are struggling German requires more effort than SpanishAre there limitations to statistical MT engines? New thinking may need to be explored?Each MT engine will have separate MT payment f

29、actors.,20,Perspective of MT Post Edit Pilots,Quality / Reliability,LOWER,HIGHER,General,Domain Specific,internal IBM,All IBM external/internal Pubs / UI,external (2011 Pilots),internal IBM,n.Fluent “machine”,WWTO “human”,New,Memory Assets,MT Post Editing has impacts across entire Translation Servic

30、e Hierarchy,21,Professional (Human) memories are the best assets and deliver the highest quality. Professional memories are a key asset for MT success. All Memory assets need to be protected and managed. Flow of memories between Professional and Machine must be properly balanced. Dynamic modeling of

31、fers significant advantage over static modeling. Continuous business analytics is needed to optimize machine assets. A single cost model per language is needed, independent of MT services/engines. An aggressive yet cautious approach is warranted to go forward.,MT Post Editing Project Key Lessons,MT Post Editing does improve productivity and efficiency of a localization supply chain.,