May/June 2005

Technology Trends: Synthetic Interviews - Beyond History Calls

By James A. Larson
Intel Corporation

Matt Nickerson¹ describes how mobile phones enable callers to speak and listen to virtual agents.  Using the same device to speak with family, friends and business associates, callers speak with software agents that enable synthetic interviews with individuals in photographs of historical events in a museum.  This represents a new way of interacting with objects that are usually only viewed.  Synthetic interviews enable callers to go beyond the picture to better understand the event or concept represented by the picture.

Today's mobile phone users can be "part of" the experience more so than just listening to a recording, watching a video, or just looking at a display in a museum. Synthetic interviews enable callers to:

• Interview individuals in a photograph, book, magazine, newspaper, or brochure.   Imagine, talking with Emeril Lagasse about one of his recipes, Brad Pitt about his latest movie, Britney Spears about her latest marriage, or Albert Einstein² about his life and theories    
• Converse with a World War I airplane pilot while examining his airplane
• Chat with soldiers and fur trappers while exploring a fort
• Talk with members of the court while exploring a castle
• Question the architect of the Brooklyn bridge while walking across it
• Talk with Leonardo daVinci while looking at the rings of Saturn through a telescope
• Talk with Abraham Lincoln while visiting the log cabin in which he was born 
• Ask a zookeeper questions about animals when visiting a zoo, or better yet, talk with the animals themselves

You can dial the phone number of the synthetic agent yourself.  Use the camera in your mobile phone to photograph the phone number (or bar code), use vision technology to recognize the telephone number, and automatically dial it.

Synthetic interviews become more interesting when there are multiple synthetic agents.  For example, suppose synthetic agents representing an Indian, a settler and a mountain man all answer questions from their unique perspectives?  History lessons can spring to life with multiple points of view.  Why wasn’t history this fun when I was in school?  Instead of writing class reports, teams of students can create sets of agents talking about an event in history, a geographical place in the world, and famous people - both living and dead.  Creative writing becomes script authoring.  Only the teacher reads student reports, but many people will enjoy interactive conversations authored by creative students.

Most phones have small displays.  While a picture in a book or newspaper may trigger a conversation with a virtual agent, the conversation is no longer constrained to that picture.  The mobile phone can download pictures, diagrams, charts, and scenes that facilitate an even more interesting discussion. 

Imagine shopping using a mobile phone.  The mobile phone displays pictures of the item you are interested in purchasing, and you ask a virtual store clerk questions about the item.  If you want to buy the item, place it in your virtual shopping cart and pay for it during checkout.  While you can’t feel the item, or try it on, you avoid the travel, the crowds, and standing in line to purchase your items.  Shoppers in a real store could also call virtual agents when a real agent is not available.

Today, we must go to the study or wherever the PC is, turn it on, wait for it to boot up, and invoke a Web browser just to get a small piece of information. The telephone can make the same information available wherever you are, whenever you need it.  George White  foresees a future world in which users interact with almost anything by speaking and listening.  

There are lots of consumer devices competing for consumer ears — radios, boom boxes, MP3 players, and the exciting iPod.  While these devices deliver content, they don’t allow users to ask questions.  Mobile phones are widely used to communicate with family and friends.  They can also be used for synthetic interviews.

¹Nickerson, Matt. History Calls: Delivering Automated Audio Tours to Mobile Phones.  Speech Technology Magazine 10 (3), 24-27.
²cs.cmu.edu/~sms/CMU_AlbertEinsteinDemo.htm