A student stands before a shopkeeper on a Chinese market street. The shopkeeper announces that he has many things to sell, from cake to postcards. The student, who is still learning how to master Mandarin, ponders the selection. Like many people learning a new language, he must consider grammar, pronunciation, and tenses before answering. When it comes to Mandarin—which the Foreign Service Institute of the U.S. Department of State considers one of the hardest languages to learn, requiring a minimum of 2,200 hours of study—that’s especially difficult.
This scene is not actually playing out in China, but at an American university where a classroom has been equipped with a learning game, enhanced by artificial intelligence, that promises immersion into a foreign culture with zero travel required.
The project is the result of a collaboration between IBM Research and Rensselaer Polytechnic Institute, a private university whose graduates include the inventors of the digital camera, ductile iron and the first commercial television. The six-week, four-credit class, called AI-Assisted Immersive Chinese, aims to create the kind of interactions a student could normally only get in a study-abroad program.
ADVERTISEMENT
The class is built like a video game, with levels of increasing difficulty and scenarios full of unexpected variables. Students enter a classroom with a 360-degree panoramic display to face an enormous screen. They can haggle with digitized vendors hawking fruits or order Peking duck from a waiter at a crowded restaurant. The conversational AI skills these avatars are imbued with make it possible for them to respond and react in real time to the students, with a degree of unpredictability that makes it similar to conversing with other humans. Unlike the real world, where people must be willing to make embarrassing public mistakes in the service of learning a new language, here students can pause the unfurling scenario after messing up a pronunciation and replay it again to fix mistakes as needed.
Inside RPI’s new language learning lab, powered by artificial intelligence.
Kris QuaLawrence Luo, 19, is one of the 12 students. “I was just browsing the summer course catalog and I thought it looked pretty cool,” he told The Daily Beast. Luo, who is American of Chinese descent, has parents who speak Mandarin and he wanted to learn, too.
“I’ve managed to learn 10 to 15 new words already,” he said. That was on day one of the course, which runs for eight hours a week, two days in the classroom and two days in a room running simulations of scenarios where Mandarin might be needed, like asking for the bathroom in an airport or navigating a university campus. By the end of the summer, his professor is betting on him being able to navigate a conversation in Mandarin like a native.
“We’ve enabled the class to be like a game,” said Hui Su, director of the Cognitive and Immersive Systems Lab at RPI, and the project’s lead researcher. And like any game, there are multiple levels with growing difficulty. The students who signed up for this course are at varying levels of Mandarin proficiency. Their professors can set the level of dialogue difficulty for them individually, allowing each student to grow at their own pace.
The germ of this idea was planted in 2013, when some language teachers and some game development teachers at RPI wondered if they could band together to create a language learning game. But the language interaction capability of the technology available was limited. In 2016, after RPI had teamed up with IBM to create the Cognitive Immersive System Lab, they finally had all the tech they needed to create what they wanted. It was a perfect marriage of RPI’s immersive technology and IBM’s speech recognition software.
“Students were excited and engaged with the technology,” says Professor Helen Zhou, who is teaching the course and has assisted with the Mandarin project since 2013. “In traditional classrooms, teachers teach grammar and vocabulary and give homework. There’s very little real-life environment,” she says. “But here it’s not just static textbooks, students are surrounded for 360 degrees by the technology.”
The avatars keep track of how many words the students speak correctly for assessment purposes and provide a list of words students need to practice more. And while most tech needs a wakeup word (think “Hi, Alexa”), this AI, with its spatial context system, can detect where students are in the room based on cameras and sensors all over the room . If a student wanders near a restaurant table on screen, the waiter might ask her what she wants to order. The sensors that detect student movement and show how close they are to different areas of the screen will tell the the avatars initiate dialogue when they sense a person is nearby.
The system could have taught any language, but Su chose Mandarin because he wanted a challenge. “Mandarin is one of the five most difficult languages to learn,” says Su. It’s a tonal language, with four different tones. Each word can have four different meanings when pronounced with different syllabic emphasis. For instance, depending on the tone, “ma” could mean mom, or horse, or hemp, or scold. To deal with that, Su included a technology called pitch-tone contour analysis that assesses pronunciation and compares student pronunciation to that of a native speaker.
Su has already invited other schools to come check his new technology out. There’s talk of creating a mobile phone app people can use to practice their language skills on the go. Su says the technology is scalable to other institutions, although RPI refused to disclose how much it would cost to recreate the classroom.
