Penn researchers teach a robot to read

A robot has one new skill under his metal belt thanks to Penn researchers.

Scientists in the General Robotics, Automation, Sensing and Perception Laboratory worked for three months to create software that successfully taught the Willow Garage PR2 robot to read.

Menglong Zhu, who is pursuing a doctorate robotics at Penn, worked in the GRASP Laboratory alongside Director Kostas Daniilidis and postdoctorate student Kosta Derpanis.

Other robots have previously been built for the purpose of reading. However, the software created at the GRASP lab allowed an existing robot built without these capabilities to learn to read.

The PR2 stands five feet seven inches tall and has two functional arms, a moving base and three different cameras.

Now that it’s literate, the robot could be “really useful” for blind people to read at home, Zhu said.

“It’s amazing how robots like these can not only accurately interpret information from their environment but also relay it back to humans,” rising Engineering junior Vivek Sharma said. “I could see applications for blind and elderly people, or as a personal assistant.”

Creating the software was not easy — Zhu worked for a month alone on the algorithm implemented in the software.

The code that allows the robot to read is implemented in three steps. In the first step, the robot locates text in a natural environment. When this is accomplished, optical character recognition searches over a dictionary for a match in similarity in letter-by-letter appearance. In the final step, recognized symbols are converted into speech.

The most difficult part of the project, according to Zhu, was programming the robot to distinguish between similarities in letters and numbers, such as among “L”, “I”, and “1”, especially in human handwriting.

The code will be considered open source, meaning it will be shared freely with other researchers in the Robot Operating System library. This allows the code to “remain open for anyone to keep working on,” Zhu said.

Zhu has started working on a new project, which will take the idea of letter recognition a step further — he plans to create a program that implements object recognition in the PR2 robot.

Please note All comments are eligible for publication in The Daily Pennsylvanian.