Point a phone at a dog and ask what breed it is. That was the pitch for Google Lens when Google showed it off at its I/O developer conference in 2017. TechCrunch’s Frederic Lardinois wrote that the feature, which tied image recognition to the Google Assistant, was among the highlights of the show.
The demo that drew the loudest applause, he noted, was Lens reading a Wi-Fi router’s network name and password off a label and connecting the phone automatically. Google did not ship it right away. In October 2017, at its hardware event, the company said a first preview would reach Pixel phones later that year and other devices “in time.”
What Lens could do at launch
Lens pulled together several Google services: image recognition, the translation tools behind Google Translate, and the Assistant. Snap a flower and it named the flower and offered more detail. Landmarks and restaurants worked the same way.
One year on
In December 2018, Google VP Aparna Chennapragada marked the first year in a post on the company’s blog. A redesigned Lens had rolled out across Android and iOS the week before, reaching iPhone users through the Google app.
She explained how the dog example worked. Lens drew on the hundreds of millions of queries Google Image Search had received for terms such as “Shiba Inu,” along with the thousands of pictures returned for each. Google’s open source TensorFlow framework connected those images to the labels “Shiba Inu” and “dog,” and the Knowledge Graph supplied the fact that a Shiba Inu is a breed of dog.
It did not always get things right. Chennapragada said photos taken in daily life look different from the web images used for training: wrong angles, poor lighting, subjects that will not hold still. Google was adding more training pictures that resembled smartphone shots.
Reading was the other big addition. Google built an optical character recognition engine and paired it with its language understanding from Search. Her examples: point the camera at a business card and add it to your contacts, or copy the ingredients from a recipe into a shopping list. Spell-correction models helped separate look-alike characters, so “c00kie” would resolve to “cookie.” The same engine let Lens read product labels, and by December 2018 it could identify more than one billion products, four times the number at launch.
A third feature, style search, let users aim the camera at outfits or home decor and get similar items, with reviews attached. If you liked a lamp at a friend’s house, Lens could show comparable designs.
Chennapragada described Lens as a first step toward what she called an “era of the camera.” Her list of what might come: directions overlaid on the street, products highlighted on store shelves, and instant translation of foreign words.
Sources
- The era of the camera: Google Lens, one year in
- Google Lens will finally make its debut on Pixel phones later this year | TechCrunch
