Pinterest has been working on redefining visual search and recommendations over the last few years, with creations such Related Pins, Similar Looks, and Flashlight, all powered by their machine-learning systems. Recently, they have released a new paper about their new Lens project which plans to bring real-world search to the Pinterest app. The idea is that a user could point their camera at a wall clock and Pinterest will scan their billions of images and return similar designs, styles, and/or other wall clocks, all powered by the Pinterest deep learning AI.
Pinterest Lens was built upon the previous technology Pinterest developed over the years. The first visual search tool that made a huge impact was Flashlight, which allowed a user to highlight any item in a Pinterest post and do a visual search on their findings (such as a black floor lamp off to the side in a living room). A year later in 2016, Pinterest followed up with automatic object detection, which allowed users to simply take a picture and have Pinterest return results for the one item (i.e. a pair of sneakers) or multiple items (i.e. a living room set).
Pinterest Lens builds upon both of these technologies (along with many others) to provide users with a quick visual search tool that connects them to the billions of pins on the site. Pinterest stated that taking a picture of blueberries wouldn’t just provide similar pictures of blueberries, but rather a large amount of related content. The delivered content is very diverse, such as blueberry recipes, beauty tips involving blueberries, and even gardening tips on how to grow your own blueberries.
Lens works through the use of 2 components which handle different tasks to return relevant results. The first component of the software is referred to as the Query Understanding Layer. This layer analyzes the visual image, the annotations, the colors, lighting and the visual clarity of the image to pass that information onto the second layer, called the Blender. The Blender layers reads all of the information from Query Understanding Layer and provides visually similar objects and/or related text searches based off of the information provided through the initial image. Lens also includes Object Search, which is a visual search system that allows users to search for objects in the real world and see how how they appear in the context of related images. The Pinterest Engineering article demonstrates this by searching for a brown chair and showing how the system would return the same style of chair in different colors in pictures next to other furniture.