• Facebook AI is on a quest to build the world’s largest shoppable social media platform, where billions of items can be bought and sold in one place.
• As a key milestone toward this goal, we’re sharing details on how we’ve improved and expanded GrokNet, our breakthrough product recognition system. Now, it’s powering new applications on Facebook, like product tagging and showing visually similar products based on the seller’s catalogue. Soon, we’ll bring visual search to Instagram so that people find similar products just by tapping on an image.
• We’re also diving into our latest AI advancements that provide a deeper, more nuanced understanding of product attributes and multimodal signals. These advancements collectively represent fundamental building blocks that could power entirely new shopping innovations of the future.

At any given moment, the web is a treasure trove of shopping inspiration — brimming with fashionable trends, seasonal tablescapes and artful shelfies. But how many times have you seen something you want, but can’t figure out how to buy it or even check it out?

Product recognition is among the most important ways to make it easier for people to shop online today. If AI can predict and understand exactly what’s in any given virtual frame, then people could — one day — choose to make any image or video shoppable. People would more easily find exactly what they’re looking for and sellers could make their products more discoverable.

Facebook AI is building the world’s largest shoppable social media platform, where billions of items can be bought and sold in one place. As a key milestone toward this goal, we’re sharing details on how we’re expanding GrokNet, our breakthrough product recognition system, to new applications on Facebook and Instagram. GrokNet identifies what products are in an image and predicts their categories, like sofa, and attributes, like color and style. Unlike previous systems, which required separate models for each vertical, GrokNet is a first-of-its-kind, all-in-one model that scales across billions of photos across vastly different verticals, including fashion, auto and home decor. GrokNet started as a fundamental AI research project with its first few applications on Marketplace, where AI analyzes search queries like “midcentury modern sofa” and predicts matches to search indexes so that the over a billion people who visit Marketplace each month get the most relevant results when searching for products.

Since 2020, we’ve expanded this technology to new applications to make posts more shoppable across new Facebook applications. Right now, when a seller posts an image on their Facebook page, our AI-powered shopping system helps identify untagged items and suggests tags based on their product catalog — so that instead of taking several minutes to manually tag their items, a seller could create and post their photo in just seconds. And when a shopper is viewing an untagged post from a seller, the system instantly suggests similar products below the post from that seller’s product catalog.

With billions of images uploaded to Shops on Facebook and Instagram by sellers, predicting just the right product at any given moment is an extremely hard, open AI challenge.

Today we’re sharing details on our newest state of the art advancements that are making our AI systems remarkably smarter at recognizing products — from multimodal understanding to learning deeper, more nuanced attributes. These advancements not only strengthen current applications but they’re also building blocks of future shopping experiences.

For example on Instagram, shopping begins with visual discovery. Every day, people scroll through the app and see thumb-stopping inspiration — whether that’s a floral dress for summer or the perfect wedding dress. With AI-powered visual search, people could find similar dresses just by tapping on an image they see within Instagram. While it’s still early, we think visual search will enhance mobile shopping by making even more images on Instagram shoppable. 

And with each new advancement, we’ll cumulatively push AI research to go beyond finding similar products to entirely new, more flexible tasks like: “Find a handbag with the similar pattern or embellishment as this dress.” When you find the right product, one day, we could build on top of this technology to create new immersive innovations of the future.

Read the full blog post with technical details on our latest AI advancements on the Facebook AI blog.