Real World AI: Checkout-Free Retail (Amazon Go and Zippin)

Checkout-free retail technology is an exciting application of AI that’s changing the way we shop.

---

Contents

• What is checkout-free retail?
• Benefits for customers and vendors
• Amazon Go and Zippin
• How checkout-free technology works
• Summary

Artificial intelligence (AI) is helping to provide better solutions for many of our everyday tasks, and a simple visit to your local grocery store is no exception.

With recent advances in computer vision, deep learning, sensor fusion technology, computer processing power and cloud computing, checkout-free retail stores are now a reality.

In this article we explore this exciting application of AI in the real world.

What is checkout-free retail?

Imagine a visit to your local grocery store. You walk in, after a quick scan of your phone, and walk around to grab the items that you need.

You pick up a few items off the shelves. You change your mind and return one or two items. You then walk through the store freely to make sure that you have everything you need.

When you’re finished, you simply walk towards the exit and straight out of the store.

No cashier. No checkout. No queues. No delays.

And the items that you bought—the ones that you picked up, but not the ones that you returned to their shelves—are fully itemized and billed to your account through a mobile app.

You’ve just experienced checkout-free shopping!

And if it isn’t already in your neighborhood, then it may be coming there soon.

Benefits for customers and vendors

The idea of checkout-free shopping has obvious appeal for customers—ease, speed, convenience and contact-free—provided the technology works.

And there are significant advantages for vendors as well, including:

• Real-time inventory updates, since all product purchases are continuously tracked
• Detailed analytics, given the rich data that’s collected
• Better use of store space, since there’s no need for bulky human-operated checkout facilities
• Lower operating costs, as dedicated checkout staff are not required

Many of the vendor advantages can also improve customer experience as well.

Lower operating costs, for instance, can feed into lower product prices, and the staff formerly employed at checkouts can move to more value-adding functions, like customer assistance.

Amazon Go and Zippin

Checkout-free shopping sounds amazing, but it’s not entirely new. It’s been around for a few years and has a growing footprint, particularly with the two leaders in the field: Amazon Go and Zippin.

Amazon Go opened its first beta store in 2016, available only to employees, and has had public stores open since January 2018.

Zippin is newer, first demonstrating its technology in August 2018 but now operating across numerous countries and venues.

Amazon Go calls its checkout-free experience “just walk out” technology whereas Zippin prefers to just call it “checkout-free“.

The basic technology framework used by Amazon Go and Zippin is similar although their business models differ—Amazon Go runs Amazon-owned and operated stores whereas Zippin sells its technology to third-party operators who wish to use it. Zippin doesn’t own and operate its own stores.

Zippin’s technology is also being used in retail environments other than grocery stores—entertainment and sporting venues, for instance, corporate campuses and even stores-within-a-store.

How checkout-free technology is implemented by Amazon Go and Zippin

The simple grocery store visit described earlier is at the heart of checkout-free technology—a customer visits a store, picks up a few items and walks out of the store without checking out.

All of the items that they select should be correctly identified and billed through a mobile app.

Quick, simple and efficient.

But while the process sounds straightforward in concept, it’s extremely difficult to implement in practice.

The challenges of checkout-free implementation

The practical challenges of implementing checkout-free technology are many and varied—let’s consider a few:

Identifying customers

Customers need to be identified as they walk in a checkout-free store, tracked as they move around and correctly associated with the products that they pick.

Biometric identification should be avoided due to privacy concerns, yet each customer needs to be uniquely tracked.

What about crowds?

When lots of customers are in the store at the same time, they shouldn’t be confused with each other and their product picks should not be misassigned.

Sometimes, two or more customers may enter the store with a shared account—family members, for instance—and each may pick their own products. In these cases, all of the selected products should be associated with the single, shared account.

Identifying selected products, accounting for the number selected and excluding returned products

The variety, shapes, sizes and colors of products on offer (and their packaging) is huge, yet all selected products need to be correctly identified.

Some products may be picked up but then put back on their shelves (if a customer changes their mind)—these shouldn’t be charged for.

Sometimes, a customer picks up a product and hands it over to someone else—the technology should not confuse which account the product belongs to.

Similarly, a product may be picked up and then put in a bag, a trolly or shopping cart—again, the technology needs to correctly account for the selected products regardless of how they’re carried around the store.

There are numerous other scenarios involving customer selection, placement and movement of products around the store—the technology needs to successfully manage as many of these as possible.

The types, placement and number of sensors

What types of sensors are best for identifying customers and products in the complex real-world scenarios that we’ve been describing?

Cameras are an obvious choice, but where should they be placed and how many should there be?

Cameras can also easily be obscured or may not capture enough image data for correct identification.

Some additional types of sensors would be required to provide a more complete sensory description of the various possible user scenarios.

But when different types of sensor devices are used together, another challenge is to coordinate their sensory data—sensor fusion—so that it’s aligned for any given customer or product interaction.

Computing and networking requirements

At the heart of the system and bringing all of the sensory data together are cutting-edge AI algorithms. This needs a sufficiently capable computing and networking solution.

Where should the different computing tasks be carried out—locally or remotely (in the cloud)? What are the bandwidth requirements? What are the required types of computing hardware and software components?

The implementation solution

It’s clearly quite a daunting task to successfully design and deploy a checkout-free technology solution, yet Amazon Go and Zippin have been doing just that.

How do they do it?

Through a tried-and-tested, carefully designed combinations of the latest AI and deep learning algorithms, versatile hardware and practical store layouts.

While some elements of their implementations differ, both Amazon Go and Zippin adopt a similar approach.

In Zippin’s case, the key elements of the implementation approach are as follows (as detailed in their descriptive white paper¹):

• A combination of over-head cameras and on-shelf weight-detecting sensors
• Advanced deep learning algorithms for visual recognition tasks
• Image processing computed on-site and sensor-fusion analysis computed in the cloud
• Backup 24×7 monitoring
• Individual mobile app-based account management

Using this framework, Zippin has tackled most of the challenges described above and is successfully deploying its technology across a range of use cases.

Zippin is now used in large retailers, small convenience stores, stadiums (Sacramento Kings Golden 1 Center and Denver Broncos Aramark), corporate campuses (Fujitsu employee convenience store in Tokyo) and stores-within-a-store (special “store-within-a-store” aisles in Azbuka Vkusa stores in Russia).

Amazon Go is also growing and had 27 stores operating in the US and 3 in the UK as at early 2021.

How the implementation solution meets the challenges

The implementation approach described above meets the challenges of checkout-free environments in a number of ways—let’s look at some:

Over-head cameras, attached to the store’s ceiling, can be placed throughout the store and avoid facial recognition. This helps to alleviate privacy concerns while the over-head positioning allows the unobstructed tracking of customers throughout the store.

Customers are not identified biometrically, but rather as pixelated objects (based on “top of the head” views) that can be uniquely tracked.

Individual customers are handed off from camera to camera as they move through the store.

Combining cameras with on-shelf weight-detecting sensors improves product identification and helps to recognize product returns.

This is important since visual data alone is insufficient to cover situations where a camera’s line of sight is obscured or where it’s hard to detect if multiple products are being handled. By combining visual data with weight-detecting sensors, these situations can be identified with much greater certainty.

By separating computing tasks between on-site and the cloud, a more efficient balance of computations can be handled. The on-site computations reduce the required bandwidth needed for the cloud and provide immediate results.

Backup monitoring can help to rectify issues and provide (remote) human assistance for complex technical glitches.

Managing each customer’s account details and purchases through a mobile app allows portability and sharing—if a family group enters a store, for instance, each family member entering the store would be scanned to the same account and the product picks from all family members can be tallied to the same account.

Implementation Feature	Benefits
Advanced AI and deep learning algorithms	Cutting-edge capabilities for tracking customers and identifying products
Combination of over-head cameras and weight-detecting on-shelf sensors	Privacy-friendly customer-tracking and accurate monitoring of product selections and returns
Distributed computing on-site and in the cloud	Lower cloud-bandwidth requirements
Backup monitoring	24×7 assistance and trouble-shooting
Customer mobile apps for account management	Scannable app for store entry and exit, real-time identification of in-store customer groups (on same account) and immediate itemized feedback on purchases (after store exit)

Overall, the solution is designed to provide a positive and versatile experience.

When the concept was fairly new in early 2019, Forbes contributor Andria Cheng visited an Amazon Go store in San Francisco to experience the technology first-hand². She seemed impressed by the setup.

There were a number of excited customers in the store during Cheng’s visit. They were engaging with the technology and had questions like “What if you put things back on the wrong shelves?” and “How does the system work for a dad who walks in with his family?“.

As we’ve described, the system can handle these situations, and the in-store Amazon Go representatives were happy to explain so on the day.

In summary

• Checkout-free shopping is a growing retail trend powered by advanced AI, computer processing and sensor fusion technology
• Checkout-free shopping is quick and convenient for customers while offering lower operating costs and better data monitoring for retailers
• Amazon Go and Zippin are two leaders in the checkout-free technology field, with Amazon operating its own stores and Zippin offering a fit-out service for other retailers’ stores—both are enjoying a growing footprint as more stores successfully roll out
• The challenges of implementing checkout-free solutions include customer and product identification, tracking movement in stores, the choice and placement of sensors and adequate systems for computing and processing
• Amazon Go and Zippin adopt a similar implementation approach that includes ceiling-mounted cameras combined with weight-detecting on-shelf sensors, advanced AI systems (deep learning computer vision and sensor fusion data), distributed computing networks and customer-centric mobile apps

References

[1] Zippin whitepaper, Checkout-Free Shopping: The Future of Retail Has Arrived, downloaded from: https://www.getzippin.com/the-future-of-retail-is-checkout-free, May 2021.

[2] Andria Cheng, Why Amazon Go May Soon Change The Way We Shop, Forbes (online), January 2019.