As e-commerce strengthens its hold on consumers, brick-and-mortar stores are being forced to rethink the way they do business. The need to create a customer experience that’s unique, fresh, exciting and cannot be reproduced in the virtual world is leading to the launch of future-type stores that leverage the power of information technology to enhance the customer shopping experience smoothly, while offsetting potential labor shortages.

To meet that trend, NEC has been using biometric technology. We develop systems to streamline store entry and payment, reducing or even eliminating waiting time at checkout. Such a system would offer more than just a new level of convenience, it would fundamentally transform the retail world and pave the way for biometrics to become a key part of everyday life in the 21st century. In this paper, we will show what such a system would look like, explain how it would work, and reflect on the profound changes it could generate in our society.

1. Introduction

The retail revolution is underway. After years of retrenchment in the face of the online onslaught, brick-and-mortar retail outlets are fighting back. Around the world — especially in the United States and China — new retail concepts that focus on speed, convenience, and the unique opportunity to try, touch, and feel products are being introduced in state-of-the-art stores that use electronics, sensors, and software to create an enhanced customer experience. The concept has now spread to Japan where companies are racing to develop stores that take full advantage of the power of information technology (IT).

Japan today is at the forefront of the “gray wave” that is sweeping across the industrial world as declining birthrate and aging population shrink the labor force and increase demand for more efficient and convenient services. With fewer potential employees, the retail industry is devoting considerable effort to streamline its operations, while at the same time enhancing the level of service provided. The goal is to ensure that consumers are still able to enjoy a comfortable, convenient, and pleasant shopping experience.

In this paper, we will reimagine the traditional store and show how new solutions using biometrics can revolutionize the shopping experience and take it into the future.

2. The retailers’ Issues

Over the past few years, e-commerce has grown by leaps and bounds, acquiring an ever larger share of the market as more and more consumers start shifting their purchasing activities from brick-and-mortar stores to online retailers. This shift has been accelerated by the proliferation of smartphones, which make instant shopping available anytime and anywhere. Between 2014 and 2017, mobile commerce grew by 150%. Nor is this growth confined solely to consumers looking for the best deals or the lowest prices. Instead, while still demanding maximum convenience, consumers are increasingly looking for premium products and services. Thus, when building a brick-and-mortar store today, emphasizing the importance of improving the customer experience is key to success.

Plummeting birthrate and a rapidly aging population have led to a marked decline in the size of Japan’s younger age cohorts, which in 2016 was 84% the size it was in 1996. At the same time, operations have become diversified in an effort to improve services, making jobs more complex and too difficult to easily assign to part-time workers. Consequently, there is growing demand for a labor-saving solution that can address these issues.

At NEC, that solution can be found in IT (Fig. 1).

3. An Overview of a Future-Oriented Retail System

In the world of e-commerce, it is now possible to discover what kinds of products and services an individual consumer wants and offer those products and services directly. Should the consumer decide to make a purchase, they can quickly and easily complete the payment procedure online.

On the other hand, to find what they want in a brick-and-mortar store, customers have to ask store clerks for information or physically search out the product themselves. Once they have found what they are looking for, they have to stand in line at the cash register, have the product scanned by the cashier, and then pay for it. It’s a time-consuming and often frustrating experience.

Future-oriented stores will eliminate most of these drawbacks by identifying individual customers via biometrics and then inferring what that customer wants based on their purchase history and the store locations they have visited so far. When the customer enters, they can be notified where to find products and given access to the store’s inventory. IT makes it possible to provide customers with assistance every step of the way, including payment.

To achieve this, a means must be established for identifying customers, understanding what they are interested in and looking for, and then offering them precisely what they want. It is also essential to have a mechanism that allows customers to easily make payment without standing in line at the cash register.

NEC is now developing solutions based on an array of state-of-the-art technologies including sensors, artificial intelligence (AI), databases, and biometrics. An overview of the new store system conceived by NEC is shown in Fig. 2. It includes processes for (1) store entry management, (2) shelf-front customer recognition, (3) customer behavior information acquisition, (4) shelf-front customer-directed promotion, and (5) payment (exit) management.

Granular analysis of customer behavior will make it possible to offer services specifically optimized for them. Collecting data on customer entry and exit, how they behave in front of the shelves, and what catches their eye (line-of-sight data) will help make it possible to optimize shelf layouts and product lineups and construct a system that will allow consumers to easily find and purchase what they want without being aware of the presence of this system.

In the future, with the customer’s consent, it will be possible to automatically complete payment simply by linking the product they want to purchase with their information. We are also investigating the feasibility of building a biometric authentication payment system using existing POS systems that will allow customers to enjoy shopping without carrying anything in their hands.

4. A Closer Look at Systems Using Biometrics

4.1 Entry/exit management system

This system manages customer entry into and exit from the store based on pre-registered facial data. Customers are asked to register their facial data before entering the store. This makes it possible to track who has entered the store, when they entered, and when they left. This data is combined with data about the customer’s circulation and interests (what they look at and what they pick up) and used to generate a basic inference of what they are interested in and what they are looking for.

For entry/exit management, we are currently thinking about adopting the walk-through facial recognition system developed by NEC. While conventional face recognition entry control systems require subjects to point their faces towards the camera so that a still image can be captured for authentication, NEC’s walk-through facial recognition system shoots video images and performs authentication on a continuous basis. This means that customers can be simply identified and authenticated as they approach the store, so there’s no need for them to stop and stand still while they are authenticated. With on-the-go authentication, customers can casually enter and exit the store with none of the stress that a more obtrusive system would instill.

At the same time as the customer is authenticated, their entry into the store is recorded. This data can be linked with the customer’s behaviors in the store and connected to the marketing information described in Section 4.3 below, facilitating targeted promotion of specific products.

4.2 Biometric authentication payment system

With the dream of a cashless society on the verge of realization, NEC is working to develop payment systems that will help achieve this. For quite some time, we have offered a multi-service gateway (MSGW) — a payment platform compatible with various payment methods, including digital currency and credit cards. To take this system to the next level, we are now planning to incorporate biometric authentication — such as face recognition — into MSGW. The idea would be to provide customers with individual IDs linked to their biometric information. A payment method — such as a credit card — would be registered in advance and linked to the customer’s ID, enabling hands-free payment with biometric authentication alone.

Customers would register their data for the MSGW using devices such as smartphones, dedicated terminals, etc. When face recognition is used as key data, the customer can use the camera in an ordinary smartphone. For other biometric authentication modalities, however, a dedicated device will be needed. When registering the data, information related to each customer is recorded in addition to the biometric information. Credit card information is registered when credit card payment is selected and company ID card information is registered if that’s the intended method of payment.

Once registration is complete, the customer is free to start utilizing the system. When they want to buy something, all they have to do is pick up the product and look at the camera to make the payment. The face recognition server identifies the customer and returns their ID. Based on this ID information, the customer is specified and the transaction payment is transferred to the payment server. This completes the process.

To speed up and simplify this process, NEC has also developed a biometric authentication payment package. This package can either be combined with the MSGW or offered as a standalone product. As for face recognition, NEC’s NeoFace Cloud can provide the basis for system that securely manages facial data. Interface functionality is shared with proxy payment service providers that handle credit card settlements to facilitate credit card payment using face recognition (Fig. 3).

Although we are currently building a system based on face recognition, our long-term goal is to incorporate other biometric information such as fingerprints, veins, and irises and combine that data with face recognition. In doing so, we can create a system with the ability to accurately identify individuals under a wide range of conditions, while maintaining convenience and minimizing intrusiveness.

4.3 Marketing system

Biometrics can also be an important tool to support marketing. Currently, we are looking at two types of biometric-linked marketing systems. The first controls digital signage content, sending targeted information to the customer’s smartphone in real time based on their inferred preferences, which are derived from their age, gender, and purchase history. The aim of this is to promote purchasing activity by offering the customer the information they want.

The second system would record the products purchased by customers who have entered the store. Then, by classifying those customers based on their customer characteristics and processing that data statistically, optimal solutions for product lineups and shelf layouts could be developed to match each store.

In the future, the use of line-of-sight detection will make it possible to capture data on what catches a customer’s attention. This information can be enhanced with image analysis technology and sensor technology that can identify those items the customer actually picks up and examines. This makes it possible to fine-tune the data regarding the customer’s preferences in an ever more granular manner. NEC is also developing a line-of-sight detection engine, object recognition engine, and motion detection engine, which can be used to create the system outlined above. Even when a customer leaves the store without buying anything, it will still be possible to obtain information about what has interested them. Analysis of this information will make it possible to enhance product lineups — which can contribute to reduction of opportunity loss.

5. Validation test at an actual store

As a first step to materialize this future-oriented concept, we opened a 7-Eleven on the 20th floor of Mita Kokusai Building (Minato-ku, Tokyo) for NEC Group employees in December 2018 as a test store.

This store is equipped with a variety of systems designed to enhance customer convenience and enjoyment, including face recognition payment and targeted advertising signage. In addition to those, various labor-saving measures to ensure smooth store operation have been implemented including a system that automatically collects the data from equipment such as refrigerators 24 hours a day, as well as a system for order proposals leveraging AI technology (Photo).

We regard this store as part of our commitment to future-oriented store systems. We are planning to continue this effort to deploy labor-saving stores in office buildings and other appropriate locations.

6. Conclusion

As biometrics helps fuse the warmth and physicality of traditional bricks-and-mortar with the speed, convenience, and immediacy of e-commerce, we can look forward to a new era in storefront retailing. In the future, even companies that until now have been strictly online will want to establish a physical presence in the real world. NEC is committed to playing a major role in this transition and is working hard to develop solutions that will continue to enhance and improve the customer experience, reduce the burden on store associates, and support innovative new retail services.

Authors' Profiles