How to fool retail image recognition (part 2)

published on 02 June 2022

In the second post about fraud, we reviewed the first three types of fraud, which are quite simple and are utilized by merchandisers all over the world. To recap, they are: using fake camera apps, taking images with the camera of another device or from a printed image, and providing images taken in other stores. There are two more types of fraud, which we will cover here.

Staged photos 

This kind of fraud can be called the highest level of fraud. In this case, a merchandiser brings a new product to every store he visits, for which he receives the highest scores. Brands struggle to bring all their new products to the stores as soon as ads are distributed via all possible channels. 

As we already discussed in one of the previous posts, brands want their products (especially new ones) to be placed in the most favorable areas in terms of sales. That’s why, in order to get the highest scores for demonstrating new products in such locations (for instance, the checkout area), merchandisers can cheat the system by placing the product they brought in the required zones. Some merchandisers even take photos in the car after they leave the store when they realize they need to take some more pictures to meet their KPI. We’ve faced these situations from time to time. 

An image of a product taken in a car
An image of a product taken in a car

In practice, this kind of fraud is hard to eliminate, as there is no way to prohibit merchandisers from moving products around the store. From a technical standpoint, it’s also hard to detect that the same exact product is appearing in more than one photo. However, Eyrene can easily recognize images taken in a car. It’s the same neural network that is trained to recognize if the picture was taken from a screen of a mobile device; it can also detect pictures taken inside and outside the store. 

This kind of fraud can still be beaten with a carefully designed reward system. We noticed that if there are no additional bonuses for taking pictures of the same product in various locations in the store, there is no cheating of this kind. If incorrect geolocation or pictures taken from a mobile device screen are easily eliminated at the software level, such things as one product photographed in several areas of a store can be eliminated at the level of reward management. This means that managers responsible for building reward systems should take into consideration these facts and shouldn’t provoke merchandisers to cheat this way.

For instance, brands are interested in placing at least several items of a specific product in one area, it’s also called multi-facing. In this case, it’s reasonable to require merchandisers to show all product items available on the shelf in one picture, as the system can be easily cheated by presenting several pictures of the same exact product in one area. 

To understand how tricky people can be when they want to cheat the system, we would like to tell you a story from one of our customers. They didn’t use the geolocation filter for their merchandisers, so the beginning and the end of the visit were not identified; the staff could provide any kind of information they wanted. After a year, the geolocation feature was enabled, and it turned out that one of the merchandisers was so inventive that he had recreated a whole store full of the required products at home. The merchandiser selected one of the stores he should visit in the app, took pictures of the required products, and sent them to the system. As a result, he got his bonuses and reached his KPIs. This situation continued for a whole year and ended when our customer decided to enable the geolocation feature available on the Eyrene platform. 

Fraud organized by a group of people

In major projects, we deal with merchandisers (or end-users) who are not the staff of a manufacturer or brand. There are professional agencies that provide merchandising services and recruit merchandisers onsite. Another group of professional companies that recruit merchandisers is distributors. The key tasks of a distributor are to bring brands’ products to regional markets, provide marketing activities around them, and so on. There is also the case of a longer supply chain, including distributors and regional merchandising agencies hired by a distributor to provide services in specific regions.

As it turns out, fraud can be organized not only by merchandisers on site but by whole groups of people who can be a part of the supply chain leading to a manufacturer. For instance, with one of our customers, we came across a situation where a merchandiser had two smartphones and did the job of two onsite specialists. These two phones were provided by the company. So the key goal was to show the job of two specialists simultaneously as if there were two merchandisers working in one store, and the company was to collect payments for them. 

On the Eyrene platform, we have a solution for this issue: simultaneous visits of several merchandisers in one store are prohibited. However, this rule is not universal for all customers, as some brands can have products in several product categories presented in stores or hypermarkets. In this situation, several merchandisers can be in one store at the same time. However, on the Eyrene platform, we can set up specific rules allowing several merchandisers to visit one hypermarket simultaneously, but prohibiting two merchandisers to visit smaller stores at the same time; such visits can simply be marked as suspicious. 

This kind of fraud is not easily detected, as there is no chance to identify it on an individual basis; that’s why the full scope of data should be analyzed. This functionality is available on the Eyrene platform; we just set up a number of triggers that detect this fraud, and that’s it: our customers are protected against it. 

Summary 

Here, we discussed two more types of fraud, including using staged photos and fraud organized by a group of people. 

Photos can be easily staged by merchandisers, and we’ve observed it many times in various situations. However, this can be beaten by implementing the geolocation and geofencing features available on the Eyrene platform. 

As for fraud organized by a group of people, it can be cured by implementing business rules in the rewards system as well as collecting data and identifying behavior patterns not only of a single merchandiser but of whole organizations participating in the supply chain, like distributors and merchandising agencies. 

Read more