In Part I of this article we discussed the basics of image recognition accuracy.
In this part, we will give you a deeper understanding of how we calculate an accuracy level and provide you with the formula we normally utilize to calculate this metric in customer projects. You will also learn about the particular aspects of calculations using our formula, as well as how we handle various challenges that arise during calculations.
What is the general formula to calculate image recognition accuracy?
Generally, the formula to calculate image recognition accuracy is:
100% - The percentage of errors the neural networks make in the course of an analysis
Accuracy is calculated for a definite number of images containing identified goods. Suppose the system detects and identifies goods in one thousand images. The next step in the process is calculating the number of errors made by the computer. Normally, one image contains up to 30-50 products, depending on the distance to the shelf from which the image was captured. This means that 1,000 images contain 30-50 thousand goods.
To define an image recognition accuracy level, one should assess how many times neural networks detect and recognize products incorrectly, or generally speaking, one should check every product in each image and determine if it is identified correctly.
This task is usually performed manually by well-instructed staff hired by a vendor for a project or by a customer as part of an in-house team. One more option widely used both by vendors and customers is to hire a third-party company specializing in the support of such projects.
Our customers frequently ask us why neural networks utilized in the Eyrene platform do not do a self-diagnosis, or, strictly speaking, why the image recognition accuracy cannot be calculated automatically without a third party. We explain that if neural networks could detect errors, they would definitely correct them. But even when neural network models are well-trained, they are still not able to make an accurate self-check for errors. The task has still not been resolved by data scientists.
A significant number of the errors made by neural networks during image analysis is due to changes happening with the goods distributed by brands at points of sale. Situations in which neural networks can make mistakes include new packaging of regularly produced goods, new products by the manufacturer, and new sales channels utilized by brands.
When working on customer projects, we frequently come across situations where neural network models integrated into our platform have not yet been trained to identify new products, as they were not added into the customer product catalog. For instance, some local products can be sold only in some specific regions. The only solution to avoid a negative outcome is to add all the updates to the product catalog in time. The customer is responsible for that part.
How do we calculate image recognition accuracy at Eyrene?
At Eyrene, we developed our own approach to calculating image recognition accuracy. We will explain why we use it in the next parts.
According to our approach, image recognition accuracy can be calculated by the following formula:
1 - (number of mistakes)/(total number of goods to be processed) * 100%
The fewer the errors, the closer the accuracy level is to 100%. The higher the number of errors, the lower the accuracy level is.
For a deeper understanding of what image recognition accuracy is, we should explain some aspects of the formula. We usually discuss most of them with customers in the course of the platform implementation and exploitation.
Most of our customers task the Eyrene platform with identifying the goods manufactured by the brand. This problem is comparatively easy to solve, as neural networks are already trained to identify brands' own products on the shelves and to retrieve detailed information from a catalog, including the flavor of yogurt, the number of items on the shelf, and so on.
At the same time, customers may also need to identify all the goods on the shelves. The task to identify competitors’ products has a higher priority and is really challenging, as the number of competitive goods’ characteristics available for image recognition is lower compared to situations where only the brand’s own products need to be identified. These characteristics may include only a product’s category, brand, and price.
One more task considered challenging in image recognition is identifying the product’s size. Errors in identifying the size are the most frequent. There are some product categories for which size identification is not a priority, for example, goods manufactured by minor competitors. For them, there is no need to identify all the characteristics; it’s more than enough to identify only the availability of competing products on the shelves. For tasks of this kind, the accuracy is higher.
Our customers commonly are quite concerned about image recognition accuracy for their own brand’s goods, as this characteristic is used for estimating commissions for sales reps.
The most relevant case for applying our formula is identifying customer’s goods. At the first step, the procedure involves identifying the brand’s goods in the images, and then the number of errors made by neural networks is calculated. After that, this figure is divided by the total number of goods. The result is the level of image recognition accuracy for the customer’s products.
One more aspect of the formula relates to cases where even a human cannot name the product in the image. This situation is quite common; we observe it in large datasets. A product on the shelf can be captured in such a way that it’s next to impossible to identify what it is. According to our approach, we exclude unidentified products from the database, as there is no way to understand if this is a mistake made by neural networks. This means that our solution doesn’t consider an unidentified product as an error.
And what if a product cannot be identified, as there is no information about it in the product catalog due to its new packaging design, or the product itself is new? In this case, the solution won’t identify the product. However, we collect data about unknown products. We discuss these cases with the customer separately, as they cannot be qualified as mistakes since the solution doesn’t have any data about the products in the database. As a result, the case of unknown products can be considered as a case of incomplete data, which needs resolution as soon as possible.
What's next
In Part III, you will learn about the dashboards available in the Eyrene platform, as well as the types of errors.
