3 fascinating applications of deep learning in image classification

Ruthger Righart
Data scientist

RSS Feed

Email: [email protected]
Web: https://www.rrighart.com

---

Image classification is the process of assigning text labels to photos. Automated image classification has the last decade had a great impulse by the rapid evolution of deep learning, based on convolutional neural networks (CNNs). CNNs are multilayer neural networks that can capture patterns from image classes, without extensive preprocessing.

In 2012, the AlexNet CNN architecture enabled a dramatic performance increase in image classification. It recognized 1000 image categories from the ImageNet data with astonishing accuracy (top1 and top5 error rates 37.5% and 17%). Now almost ten years later CNNs have been successfully applied in diverse domains, such as face recognition, digit recognition, classification of objects and scenes, often based on big data needing expensive computing resources.

There are however use cases using smaller data that nevertheless achieved quite impressive results. It is very useful to learn from these use cases and how they could implement their wonderful applications in diverse "niche" areas.

Below I will shortly highlight 3 remarkable use cases that are already changing research and industry.

Photo by Francesco Ungaro ( https://unsplash.com/@francesco_ungaro ).

Identifying pieces of ancient pottery

Classifying ancient ceramic pottery is a laborous job for archaeologists. It takes years of professional experience to learn the job. Studies have shown substantial disagreements on ceramic type assignment between even experienced archaeologists.
Convolutional neural networks (CNN) have been used to classify 7 types of Tusayan White Ware (TWW) pottery based on the decorative style [1]. CNN could achieve an accuracy that is comparable to, and sometimes better than, that of experienced human classifiers. In addition, using heatmaps the CNN could show which image design features were most important for type classification. This information was used to determine the degree of similarity between different images (using the cosine similarity metric). In turn, they could make a search engine, which they called with a wordplay «sherdch engine», in order to find pottery with similar characteristics.
Automatic screening of pottery allows archaeologists to concentrate on more ambiguous classifications and will free up their time to focus on the scientific implication of the results.

Early detection of melanoma

The early detection of melanoma is a crucial prognostic factor for skin cancer. Using CNN in this domain promises to improve diagnostic testing.
A clinical research study used biopsy-verified images and trained algorithms to perform binary classification of melanoma versus nevi [2]. Nevi, also called moles, develop when pigment cells grow in clusters. Unlike melanoma, they are not cancerous. It is important to distinguish between the two classes.
Performance of a CNN deep learning algorithm was compared with a group of dermatologists that evaluated 804 images (402 melanoma, 402 nevi). The dermatologists were from 9 university hospitals in Germany. To evaluate performance, sensitivity (number of true positive cases relative to all positive cases) and specificity (number of true negative cases relative to all negative cases) were used.
The algorithm (sensitivity=82.3%, specificity=77.9%) outperformed the junior and board-certified dermatologists (resp. 67.2% and 62.2%). As the authors of the study indicated, a drawback was that the testset images originated from the same overall dataset that was used for training, which limits generalisation to external data, like data from other institutes.
The results show that deep learning can be used as a tool to help dermatologists in the clinical practice. In the future, similar technology may be used in mobile devices which will help diagnosis outside the clinic.

Monitoring coral reef

The health of coral reef is under pressure due to climate changes such as ocean acidification and rising seawater temperature. Decline of the coral reefs endangers oceanic biodiversity and leads to the breakdown of the ecosystem. Therefore, rapid detection of coral reef damage is essential.
The manual classification of coral species is tedious and the process requires professional expertise.
To automate this process, researchers decided to train a CNN algorithm on 5500 underwater coral images in the gulf of Eilat, classified into 11 species [3]. The overall accuracy was at 80.13%. Challenges of applying deep learning in this context are the large number of coral species, the great variance among images of the same coral, the different lighting conditions as well as cluttered and occluded species. One of the future challenges is using deep learning technology to monitor the temporal dynamics of the coral reef health.

Recommandations for building your own system

These are some specialized and diverse domains that benefit from deep learning technology.
All these studies used reasonably large but not extremely big datasets.

Big data are not obligatory for every use case. In fact, it is more relevant to have high quality data that represent the categories that your system will need to classify when it is in production. Start building a classifier with a small number of classes and estimate which performance you would expect. It is good to realise that performance requirements vary across domains, for example misclassifications in melanoma diagnosis have a larger human impact as compared with misclassification in pottery or coral reef research.

My rough estimate is to collect for each category inbetween 500 and 1000 images. This can be more or less depending on how hard it is for the algorithm to discriminate between the classes. After this, evaluate how a simple CNN model is doing the job, possibly adding transfer learning to the game. Examine what kind of errors the model is making (using confusion matrices), and evaluate which categories seem to be difficult and why. Try to improve the image data by removing annotation errors and increasing the number of images. There is much to win by using a data-centric approach [4]. In this way, iterate multiple times through the cycle of data quality control, training the algorithm and cross validation.

Please feel free to contact me for questions and advice, I am glad to share my personal experience setting-up an image classification system.

Ruthger Righart (PhD)
Data scientist
Email: [email protected]

References

[1] Pawlowicz, L., & Downum, C. (2021). Application of deep learning to decorated ceramic typology and classification : A case study using Tusayan White Ware from Northeast Arizona. Journal of Archaeological Science, 130, 105375.
[2] Brinker, T., Hekler, A., Enk, A., Berking, C., Haferkamp, S., Hauschild, A. (2019). Deep neural networks are superior to dermatologists in melanoma image classification. European Journal of Cancer, 119, 11-17
[3] Raphael, A., Dubinsky, Z., Iluz, D., Benichou, J., Netanyahu, N. (2020). Deep neural network recognition of shallow water corals in the Gulf of Eilat (Aqaba). Scientific Reports, 10 : 12959.
[4] A Chat with Andrew on MLOps: From Model-centric to Data-centric AI. https://www.youtube.com/watch?v=06-AZXmwHjo