Projects / Programmes
Detection of inconsistencies in complex visual data using deep learning
| Code |
Science |
Field |
Subfield |
| 2.07.00 |
Engineering sciences and technologies |
Computer science and informatics |
|
| Code |
Science |
Field |
| P176 |
Natural sciences and mathematics |
Artificial intelligence |
| Code |
Science |
Field |
| 1.02 |
Natural Sciences |
Computer and information sciences |
Deep learning, computer vision, supervised learning, semi-supervised learning, unsupervised learning, visual learning, segmentation, anomaly detection
Researchers (14)
| no. |
Code |
Name and surname |
Research area |
Role |
Period |
No. of publicationsNo. of publications |
| 1. |
50367 |
Borja Bovcon |
Computer science and informatics |
Researcher |
2018 - 2019 |
19 |
| 2. |
25402 |
PhD Jože Guna |
Communications technology |
Researcher |
2019 |
238 |
| 3. |
30155 |
PhD Matej Kristan |
Computer science and informatics |
Researcher |
2018 - 2022 |
323 |
| 4. |
31985 |
PhD Janez Križaj |
Systems and cybernetics |
Researcher |
2018 - 2022 |
39 |
| 5. |
05896 |
PhD Aleš Leonardis |
Computer science and informatics |
Researcher |
2018 - 2022 |
455 |
| 6. |
50843 |
Jon Natanael Muhovič |
Computer science and informatics |
Researcher |
2018 - 2022 |
23 |
| 7. |
21310 |
PhD Janez Perš |
Systems and cybernetics |
Researcher |
2018 - 2022 |
238 |
| 8. |
39056 |
Anže Pratnemer |
|
Technical associate |
2019 |
0 |
| 9. |
18198 |
PhD Danijel Skočaj |
Computer science and informatics |
Head |
2018 - 2022 |
309 |
| 10. |
28458 |
PhD Vitomir Štruc |
Systems and cybernetics |
Researcher |
2018 - 2022 |
361 |
| 11. |
34398 |
PhD Domen Tabernik |
Computer science and informatics |
Researcher |
2018 - 2022 |
50 |
| 12. |
18185 |
PhD Andrej Trost |
Electronic components and technologies |
Researcher |
2020 |
328 |
| 13. |
53924 |
Vitjan Zavrtanik |
Computer science and informatics |
Researcher |
2019 - 2020 |
14 |
| 14. |
19237 |
MSc Rok Žurbi |
Telecommunications |
Researcher |
2019 |
20 |
Organisations (2)
Abstract
Huge volumes of data are constantly being collected and a lot of this data is available at no additional cost. However, making use of this data poses a huge challenge. Due to enormous quantities of data it is clear that manual processing is prohibitive and calls for automated procedures. Computer vision, driven with machine learning methods, is a scientific discipline that addresses automatic extraction of information from raw visual data. In the last couple of years we have witnessed a great increase in the performance of computer vision methods, mainly due to the success of deep learning approaches, that are well suited for extracting information from large quantities of data. However, most of the deep learning approaches rely on labelled data; they therefore still require a significant human effort for labelling the required amounts of data, which is very costly, tedious and sometimes error-prone or even impossible.
In this project we will address this issue for a particular computer vision task of anomaly detection in images. We define an anomaly as a part or parts of an image that significantly deviate from a regular appearance observed over a large number of images. As main problem domains we selected three different tasks: detection of surface anomalies in industrial visual inspection, detection of changes in satellite images, and detection of unusual scenes in surveillance images. Most of the data in each of these three problem domains is quite consistent. The objective of the proposed project is to develop novel deep learning methods for modelling complex consistency and detecting inconsistencies in visual data using training images annotated with different levels of accuracy.
We will therefore go beyond the traditional supervised learning, where all anomalies on all training images have to be adequately labelled. We will relax this requirement in several ways. First, we will allow coarser labelling of anomalous regions and develop methods for weakly-supervised discriminative learning to solve this problem. We will then go further and allow even weaker annotations, on the level of images, without any region annotations, and consider the case where only a subset of training images is labelled. We will develop corresponding methods for semi-supervised learning based on Multiple instance learning and self-supervised surrogate objective methods. The final goal will be to train the model only on anomaly-free training samples using deep generative modelling. We will base our work on methods such as variational autoencoders and generative adversarial networks by replacing classic deep networks with the architectures that have explicit compositional structure. Since such fully-unsupervised learning process may lead to detection of data inconsistencies that are meaningless from the predefined task point of view, we will also consider the cognitive relevance of the learned models and will adapt the learning process accordingly. The goal of this project is to develop general methods for anomaly detection in images that require minimal amount of labelled data. We will validate the developed methods in three related but different problem domains; visual inspection, remote sensing, and visual surveillance. The corresponding image datasets will be acquired and used to assess the performance and the limits of the developed methods.
We will develop several concepts related to the fundamental understanding of deep learning, which we will apply to the problem of anomaly detection, where the primary impact is to be expected. However, these concepts could be applied in deep learning approaches in general, thus applicable on various computer vision problems. On the other hand, we will also apply the developed approaches on three specific problem domains, therefore a significant impact can be expected also in more narrower research subfields, such as machine vision and remote sensing and related disciplines.
Significance for science
The project addresses research issues important to several research areas. Our primary goal is to go beyond the supervised deep learning for the specific case of automated detection of anomalies in visual data. We therefore expect to go beyond the state-of-the-art in this specific research area. However, to achieve this goal, we will develop several concepts related to the fundamental understanding of deep learning, which could be applicable in deep learning approaches in general, thus applicable on various computer vision problems.
More specifically, our work will primarily focus on the development of basic methods for semi- and unsupervised learning. We will address adversarial learning of generative deep networks and show novel methods that incorporate compositional properties in deep models. Using the properties of compositional hierarchies, we will improve the understanding of deep networks. Although the computer vision and deep learning community has started investigating approaches that do not require a huge number of labelled training data, the research field still predominantly relies on supervised learning, so we expect that the results of the proposed project will have a significant impact to the future development in this research area.
We will validate the proposed methods on three problem domains, namely visual inspection, remote sensing, and visual surveillance. The proposed approaches will therefore have a direct impact in each of these three subfields of computer vision. Moreover, through interdisciplinary research, the results could also achieve a significant impact in other research areas. Our methods for anomaly detection in satellite images will be directly applicable to the problems related to the environment monitoring. In particular, monitoring of agricultural fields, forests and sea surfaces through satellite images can have significant impact on advancements in the related research fields. Similarly, with novel methods for detection of anomalies on surfaces of industrial products, we expect to enter the rather conservative field of machine vision, introducing the deep learning methods into machine vision systems for automated visual inspection even where not a lot of training samples are available. There will be therefore numerous possibilities to transfer our findings into practice.
Besides the research endeavours we also plan to contribute our share in community building. During the project we will build image datasets for evaluation and benchmarking developed methods for visual anomaly detection. In particular, as today, there is almost no realistic dataset for detection of anomalies on object surfaces, which makes the comparison of developed approaches difficult. We expect that public release of our datasets will facilitate further research in this field on part of other, international, research groups. It will make the comparison of developed approaches possible, which will set a solid basis for further research and community building.
Significance for the country
The project addresses research issues important to several research areas. Our primary goal is to go beyond the supervised deep learning for the specific case of automated detection of anomalies in visual data. We therefore expect to go beyond the state-of-the-art in this specific research area. However, to achieve this goal, we will develop several concepts related to the fundamental understanding of deep learning, which could be applicable in deep learning approaches in general, thus applicable on various computer vision problems.
More specifically, our work will primarily focus on the development of basic methods for semi- and unsupervised learning. We will address adversarial learning of generative deep networks and show novel methods that incorporate compositional properties in deep models. Using the properties of compositional hierarchies, we will improve the understanding of deep networks. Although the computer vision and deep learning community has started investigating approaches that do not require a huge number of labelled training data, the research field still predominantly relies on supervised learning, so we expect that the results of the proposed project will have a significant impact to the future development in this research area.
We will validate the proposed methods on three problem domains, namely visual inspection, remote sensing, and visual surveillance. The proposed approaches will therefore have a direct impact in each of these three subfields of computer vision. Moreover, through interdisciplinary research, the results could also achieve a significant impact in other research areas. Our methods for anomaly detection in satellite images will be directly applicable to the problems related to the environment monitoring. In particular, monitoring of agricultural fields, forests and sea surfaces through satellite images can have significant impact on advancements in the related research fields. Similarly, with novel methods for detection of anomalies on surfaces of industrial products, we expect to enter the rather conservative field of machine vision, introducing the deep learning methods into machine vision systems for automated visual inspection even where not a lot of training samples are available. There will be therefore numerous possibilities to transfer our findings into practice.
Besides the research endeavours we also plan to contribute our share in community building. During the project we will build image datasets for evaluation and benchmarking developed methods for visual anomaly detection. In particular, as today, there is almost no realistic dataset for detection of anomalies on object surfaces, which makes the comparison of developed approaches difficult. We expect that public release of our datasets will facilitate further research in this field on part of other, international, research groups. It will make the comparison of developed approaches possible, which will set a solid basis for further research and community building.
Most important scientific results
Interim report
Most important socioeconomically and culturally relevant results
