Technologies and systems for automation of maintenance tasks and equipment management are used in several cases of different business fields like security or food-processing industry for instance. The technologies used have reached a maturation stage allowing their use in real case, according to practical application specificity. However, it remains uncertainties about developing an approach allowing to treat several different uses-cases (required to automatize the IQ validation process) because we do not know so much work on this topic. So we realized a first phase analysis from images (or images-types) we have, to reduce the tracks field to follow.
Context
The goal of this project is to automatize a part of the actions for equipment maintenance. These actions are mainly inventory and temporal monitoring of equipment, to guarantee their functioning and prevent their deterioration. Some of them are repetitive and require staff on the field to collect information, for equipment inventory for instance. It would be possible to use sensor systems to collect these information and dedicate human work to their interpretation and decision making.
In this project, we focus on maintenance intervention queries (IQ) on Carl Source. They are written by clients who noticed a problem in a certain place, like a water leak in the bathroom, an alarm on the control screen, an equipment which isn’t working, etc. or to express a certain need like equipment improvement, special maintenance (deratization, oil change, etc.).
The current process for clients to request an intervention is to write it of the maintenance software of the site, which will validate it (or not) and make it done by a technician. Some work have been done in the DRIT to automatically classified written IQ. However, in this project we’re studying the possibilities to automatized IQ from a simple picture. Concrete cases to use image analysis for IQ automation would be: a crack in a wall, corrosion on a pipe, hanging wire, etc.
This article will report the existing work about image processing, on the concrete cases mentioned and globally think about what could be concretely consider.
DRIT’s existing work
Several works have been done at Berger-Levrault to evaluate potential approaches using deep learning for equipment/objects management and maintenance. Their objectives are to automatize the inventory phase on some kind of objects or equipment, from data images:
- Thesis work of Jérôme Pasquet about automatic detection of gravestone from aerial imagery of cemetery,
- Thesis work of Lionel Pibre about urban tree detection from aerial imagery multi-sensor,
- Collaborative work on manhole detection from aerial imagery,
- Thesis work of Younes Zegaoui about urban object detection from land-based LiDAR cloud of points,
- Transposition of these work on 3D LiDAR interior scene to automatize equipment inventory inside the rooms and evaluate the security norm satisfaction (fire extinguisher, fire alarm),
- 3D imagery visualization of building and network by Mehdi Ayadi and Clément Colin.
General scientific and technical state of the art
Image processing is a discipline of computer science and applied mathematics which study digital images and their transformation, to improve their quality or extract information. Pattern recognition and more broadly computer vision is one of the most important task in this large research field.
Historically, it’s David Marr who formalized the first precise contours line detection on an image in 1980. In 1990’s resolution technologies have improved, especially thanks to the wavelet method. From 2000’s, important innovation have been made in specific use-case by limiting the general problem, like optical character recognition (OCR) or facial recognition.
Then, object recognition have made great progress with machine-learning techniques. Those techniques propose to represent the images as multi-dimensional pattern vectors call features vector. A learning method is then used to deduce useful information (like semantic content on an image) from descriptor.
From 2012, the rise of Convolutional Neural Network (CNN) using Deep Learning allowed important progress in several classic task of image processing like:
- Image classification, associating a class to an entering image,
- Semantic segmentation, associating a class to each pixel of the image,
- Object detection, adjust a wrapper box around each object in the image,
- Instance segmentation, precisely cropping each object in the image.
Use-case for maintenance
We distinguished several uses-cases, which although very different from each other, intersect with the issue of automated maintenance:
- Corrosion/rust: precise corrosion location on metallic surface, either from raw materials pictures, or on pipes, manufactured pieces pictures.
- Crack: precise location of crack on building facade pictures or traffic video streams.
- Windows: windows detection and characterization (good state or broken) from building facade pictures or photos taken from inside.
Deep Learning methods
CNN, allowing a revolution of processing image field, are based on a successive sequence of simple operation (called layer) which put end to end allows the calculation of a very complex non-linear function.
The main CNN operation is the convolution layer. It enables the extraction on interesting characteristics to be optimized together with the resolution of a specific problem such as image classification, where the use of descriptors only enable to optimize the second.
In the most important CNN architecture, we can cite:
AlexNet, first “working” CNN, consisting in a three layer convolution and a classifier;
UNet and SegNet, which enable the semantic segmentation task with the introduction of deconvolution layers allowing to get a “U” structure simplifying the problem of class predicting for each pixel.
Object detection architecture in which we distinguish:
- Approach in two-stroke, such as FasterRCNN, enabling to predict robust wrapping boxes, but require more calculation time .
- Single shot approach, such as YOLO which is faster but less robust.
Existing approach for studied uses-cases
All studied uses-cases are based on the same approaches from a technical point of view. We can cite thresholds, contour detectors and morphology operations.
More recent works relies more on CNN to complete various tasks related to our uses-cases, such as classification and objects detection.
Corrosion case
We found numerous studies, programs and data sets which seem to work adequately (we note that these studies are realized on flagrant corrosion or on large corrosion zone, which makes difficult to evaluate the methods performances). They use either colorimetry and texture concept using classic analyze techniques, or CNN.
We found on Github, some existing programs (Corrosion Detector for instance) and numerous available data set available on internet for images training.
Crack case
For this case, we observe a lot of work about crack on roads or bridges. Their diagnostics are based on the crack definition according to Chambon theory: “Crack: is a set of pixels with a darker gray level than the background of the image. In addition, a crack can be seen as a set of small segments with different orientation connected together”. Though we’ll note again than the images show extremely sharp cracks.
A lot of publications show than the results obtained using classical images processing techniques enable to get good results.
We even tried some simple recognition models thanks to the various contours detection algorithm offer by Open-Cv with satisfying results. Even if these methods show great asset as they don’t need specific learning, they seem only adapted to plain surface like road or concrete facade.
Windows case
This last case seem the most delicate, because image processing techniques seem to be much less efficient, especially if the photos are taken from a certain distance. We immediately note that transparent glass walls recognition is much more difficult.
Some applications exist to analyze broken mobile screen, though images are close-up pictures, so detection seems easier. We found some elements enabling to recognize windows on a facade, but they are based on their contours, using a corner detection algorithm. We also found an application to recognize mirrors and glasses wall which seems to work.
Technological solutions
Automatic inventory / Logistic
Several solutions are commercialized to automatize inventory tasks or package processing in warehouse. They are based on restrictive logistics: equipment must be store in a precise storage, each spot must have its own QR code or barcode, the access way must be cleared for the drones (aerial or terrestrial) etc. The image processing task is then much less complicated as only the QR code/barcode must be detected.
Industrial vision
The common industrial approach is to take a close-up picture of each product, to then guide the manufacturing process, most part of the time on quality controls.
Here we found a rising use of CNN to classify, indeed, as the picture process is completely controlled. Big data base can be feed to train CNN to recognize the different manufacturing imperfections or the quality level of the product.
In the other hand, we observe an important rise of image processing in industrial automation process: control of production line, package reception, automatize process of bills, renting car return etc.
Counting and security solution
Numerous companies are calling for object recognition solution, for commercial reasons or for security needs. We observe a rise of security camera installation using IA alerts systems.
Recognition and drone study
Several companies launched image processing with drones which allow to collect precious information from far distance. These solutions benefits is their ability to pre-process the images and generate real-time alerts.
To conclude
Nowadays, we observe the possibilities to execute various work with image processing techniques. Our three studied uses-cases show crack, corrosion and broken window detection doable. Others scenarios seem possible like hanging wire or ripped-off socket.
We believe than the main concern of this project is about the entrance picture, its quality and and the ability to recognize the involved equipment.
