Project completion at the Heinz Nixdorf Institute
So-called depth estimation serves the three-dimensional perception of scenes in photographs and plays an important role in various areas of application, such as robotics or autonomous driving. Since 2019, scientists at the Heinz Nixdorf Institute of Paderborn University have been collaborating with the L3S Research Department of Leibniz University Hannover (Visual Analytics working group, Professor Ralph Ewerth) on machine learning methods, which are intended to make this process more universal and more cost-effective. Two years into the project entitled “Weakly Supervised Learning of Depth Estimation in Monocular Images”, promoted by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) to the tune of close to EUR 175,000, the project group is now able to see results for various areas of application.
From the second to the third dimension
A conventional camera transforms the recording of the three-dimensional world into a two-dimensional photograph. However, photographing with several lenses or with specific sensors, which are capable of measuring the depth of the depicted objects, allows the third dimension to be reconstructed, thus generating a three-dimensional representation. During this process, a value measuring the distance to the camera lens is determined for every pixel. Such information for individual images is collected on a so-called depth map, the preparation of the latter for learning predictable models admittedly being not only time-consuming but also, and primarily, extremely cost-intensive, due to the use of more expensive and more energy-consuming sensors, such as LiDAR. Accordingly, a wider use in arbitrary scenarios is not yet practicable.
“In order to be able to make depth-estimation methods more practicable, in the cooperative project we concerned ourselves with approaches whereby weaker information suffices for training possible models allowing depth maps to be predicted. Sensors producing only approximate values or synthetic values generated artificially render the process significantly more cost-effective”, explains Julian Lienen, Research Associate of the former “Intelligent Systems and Machine Learning” expert group at the Heinz Nixdorf Institute of Paderborn University under the direction of Professor Eyke Hüllermeier. The generalisation of conventional machine learning methods and the construction of synthetic training data, generated by the scientists with the weak information within the scope of their research, reduce the requirements for the use of depth-estimation models. Thus, tolerance ranges or relative data on the individual pixels now suffice to correctly estimate the relevant depth and generate high-quality depth maps. The cost benefit generated by this means has allowed the expansion of the potential area of application of depth-estimation methods.
Course-setting measures for three-dimensional photographs via smartphones
According to Lienen, it is quite conceivable that even smartphones with cost-effective camera sensors could be standardly equipped with high-quality depth estimation in the future. The research results are also a gain for the field of augmented reality. “An expansion in the methods developed so far would allow them to be applied to video recordings as well. That in turn would allow the development of further cost-effective data sources that could increase the ability to generalise the models learnt. Particularly in the field of augmented reality, that would then allow computer-generated scenes to be optically adapted to real life in such a way that practically no distinction to reality would be discernible”, as Lienen points out in conclusion.
