Research VirtuaLearn3D

Virtual Laser Scanning for Machine Learning Algorithms in Geographic 3D Point Cloud Analysis

NEWS

Stay up-to-date by following VirtuaLearn3D via our GIScience News Blog and on X/Twitter and LinkedIn: #VirtuaLearn3D #3DGeo.

New Publication
Esmorís, A.M., Weiser, H., Winiwarter, L., Cabaleiro, J.C. & Höfle, B. (2024): Deep learning with simulated laser scanning data for 3D point cloud classification. ISPRS Journal of Photogrammetry and Remote Sensing. Vol. 215, pp. 192-213.

Click here!

Find out how we use models trained on virtual laser scanning data to classify real point clouds, achieving state-of-the-art results even without real training data involved.

OBJECTIVE

VIRTUAL LASER SCANNING - OVERCOMING BARRIERS

GENERAL INFORMATION & AIMS

Topographic laser scanning (LS) is a leading remote sensing technique to derive detailed 3D point cloud representations of the Earth’s surface and its objects. Virtual laser scanning (VLS) simulations recreate real-world scenarios of LS acquisitions in a computer environment. VLS is useful when real experiments are not feasible, e.g. due to technical, economic and logistic constraints.
Recent advances in machine learning, in particular supervised deep learning, indicate a huge potential to improve geographic 3D point cloud analysis of complex natural objects (e.g. vegetation) and scenes (e.g. geomorphological settings). The success of deep learning algorithms strongly depends on the availability of high-quality and appropriately large amounts of training data. The main aim of this project is to advance the concept of virtual laser scanning to tackle the lack of training data to enable powerful machine learning algorithms for geographic point cloud analysis.

HELIOS++

VirtuaLearn3D

Principle of VirtuaLearn3D

WORKFLOW

UAV-borne LS & classification

FOCUS

This proposed methodological step will push large-scale usage of VLS simulations for machine learning and opens up completely new fields of applications. Focus of this project will be on airborne laser scanning (incl. UAV-borne LS) and the tasks of object-based tree species classification and semantic urban scene classification, though the relevance of the developed generic concepts is not limited by the investigated examples.

CENTRAL TARGETS

Optimal Real-Simulated LS Data Fusion for Unsupervised Training: Find effective combinations of real LS data with theoretically unlimited amounts of simulated LS data for supervised training. Effective solutions can close the reality gap from simulated to real data and keep high classification accuracy while reducing costly input data.
Enhancing Transfer Learning with VLS-Generated Data: Find out to what degree VLS data generation can support transfer learning strategies to enable the usage of pre-trained models for transfer to different geographic characteristics and types of LS data. VLS-supported transfer learning is highly demanded due to drastically increasing availability of LiDAR technology in sciences and also on daily-life devices.
Development of Dynamic Objects in VLS simulations: Develop and test a new concept of ‘dynamic objects’ in VLS simulations, which enables e.g. to include vegetation with phenological changes and also moving objects (e.g. plants, cars, people).

Therefore: VirtuaLearn3D will push large-scale usage of VLS simulations for machine learning - in particular deep learning approaches - and our results will open up new fields of applications in science and industry.

RELATED PROJECTS

HELIOS++ meets forestry

SYSSIFOSS

Within the framework of SYSSIFOSS we developed a new approach to create synthetic LiDAR data by combining the outputs of an established forest growth simulator with a to-be-created database of species-specific model trees extracted from real LiDAR point clouds. This approach will result in inventory information at the single tree level and a matching 3D forest structure for large areas. The 3D forest structure will serve as input to HELIOS++ to conduct a sensitivity analysis and to examine the potential of the created synthetic data for the minimization of field-collected reference data.

HELIOS++

Heidelberg LiDAR Operations Simulator

FUNDING

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)
Project number: 496418931
Duration: 2022 - 2025

RELATED PUBLICATIONS

Table

Esmorís, A.M., Weiser, H., Winiwarter, L., Cabaleiro, J.C. & Höfle, B. (2024): Deep learning with simulated laser scanning data for 3D point cloud classification. ISPRS Journal of Photogrammetry and Remote Sensing. Vol. 215, pp. 192-213. DOI: 10.1016/j.isprsjprs.2024.06.018.
Schäfer, J., Winiwarter, L., Weiser, H., Höfle, B., Schmidtlein, S., Novotný, J., Krok, G., Stereńczak, K., Hollaus, M. & Fassnacht, F.E. (2024): CNN-based transfer learning for forest aboveground biomass prediction from ALS point cloud tomography. European Journal of Remote Sensing, 57(1), pp. 2396932. DOI: 10.1080/22797254.2024.2396932.
Fernández-Arango, D., Varela-García, F.-A. & Esmorís, A.M. (2024): Methodology for Identifying Optimal Pedestrian Paths in an Urban Environment: A Case Study of a School Environment in A Coruña, Spain. Smart Cities. Vol. 7(3), pp. 1441-1461. DOI: 10.3390/smartcities7030060 .
Weiser, H., Esmorís, A.M. & Höfle, B. (2024): How Tree Movement Influences Tree Metrics Derived from Laser Scanning Point Clouds.EGU General Assembly 2024. Vol. EGU24 pp. 1-2.DOI: 10.5194/egusphere-egu24-1633.
Höfle, B., Tabernig, R., Zahs, V., Esmorís, A.M., Winiwarter, L. & Weiser, H. (2024): Machine-learning based 3D point cloud classification and multitemporal change analysis with simulated laser scanning data using open source scientific software.EGU General Assembly 2024. Vol. EGU24, pp.1-2. DOI: 10.5194/egusphere-egu24-1261.
Zahs, V., Anders, K., Kohns, J., Stark, A. & Höfle, B. (2023): Classification of structural building damage grades from multi-temporal photogrammetric point clouds using a machine learning model trained on virtual laser scanning data. International Journal of Applied Earth Observation and Geoinformation. Vol. 122, pp. 103406. DOI: 10.1016/j.jag.2023.103406.
Zahs, V., Anders, K., Kohns, J., Stark, A. & Höfle, B. (2023): Classification of structural building damage grades from multi-temporal photogrammetric point clouds using a machine learning model trained on virtual laser scanning data [Data and Source Code]. heiDATA. DOI: 10.11588/data/D3WZID.
Winiwarter, L., Esmorís Pena, A., Weiser, H., Anders, K., Martínez Sanchez, J., Searle, M., Höfle, B. (2022): Virtual laser scanning with HELIOS++: A novel take on ray tracing-based simulation of topographic full-waveform 3D laser scanning. Remote Sensing of Environment. Vol. 269, pp. 112772. DOI: 10.1016/j.rse.2021.112772.
Winiwarter, L., Esmorís Pena, A. M., Zahs, V., Weiser, H., Searle, M., Anders, K., and Höfle, B. (2022): Virtual Laser Scanning using HELIOS++ - Applications in Machine Learning and Forestry. EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022. DOI: 10.5194/egusphere-egu22-8671.
Weiser, H., Winiwarter, L., Anders, K., Fassnacht, F.E. & Höfle, B. (2021): Opaque voxel-based tree models for virtual laser scanning in forestry applications. Remote Sensing of Environment. Vol. 265, pp. 112641. DOI: 10.1016/j.rse.2021.112641.
Winiwarter, L., Mandlburger, G., Schmohl, S. & Pfeifer, N. (2019): Classification of ALS Point Clouds Using End-to-End Deep Learning. PFG 2019/(87), pp. 75-90. DOI: 10.1007/s41064-019-00073-0.
Hämmerle, M., Lukač, N., Chen, K.-C., Koma, Zs., Wang, C.-K., Anders, K., & Höfle, B. (2017): Simulating Various Terrestrial and UAV LiDAR Scanning Configurations for Understory Forest Structure Modelling. In:ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci., IV-2/W4, pp. 59-65. DOI: 10.5194/isprs-annals-IV-2-W4-59-2017.
Bechtold, S., Hämmerle, M. & Höfle, B. (2016): Simulated full-waveform laser scanning of outcrops for development of point cloud analysis algorithms and survey planning: An application for the HELIOS lidar simulation framework. In:Proceedings of the 2nd Virtual Geoscience Conference/Bergen, Norway, 21-23 September 2016 pp. 57-58. DOI: 10.5194/isprs-annals-IV-2-W4-59-2017.