WebODM

From OpenStreetMap Wiki
Jump to navigation Jump to search

Description

WebODM is and easy to use interface for OpenDroneMap and software for drone image processing.[1]

At the moment of this writing it supports ODM and MicMac engines, although the latter is still experimental.


Before anything else, there's an official installation page available on Github with lots of information: https://github.com/OpenDroneMap/WebODM#getting-started

Another installation step by step approach is outlined here: https://docs.opendronemap.org/installation

Both have been sources for this documentation, this documentation has some different information for complete newbies, focused on additional information especially on the Linux-deployment for absolute novice users.

Installation and setup

WebODM on VirtualBox for Windows 11 using docker

The following is a step by step tutorial how to set up WebODM using Oracle VM VirtualBox on a Windows OS.

To do this straight under Linux, skip the Windows and VirtualBox parts.

The tutorial is based on the Tutorial by TJ Corbett[2] with slight alterations for the versions as mentioned below (up-to-date at the moment of this writing)

Docker simplifies operations, but does slow down a bit the process

Software used for deployment, installation and setup

Version Download link Version used for this tutorial Release date
Windows Windows 11 2023 Update | version 23H2 build 22631 26 Sep 2023
Oracle VM VirtualBox VirtualBox 7.0.10 r158379 18 Jul 2023
Ubuntu Desktop Ubuntu 22.04.3 LTS 10 Aug 2023
WebODM WebODM 2.2.0 5 Oct 2023

Specifications of hardware

Processing drone imagery takes a lot of resources! It most likely works with minimal hard specifications but I recommend at least 8 core and 32 GB RAM, so you can allocate at least 4 cores and 24 GB for the virtual system that does the actual processing.

Note here WebODM supports the use of multiple computers on a network or even cloud based processing power.

Installation and setup for Ubuntu

  1. First download the VirtualBox and install
  2. Now download the Ubuntu ISO file
  3. Fire up Virtualbox and click on the blue "New" button for a new virtual machine
    1. ISO Image: the downloaded Ubuntu Image
    2. Type: Linux
    3. Version: Ubuntu 22.04 LTS (Jammy Jellyfish) (64-bit), this may be greyed out and default to Ubuntu (64-bit) depending on your settings, which is totally fine
    4. Skip Unattended Installation - make sure to select this, otherwise you may end up with the common bug the terminal won't be working (Probably going to get fixed in future releases)
    5. All other settings at your choice
    6. Click Next
    7. Hardware - my advise:"max out the recommended settings," but feel free to experiment with it. These settings can be changed later anyways.
    8. Click Next
    9. Virtual Hard disk - 25GB + how much you're going to need for your drone images + stitching result.
    10. Click Next
    11. Click Finish
  4. Now click on the green "Start" button to fire up your installation of Ubuntu on a virtual machine and simply follow all the instructions and answer the questions. I leave here the choices to the user what kind of installation you like to have and the choices upon installation you make, but for WebODM I'd have one recommendation:
    1. Minimal Installation instead of Normal
  5. After the installation is completed I recommend to turn off the virtual machine and first go to the "Settings" for your virtual Machine
    1. Go to the tab "Network"
    2. Change the Adapter 1 from "Attached to: NAT" into "Bridged Adapter." This will ensure that you can access the WebODM interface and it gets the IP from your DHCP where you can give it static lease so it's easy to bookmark.
Optional installation and setup for Guest Additions

At this point, although not essential but very handy, I'd recommend the Guest Additions. A simple tutorial how to install it can be read here but in short:

  1. Start your Virtual Machine running Ubuntu
  2. In the top bar go to "Devices" and select "Insert Guest Additions CD Imagine"
  3. It will mount automatically in the background
  4. On the left side you'll see a CD icon, click on it
  5. You'll see a file named "autorun.sh," right-click on it and select "Run as a Program" (it'll ask your password for executing the command)


After the installation has completed you can:

  • Drag the window and resize it as you please
  • at the top under "Devices" you can enable
    • Shared Folders
    • Shared Clipboard
    • Drag and Drop

This allows seamless interaction between your Ubuntu VM Client OS and Windows Host OS

Installation and setup for WebODM

At this point we assume you got your Ubuntu up and running properly on your VirtualBox and continue with the commands to install WebODM.

Note - do NOT mix different docker installations!

Start the Terminal and proceed with the following commands (answer "y" for any questions to update/upgrade):

  1. sudo apt update
  2. sudo apt upgrade
    1. Before you continue to the next step, make sure to restart if updates were found
  3. sudo apt install curl
  4. curl -fsSL https://get.docker.com -o get-docker.sh
  5. sh get-docker.sh
  6. sudo apt install -y git python2 python3-pip
  7. sudo pip install docker compose

Now check if your Docker is working by entering the following command

  1. docker compose version or older versions docker compose --version
  2. You'll see something like docker-compose version v2.21.0


Next we download WebODM and get it up and running.

  1. git clone https://github.com/OpenDroneMap/WebODM --config core.autocrlf=input --depth 1
  2. cd webodm
  3. sudo ./webodm.sh start (for a Virtual Machine that's exposed to the internet you should definitely not run under sudo but set the proper privileges for the folders. Use a search engine to find how to do that)

After this completes you should see something like

Congratulations! 
webapp      | ==========================
webapp      | 
webapp      | If there are no errors, WebODM should be up and running!

Note: at this point you can start the VM in "headless" mode as the WebODM will automatically start services.

Using WebODM

At this point seeing the Congratulations message you can login to your WebODM. Since you're using the bridged adapter your virtual system is retrieving it's IP from your DHCP so you can simply look up the IP from your DHCP server. Give it a static lease if you like.

To access it at this point with the default installation you simply go to your preferred browser and connect to your instance using the http (without s) and port 8000 for instance: http://192.168.1.20:8000

You can read on the WebODM manual at Github how to enable all the functionality you may need like SSL, additional processing nodes, CUDA support and customizing[1].

An issue for RAM, you may have to tweak a bit your sysctl.conf file

add 'vm.overcommit_memory = 1' to /etc/sysctl.conf 
run the command 'sysctl vm.overcommit_memory=1'

Using your own presets[3]

You may at this point wish to avoid the default presets being reset every time you restart/reboot the docker/WebODM or server/computer. In this case please proceed as follow

Go to your /app folder inside your webodm docker, see below if you have questions how to get inside the docker

Open the file boot.py , you may need to apt-get update & install nano

Comment out the line(s) add_default_presets()

Now the presets won't come back. Please keep in mind if you update WebODM, this file will be default again.

Clean or Increase Storage[4]

Before you do any of the below mentioned choices you have, keep in mind if you delete projects there's a 48h flush regime. So if you delete objects and there's temporary files left from crashes filling your disk, within 48hours all of these will be flushed and free up your storage. So if you keep your storage big enough and delete projects in a timely manner where you don't need them anymore, any left-overs from crashes will also be flushed so you'll never really run out of disk space.

Stop and Restart WebODM

keep in mind below is an example for my installation and location of files, it may be different on your system of course

sudo ./webodm/webodm.sh stop
sudo ./webodm/webodm.sh start
sudo ./webodm/webodm.sh down                       stop and remove docker containers
sudo ./webodm/webodm.sh update                     stop and update to latest release
sudo ./webodm/webodm.sh liveupdate                 update to latest release without stopping
sudo ./webodm/webodm.sh rebuild                    rebuild all containers and cleanups
sudo ./webodm/webodm.sh checkenv                   check on missing components or other issues
sudo ./webodm/webodm.sh resetadminpassword

If you run into issues like rights or illegal options make sure the sh is executable[5]

sudo chmod +x ./webodm/webodm.sh

Updating and rebuilding the distribution is a valid option as well if you run into problems[6]

sudo ./webodm/webodm.sh update
sudo ./webodm/webodm.sh rebuild

Please keep in mind rebuilding wipes clean your existing projects, DATA LOSS!

Prune[7]

For the following you have to stop webODM using the stop command.

$ sudo docker system prune -a --volumes

WARNING! This will remove:

  - all stopped containers

  - all networks not used by at least one container

  - all anonymous volumes not used by at least one container

  - all images without at least one container associated to them

  - all build cache

If prune doesn't help you can try the following commands.

Remove Volumes [8]

$ docker volume rm $(docker volume ls -f dangling=true -q)

// or the following older solution if the first doesn't work

$ docker volume rm $(docker volume ls -qf dangling=true)
$ docker volume ls -qf dangling=true | xargs -r docker volume rm

Remove Docker Images[9]

$ docker images
$ docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

$ docker images | grep "none"
$ docker rmi $(docker images | grep "none" | awk '/ / { print $3 }')

Remove Docker Containers[10]

$ docker ps
$ docker ps -a
$ docker rm $(docker ps -qa --no-trunc --filter "status=exited")

Resize Docker Disk Space

$ docker-machine create --driver virtualbox --virtualbox-disk-size "200000" default

Increase VMs' Storage

If cleaning doesn't help you and you simply have no other choice but to increase your VM's storage here is the step by step for Oracle VM VirtualBox Manager. There's basically 2 steps you need to complete here.

  1. Increase the VM storage
  2. Increase the Partition on that storage

Step 1

  • Start your Oracle VM VirtualBox Manager
  • Go in the upper left to File -> Tools -> Virtual Media Manager or press Ctrl+D
  • Select here the vdi VM file you wish to increase
  • At the bottom move the slider or enter the value you wish for and click apply

Step 2

  • Download GParted live iso from here
  • In the VM Manager select your VM deployment
  • On the right you see the yellowish "settings" icon, click on it
  • Go to storage
  • When you select "Controller: IDE" you see a disk with a plus, click on it
  • Click on add and select the downloaded "gparted-live.........iso" file
  • At the bottom select "choose" and make sure once back in the Storage panel there's only 1 drive under the "Controller: IDE", otherwise you may be booting the wrong one
  • Now go to "System" on the left and make sure the "Boot Order" is set to optical prior to Hard Disk
  • Click "OK" and then click on the green "Start" icon to boot GParted
  • If you're an English language user just continue clicking enter on every question
  • Inside GParted , right-click the partition with the "ext4" File System and select "Resize/Move"
  • Now drag the partition all the way to the left to increase it's size maxed out and click at the lower right the button Resize/Move
  • Click on the green "check" button at the top somewhat in the center of the screen and select "Apply"
  • If errors appear, usually there's the button "fix" in GParted which is safe to use under normal circumstances. Do that and repeat the increase size steps.
  • Once done, Quit GParted or even just shutdown the VM
  • Now you can start your VM with an increased storage

General help with dockers

Command in linux Description 
sudo docker container ls
This will list the existing containers, the 'opendronemap/webodm_webapp' is the image holding the files

Results location: /webodm/app/media/project/<NR>/task/<TASK_ID>/assets (replace what's between <> with the proper number obviously 

docker exec -it <CONTAINER_ID> bash
This allows you to access the file system of the docker, you get the CONTAINER_ID from the above mentioned 'docker container ls' command 
./webodm.sh restart --media-dir /path/to/webodm_results
This one stores the results to a directory of your choosing.[11] 
./webodm.sh restart --default-nodes 4
parallel processing of up to 4 tasks.[12] 
./webodm.sh restart --media-dir /home/user/webodm_data --db-dir /home/user/webodm_db
You can use this command to start webodm using the host file system instead of inside the docker itself.[13]

Before doing this, please read about risks, benefits, downsides of either solution to use volumes inside docker or using the mount of the host filesystem functionality[14] 

sudo ./WebODM/webodm.sh restart	--port 8081 --default-nodes 6 --media-dir /var/snap/nextcloud/common/nextcloud/data/user/files/
This will have webodm listen to port 8081 while 6 nodes on the local machine are available and the media dir is set for the output 
sudo docker ps --size
Reports the use of the containers on your file system[15] 
sudo apt-get purge docker-engine
sudo apt-get autoremove --purge docker-engine
rm -rf /var/lib/docker
This will completely wipe the docker! Only do this if you wish to start with a clean slate[16]

WebODM on VirtualBox for Windows 11 install natively (non-Docker)

- using webodm_install.sh for native installation

TO BE CONTINUED - WORK IN PROGRESS - BEAR WITH ME

WebODM Analysis

This is work in progress, I can't find any source online where the step by step being done by WebODM is listed, so I'm going to add it here

I'll update this accordingly to present some data on which processing steps are taking place and how much hardware requirements to complete a process successfully

Step nr Process name Console Messages Description Single-threading Multi-threading? General stats Time per image Time per GB Planned RAM Scale-space per image
1 opensfm - detect_features INFO: Reading data for image

DEBUG: Found 24000 points DEBUG: Computing sift INFO: Extracting ROOT_SIFT features

no yes Images 1558

17.52 GB Line 112 to 7903

total time 5880 s

3.77 s 335.6 s 84 GB 458 MB at 200

Img max in queue 92 GB Storage total

2 opensfm - match features DEBUG: Matching .... Matcher: FLANN (symmetric) T-desc no yes Image pairs 14057

line 7907

total time 4559 s

2.926 s 260.2 s
3 opensfm - create_tracks INFO: reading features

DEBUG: Merging features onto tracks DEBUG: Good tracks

yes no 690 s 0.4429 s 39.4 s
4 opensfm - reconstruct INFO: Starting incremental reconstruction

INFO: Removed outliers

INFO: Shots and/or GCPs are well-conditioned

DEBUG: Ceres Solver Report

Termination: CONVERGENCE

[WARNING] Multiple reconstructions detected

[WARNING] Cannot merge shot id

[INFO]    Export reconstruction stats

line 21970

time total 17340 s

11.130 s 989.7s
5 opensfm - compute_statistics DEBUG: locator

DEBUG: Setting pcolormesh DEBUG: Setting pcolormesh

yes no line 26860

total time 100 s

0.064 s 5.7 s
6 opensfm - export_geocoords --reconstruction DEBUG: Undistorting image no yes line 26868

total time 2540

1.63 s 145 s
7 opensfm - export_visualsfm yes no just a few seconds
8 opensfm - export_openmvs Estimated depth-maps no yes total time 95065 61.017 s 5426.1 s
9 opensfm - export_openmvs Geometric-consistent estimated depth-maps (first run) no yes total time 22812 14.642 s 1302.1 s
10 opensfm - export_openmvs Geometric-consistent estimated depth-maps (second run) no yes total time 21890 14.05 s 1249.4 s
11 opensfm - export_openmvs Filtered depth-maps no yes total time 1455 0.934 s 83.0 s
12 opensfm - export_openmvs Fused depth-maps no yes total time 18927 12.148 s 1080.3 s RAM & SWAP ~500 GB
13 Densifying point-cloud total (openmvs process) no yes total time 169357 108.702 s 9666.50 s
14 scene_dense.mvs Point visibility checks
  1. <placeholder>
  2. <placeholder>
  3. Geometric-consistent estimated depth-maps


References

  1. ↑ 1.0 1.1 https://github.com/OpenDroneMap/WebODM
  2. ↑ https://community.opendronemap.org/t/using-webodm-with-virtualbox/9500
  3. ↑ https://community.opendronemap.org/t/defaults-automatically-come-back/19419
  4. ↑ https://gist.github.com/bastman/5b57ddb3c11942094f8d0a97d461b430
  5. ↑ https://stackoverflow.com/questions/54055549/linux-ubuntu-set-illegal-option-o-pipefail
  6. ↑ https://community.opendronemap.org/t/webodm-db-problem-stuck-on-restarting/16091
  7. ↑ https://docs.docker.com/engine/reference/commandline/system_prune/#examples
  8. ↑ https://github.com/chadoe/docker-cleanup-volumes
  9. ↑ http://stackoverflow.com/questions/32723111/how-to-remove-old-and-unused-docker-images
  10. ↑ http://stackoverflow.com/questions/32723111/how-to-remove-old-and-unused-docker-images
  11. ↑ https://docs.opendronemap.org/installation/#step-1-install-requirements
  12. ↑ https://community.opendronemap.org/t/changing-number-of-projects-running-concurrently-in-webodm/9049
  13. ↑ https://github.com/OpenDroneMap/WebODM/#where-are-my-files-stored
  14. ↑ https://docs.docker.com/storage/volumes/#backup-restore-or-migrate-data-volumes
  15. ↑ https://www.digitalocean.com/community/questions/how-to-check-the-disk-usage-of-all-running-docker-containers
  16. ↑ https://stackoverflow.com/questions/42264617/how-to-repair-docker-or-reinstall-it
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=WebODM&oldid=2740940"