Osmupdate

From OpenStreetMap Wiki
Jump to navigation Jump to search
Osmupdate - Other languages
Osmupdate
License: GNU Affero General Public License v3
Platforms: Windows and Linux
Version: 0.9 (2018-05-30)
Source code: https://gitlab.com/osm-c-tools/osmctools
Programming language: C

osmupdate downloads and cumulates OSM Changefiles of different categories (minutely, hourly, daily). This allows you to create new changefiles for every time period you need and to update your OSM data files in regular or irregular intervals, just as you like.

Note that this is not the only mechanism for automatic OSM data updates, you can also use other programs for this purpose, for example Osmosis. osmupdate is a small and fast program which has been developed just for this task. It might offer a few functions Osmosis does not offer, but there are also functions you might miss in osmupdate.

Use Cases

  • You want to keep your local PostgreSQL database up-to-date and minutely in sync with the servers of openstreetmap.org.
    • You can use osmupdate. osmupdate cannot update databases itself, but can do it in conjunction with other tools. For osm2pgsql database, see furry-sansa updater.
  • You need to update the OSM file of a small geographical region every other week.
    • Do not use osmupdate. Redownload the file from one of the OSM file services, e.g. geofabrik.de.
  • You just found an old OSM planet file from a year ago and want to update it.
    • Do not use osmupdate. Redownload the whole planet file.
  • You have an OSM data file of a larger geographical region and need to update it every few hours, every day, or every week.
    • It is recommended to use osmupdate for this purpose.

Download

Important Note: osmupdate fails to process minutely diffs 4425368 to 4431447 due to a temporary format change in the diff replication state.txt file. See this thread on OSM dev mailing list for discussion and ways to mitigate this issue.

These Downloads are available:

As usual: There is no warranty, to the extent permitted by law.

Prerequisites

osmupdate will call subprograms which must be available on your system:

  • osmconvert (also included in the osmctools Debian package)
  • wget
  • gzip (if you want to write gzip compressed files)

Please ensure that these programs have been downloaded and installed.

Program Description

This program cares about updating an .osm, .o5m or .pbf file. It will download and apply OSM Change files (.osc) from the servers of "planet.openstreetmap.org". It also can assemble a new .osc or .o5c file which can be used to update your OSM data file at a later time.

Two command line arguments are mandatory: the name of the old and the name of the new OSM data file. If the old data file does not have a file timestamp, you can specify this timestamp manually on the command line or the program itself analyzes it with the program osmconvert.

Instead of the second OSM file name, you alternatively may specify the name of a changefile (.osc or .o5c). In this case, you also may replace the name of the old OSM data file by a timestamp.

Command-line arguments which are not recognized by osmupdate will be passed to osmconvert. Use this opportunity to supply a bounding box or a bounding polygon if you are going to update a regional osm data file (.osm or .o5m).

To get a detailed description, please use the built-in help function of the program: 

./osmupdate --help

The following chapters demonstrate some usage examples.

Updating OSM Files

Out-of-date OSM data files can be updated and written as new files (.o5m, .o5m.gz, .osm, .osm.gz or .pbf). For safety reasons osmupdate will not delete the old file. If you do not need it as backup file, please delete it by yourself. Examples: 

./osmupdate old_file.o5m    new_file.o5m
./osmupdate old_file.o5m.gz new_file.o5m.gz
./osmupdate old_file.osm    new_file.osm
./osmupdate old_file.osm.gz new_file.osm.gz
./osmupdate old_file.pbf    new_file.pbf

If your old OSM data file does not contain a file timestamp, this timestamp can be specified manually 

./osmupdate old_file.o5m 2012-03-15T23:30:00Z new_file.o5m

otherwise the program tries to extract the timestamp from the old file automatically.

Note: It is not recommended to update large .osm formatted files as this may take hours or even days. Please use .pbf or .o5m (a bit faster) for this purpose.

Assembling an OSM Change file

Here, the old OSM data file is not updated directly. An OSM changefile is written instead. This changefile can be used to update the OSM data file afterwards. You also can use the changefile to update a database with Osmosis or osm2pgsql. Examples for assembling changefiles: 

./osmupdate old_file.o5m change_file.o5c
./osmupdate old_file.osm change_file.osc
./osmupdate 2011-02-15T23:30:00Z change_file.o5c
./osmupdate 2011-02-15T23:30:00Z change_file.osc.gz

Applying Geographical Borders

In the following example, we want to update an OSM file which contains only European OSM data. Since the downloaded Planet Changefiles contain not only Europe, but the whole planet, a lot of unneeded data would be added to this regional file. The -B= argument will clip this superfluous data. 

./osmupdate europe_old.o5m europe_new.o5m -B=europe.poly

You also may use -b= option of osmconvert to specify a bounding box. Please refer to osmconvert program description for further information.

It is not possible to use the --complete-ways option from osmconvert (used to retain all the nodes belonging to a way, even when they lie outside of the bounding box or polygon) when updating.

A more effective way to update a regional OSM file is to use a regional update source instead of a border polygon. To do this, please change the base URL (see a few sections below).

Limit Changefile Categories

If you need to update your OSM file only once a day or once a week, you may want to consider the limitation of changefile category to daily. This will prevent minutely and hourly changefiles from being downloaded and therefore save a lot of data traffic. Use this option: 

--day

For updates every few hours, you usually would not need minutely changefiles. So please consider applying this option: 

--hour --day

Keep Temporary Files

Another way to minimize data traffic is to keep local copies of every downloaded file. This is especially recommended if you are going assemble chronologically overlapping changefiles. The command line option: 

--keep-tempfiles

Verbose Output

osmupdate works very quietly. If you have suspicions that something might go wrong, or if you are just curious, then activate the verbose mode: 

-v

Data Source

Usually, osmupdate downloads the changefile data from https://planet.openstreetmap.org/replication. If you want to use a different source, for example the Fork Project FOSM, a regional update repository, or the data available during the license-change period, you may specify this by the --base-url= option. Some sources require to define a suffix, for example --base-url-suffix=-replicate. Examples: 

--base-url=mirror
--base-url=http://planet.fosm.org/planet/ --base-url-suffix=-replicate
--base-url=https://planet.openstreetmap.org/ --base-url-suffix=-replicate
--base-url=https://planet.openstreetmap.org/redaction-period/ --base-url-suffix=-replicate
--base-url=download.geofabrik.de/europe/germany/bremen-updates

Benchmarks

updating the whole planet

hardware: 16GB ram, 7200rpm disk, source file: No timestamp, 14 days old planet from 30 Aug 2012. 

time ./osmupdate planet-latest.o5m new-planet-latest.o5m --planet-url=https://planet.openstreetmap.org/redaction-period/
real	31m12.246s
user	13m45.716s
sys	0m57.056s

Hardware: Core i7 2600, 16GB RAM, RAID6, source file: pbf, no timestamp, 5 days old planet file from 7 Nov 2012 

time ./osmupdate32 -v planet-latest.osm.pbf planet-latest2.osm.pbf
real    52m52.208s
user    43m39.028s
sys     0m43.891s

Hardware: Phenom II x6 1090T, 32GB RAM, RAID1, 160MB/s per drive sequential, source file: planet-130515.osm.pbf, 15 days old planet file from 16 May 2013 

time osmupdate --day planet-130515.osm.pbf planet-new.osm.pbf
osmupdate: Aging the timestamp by 4 hours for safety reasons.
real    67m30.271s
user    53m33.933s
sys     2m12.440s

Hardware: Low end PC: Pentium G630T, 4GB RAM, 7200 RPM 2,5" HDD, source file: planet.pbf, 3 days old planet file from 22 Jun 2013. Windows 7. 

osmupdate planet.pbf planet.o5m
osmupdate: Aging the timestamp by 4 hours for safety reasons.
Time spent: 50 minutes

Hardware: E3-1231 v3 @ 3.40GHz, 32GB RAM, HDD + SSD ZFS array, FreeBSD, all but two days in cache. planet file of 15 Apr 2013 

osmupdate --max-merge=25 --day -v --keep-tempfiles planet-150413.osm.pbf planet-150824.osm.pbf

Time spent: 1h18m

If downloading diffs, run time is 2h42m.


Please add your benchmark results.

Known issues

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Osmupdate&oldid=2451024"