User:Species/howto-osm2shape

Shapefiles and OSM data

Most commercial GIS software uses Shapefiles.

You have to understand that Shapefiles work differently (much more like relational databases) than OSM:

1. In a Shapefiles there are three entities, Points, Polylines and Polygons - each on a single layer:
   • Points are quite the same as a Node in OSM
   • Polylines represent linear features like streets, fences etc.
   • Polygons represent closed ways (areas), including: highways with area=yes; landuses; buildings, Multipolygons
2. In a shapefile, ways are separated from their nodes - of course the ways are represented by nodes, but their tags are stored in nodes on a different point-layer, with their same coordinates as their only connection - Ways cannot have tagged nodes!
3. tags (key-value) pairs are not stored directly, but in "columns" like in a database. For each key, an explicit column has to be used. For a sufficiently amount of different tags you can end up with hundreds of columns - most of their fields empty (e.g. an OSM-file with 100MB xml can ends up in 10GB shapefile, if a lot of columns are created).
4. The "shapefile" consist of minimum three different files:
   • .shp — shape format; the feature geometry itself.
   • .shx — shape index format; a positional index of the feature geometry to allow seeking forwards and backwards quickly.
   • .dbf — attribute format; columnar attributes for each shape, in dBase IV format.
5. Shapefiles can have any projection, defined in an .prj-file! The projection can be a major source of error - whereas in OSM for data always EPSG:4326, aka WGS84 is used. Slippy Maps are in EPSG:3857, aka Mercator. For most OSM Editors (e.g. Merkaartor), Shapefiles have to be re-projected into WGS84 to be used correctly!

To use Shapefiles within OSM, convert it into WGS84 with QGIS:

• load the Shapefile: Layer -> add Vector Layer, choose the Shapefile
• In the (left) Layers Tab, right-click the Layer -> Save As: Shapefile
• Format: ESRI Shapefile
• Name: Click Browse, enter a name, save
• Encoding: UTF-8
• as CRS (Coordinate Reference System) click browse:
  • In the search field, type: 4326
  • Choose WGS 84 - EPSG:4326 - OK
• now click OK, you're finished.

For the inability to efficiently store less used keys, extracts from Geofabrik or the OSM2GIS website provide only the most-used keys, e.g highway, building, name.

How to create Shapefiles from OSM data

Some people report this can be done natively with QGIS 2.0.

Where this is not possible or simply the data files are too big i found a way here:

Commandline Way

We are using the commandline if QGIS 2.0 cannot handle big files (like whole countries) or we want to automate the process. Used tools:

• Osmconvert
• wget or other download tool
• ogr2ogr (version 1.10 NEEDED!). ogr2ogr can be found in package „gdal-bin“ on Debian/Ubuntu.

POIs (incl. ways, rels!) to nodes

this one is easy thanks to Osmconvert:

• Download OSM file (either via JOSM, osm.org or whole extracts from geofabrik)
• osmconvert $infile.filename --all-to-nodes [--max-objects=1000000000] -o=$outfilename.osm
• open in JOSM, save as .geojson
• open in QGIS, save as shape

Boundaries

Can be problematic, because sometimes ogr2ogr fails in the process. Basic requirement is that the relations contains no errors because of broken inner/outer rings.
No other data except the relations have to be present in the .osm-file! Even that sometimes ogr2ogr fails with ERROR 1: Attempt to write non-polygon (GEOMETRYCOLLECTION) geometry to POLYGON type shapefile, haven't found out why.

Data first has to be downloaded via overpass, example with wget:

wget --post-file=$QUERY-FILE $OVERPASS_SERVER -O $OVERPASS_RESULTFILE 

where $QUERY-FILE is a simple text-file with the following content:

[out:xml]
[timeout:900]
;
( 
 relation
   ["boundary"="administrative"]
   ["type"~"boundary|multipolygon"]
   ["admin_level"="8"]
   (46.3165842,9.3164063,49.0234615,17.1826172);
);
out body;
>;
out skel;

hints for query-building:

• If you want to download only e.g. austrian „Bezirke“, add a country-specific tag like ["ref:at:gkz"].
• The BBOX must be touching all boundaries you want to have with this admin_level.
• If you want to filter further, add e.g. ["name"="Liechtenstein"], if you want only the country boundary (admin_level 2).

The result-file is NOT sorted node;way;relation, you have to do sort it yourself!

Eventually if too much tags are on it (like name:* on countries), you have to filter to specific tags:

osmfilter $SORTED_FILE "--keep-tags=all name= admin_level=" -o=$OUTFILE.osm

Now you can convert it with ogr2ogr

ogr2ogr -nlt MULTIPOLYGON $DEST-FOLDERNAME $FILENAME.osm -skipfailure

Maybe you want to add specific columns: -select name,admin_level,ref:at:gkz

GUI Way

Only works with points and ways!

But now there is a much simpler way: via GeoJSON. QGIS can handle GeoJSON, but only one datatype (node, way, polygon) per file, so you have to split the data for export in JOSM:

• Load data into JOSM
• For Points:
  • Search (CTRL-F) type:node, copy selected nodes (CTRL-C)
  • File -> add new Layer, Paste (CTRL-V)
  • (Search for untagged nodes and remove: CTRL-F "untagged", delete selected nodes)
• For Ways:
  • Search (CTRL-F) "type:way", copy selected ways (CTRL-C)
  • File -> add new Layer, Paste (CTRL-V)
  • Remove all tagged nodes (important!): CTRL-F "type:node", delete all tags of now selected nodes
  • (Search for closed ways, e.g. landuses and remove them: CTRL-F "closed")
• Save as .geojson (use file-ending .geojson, not .json!)
• in GGIS, Layer -> Add Vector Layer... : choose type GeoJSON, open file.
• in Layers tab, right click "Save As... : ESRI Shapefile"