User:Bgirardot/How To Convert osm .pbf files to Esri Shapefiles

Very quick guide to converting .pbf files to shapefiles

Use GDAL 1.10 or above.

Short version:

• The .pbf file itself can be thought of as a database
• GDAL (ogr2ogr) will automatically view the .pbf as having 5 tables: points, lines, multipolygons, multilinestrings (routes), other_relations (osm relations). These notes only deal with the first 3 tables.
• The osmconf.ini file tells GDAL what columns you want in your tables. Columns are the key name from key=value tag pairs in OSM.
• You will use gdal sql statements to select the data you want from your tables in the .pbf file to go into the shapefile.
• Your sql statements can only use the column names you defined in the osmconf.ini file.
• Data from points, lines, multipolygon tables can not be mixed in shapefiles. If you have objects or features made up of points, lines, multipolygons, you will need a shapefile for each one. For example in OSM airports often involve all 3 data types so to get all of the OSM data for airports, you would need 3 shape files.
• I used ogr2ogr to generate a sqlite file and then used sqlitebrowser to review the data and used that to guide me in editing my osmconf.ini file. The default .ini file will get you going and I just modified that.
• Basically I looked in the 'other_tags' field to see what tags were not being broken out into columns and then added them to my osmconf.ini file so I could access them in the ogr2ogr sql statements.
• Examples below, many thanks to dodobas for helping me get this figured out.

My final osmconf.ini file:

#
# Configuration file for OSM import
#

# put here the name of keys for ways that are assumed to be polygons if they are closed
# see http://wiki.openstreetmap.org/wiki/Map_Features
closed_ways_are_polygons=aeroway,amenity,boundary,building,craft,geological,historic,landuse,leisure,military,natural,office,place,shop,sport,tourism

# comment to avoid laundering of keys ( ':' turned into '_' )
attribute_name_laundering=yes

# uncomment to report all nodes, including the ones without any (significant) tag
#report_all_nodes=yes

# uncomment to report all ways, including the ones without any (significant) tag
#report_all_ways=yes

[points]
# common attributes
osm_id=yes
osm_version=no
osm_timestamp=no
osm_uid=no
osm_user=no
osm_changeset=no

# keys to report as OGR fields
attributes=name,barrier,highway,ref,address,is_in,place,man_made,amenity,seamark,aeroway,tourism,natural,bridge,ford,shop,power,building,harbour,fixme
# keys that, alone, are not significant enough to report a node as a OGR point
unsignificant=created_by,converted_by,source,time,ele
# keys that should NOT be reported in the "other_tags" field
ignore=created_by,converted_by,source,time,ele,note,openGeoDB:,fixme,FIXME
# uncomment to avoid creation of "other_tags" field
#other_tags=no

[lines]
# common attributes
osm_id=yes
osm_version=no
osm_timestamp=no
osm_uid=no
osm_user=no
osm_changeset=no

# keys to report as OGR fields
attributes=name,highway,waterway,aerialway,barrier,man_made,amenity,natural,bridge,aeroway
# keys that should NOT be reported in the "other_tags" field
ignore=created_by,converted_by,source,time,ele,note,openGeoDB:,fixme,FIXME
# uncomment to avoid creation of "other_tags" field
#other_tags=no

[multipolygons]
# common attributes
# note: for multipolygons, osm_id=yes instanciates a osm_id field for the id of relations
# and a osm_way_id field for the id of closed ways. Both fields are exclusively set.
osm_id=yes
osm_version=no
osm_timestamp=no
osm_uid=no
osm_user=no
osm_changeset=no

# keys to report as OGR fields
attributes=name,type,aeroway,amenity,admin_level,barrier,boundary,building,craft,geological,historic,land_area,landuse,leisure,man_made,military,natural,office,place,shop,sport,tourism
# keys that should NOT be reported in the "other_tags" field
ignore=area,created_by,converted_by,source,time,ele,note,openGeoDB:,fixme,FIXME
# uncomment to avoid creation of "other_tags" field
#other_tags=no

[multilinestrings]
# common attributes
osm_id=yes
osm_version=no
osm_timestamp=no
osm_uid=no
osm_user=no
osm_changeset=no

# keys to report as OGR fields
attributes=name,type
# keys that should NOT be reported in the "other_tags" field
ignore=area,created_by,converted_by,source,time,ele,note,openGeoDB:,fixme,FIXME
# uncomment to avoid creation of "other_tags" field
#other_tags=no

[other_relations]
# common attributes
osm_id=yes
osm_version=no
osm_timestamp=no
osm_uid=no
osm_user=no
osm_changeset=no

# keys to report as OGR fields
attributes=name,type
# keys that should NOT be reported in the "other_tags" field
ignore=area,created_by,converted_by,source,time,ele,note,openGeoDB:,fixme,FIXME
# uncomment to avoid creation of "other_tags" field
#other_tags=no

The command I used to create the sqlite database file to help me configure my .ini file and test my sql statements:

ogr2ogr -f SQLite my.sqlite OSM_data.pbf

And here are the commands I used to generate my shapefiles based on thematic layers:

ogr2ogr -overwrite -f "ESRI Shapefile" amenities_all_points.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,amenity,other_tags from points where amenity is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" amenities_all_polygons.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,amenity,other_tags from multipolygons where amenity is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" health-schools_points.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,amenity,other_tags from points where amenity="clinic" or amenity="hospital" or amenity="school" or amenity="pharmacy"' &&
ogr2ogr -overwrite -f "ESRI Shapefile" health-schools_polygons.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,osm_way_id,name,amenity,other_tags from multipolygons where amenity="clinic" or amenity="hospital" or amenity="school" or amenity="pharmacy"' &&
ogr2ogr -overwrite -f "ESRI Shapefile" airports_points.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,aeroway,amenity,other_tags from points where aeroway is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" airports_polygons.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,osm_way_id,name,aeroway,amenity,other_tags from multipolygons where aeroway is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" airports_lines.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,aeroway,amenity,other_tags from lines where aeroway is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" villages_points.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,place,other_tags from points where place is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" buildings_polygons.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,osm_way_id,name,building,amenity,other_tags from multipolygons where building is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" natural_polygons.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,osm_way_id,name,natural,amenity,other_tags from multipolygons where natural is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" natural_lines.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,natural,amenity,other_tags from lines where natural is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" landuse_other_polygons.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,osm_way_id,name,landuse,other_tags from multipolygons where landuse is not null and landuse!="residential"' &&
ogr2ogr -overwrite -f "ESRI Shapefile" landuse_residential_polygons.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,osm_way_id,name,landuse,other_tags from multipolygons where landuse="residential"' &&
ogr2ogr -overwrite -f "ESRI Shapefile" roads-paths_lines.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,highway,amenity,other_tags from lines where highway is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" waterways_lines.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,natural,amenity,other_tags from lines where waterway is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" towers-antennas_points.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,man_made,other_tags from points where man_made="tower" or man_made="antenna" or man_made="mast"' &&
ogr2ogr -overwrite -f "ESRI Shapefile" harbours_points.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,name,harbour,other_tags from points where harbour is not null' &&
ogr2ogr -overwrite -f "ESRI Shapefile" grassy-fields_polygons.shp vanuatu-2015-04-06T202102Z.pbf -progress -sql 'select osm_id,osm_way_id,name,leisure,sport,amenity,other_tags from multipolygons where leisure="pitch" or leisure="common" or leisure="golf_course"'