User:G0ldfish/Scalable Vector Maps
Scalable Vector Maps with Osmium and QGIS
Osmium looks much like the tool I have always been waiting for to create shapefiles from OSM data for use in QGIS (and create scalable maps from it which is my main aim) without the need to set up a database to access with QGIS.
Conversion with Osmium
For the first part you need Linux. Not much of a problem nowadays though. Providing you have enough RAM and free disk space, you can install a VM player like VirtualBox and install e.g. Ubuntu or Debian into it.
The installation of Osmium following the description in the wiki is straightforward with most versions tested. You can try to build the examples after installing, but to me they did not mean much (maybe meant for those who intend to use Osmium as library in own software projects rather).
Instead, change to the osmjs subdirectory and build osmjs:
make osmjs
will create a new executable file named osmjs.
Now we need osm data and a configuration file to test. My configuration file currently looks like this:
var shp_trails = Osmium.Output.Shapefile.open('trails', 'line');
shp_trails.add_field('id', 'integer', 10);
shp_trails.add_field('rel', 'integer', 10);
var shp_pois = Osmium.Output.Shapefile.open('pois', 'point');
shp_pois.add_field('id', 'integer', 10);
shp_pois.add_field('type', 'string', 32);
shp_pois.add_field('name', 'string', 32);
var shp_areapois = Osmium.Output.Shapefile.open('areapois', 'polygon');
shp_areapois.add_field('id', 'integer', 10);
shp_areapois.add_field('type', 'string', 32);
shp_areapois.add_field('name', 'string', 32);
var shp_places = Osmium.Output.Shapefile.open('places', 'point');
shp_places.add_field('id', 'integer', 10);
shp_places.add_field('type', 'string', 32);
shp_places.add_field('label', 'string', 32);
shp_places.add_field('size', 'integer', 3);
var shp_tunnels = Osmium.Output.Shapefile.open('tunnels', 'line');
shp_tunnels.add_field('id', 'integer', 10);
shp_tunnels.add_field('type', 'string', 32);
shp_tunnels.add_field('level', 'integer', 3);
shp_tunnels.add_field('ref', 'string', 32);
shp_tunnels.add_field('grade', 'integer', 1);
var shp_roads = Osmium.Output.Shapefile.open('roads', 'line');
shp_roads.add_field('id', 'integer', 10);
shp_roads.add_field('type', 'string', 32);
shp_roads.add_field('ref', 'string', 32);
shp_roads.add_field('grade', 'integer', 1);
var shp_lines = Osmium.Output.Shapefile.open('lines', 'line');
shp_lines.add_field('id', 'integer', 10);
shp_lines.add_field('type', 'string', 32);
shp_lines.add_field('temporary', 'string', 32);
var shp_background = Osmium.Output.Shapefile.open('background', 'polygon');
shp_background.add_field('id', 'integer', 10);
shp_background.add_field('type', 'string', 32);
var trails = new Array();
Osmium.Callbacks.relation = function() {
if (this.tags.type == 'route' && this.tags.route.match(/^(hiking|foot)$/)) {
for (var i=0; i < this.members.length; i++) {
m = this.members[i].ref;
trails[m] = this.id;
}
}
}
Osmium.Callbacks.node = function() {
if (this.tags.amenity) {
shp_pois.add(this.geom, {
id: this.id,
type: this.tags.amenity,
name: this.tags.name
});
}
else if (this.tags.tourism) {
shp_pois.add(this.geom, {
id: this.id,
type: this.tags.tourism,
name: this.tags.name
});
}
else if (this.tags.place && this.tags.place.match(/^(city|town|village|hamlet|suburb)$/)) {
var size;
if (this.tags.place == 'city') {
size = 144;
} else if (this.tags.place == 'town') {
size = 120;
} else if (this.tags.place == 'village') {
size = 96;
} else {
size = 60;
}
shp_places.add(this.geom, {
id: this.id,
type: this.tags.place,
label: this.tags.name,
size: size
});
}
}
Osmium.Callbacks.way = function() {
if (this.tags.highway) {
if (trails[this.id]>0) {
shp_trails.add(this.geom, {
id: this.id,
rel: trails[this.id]
});
}
var grade;
if (this.tags.tracktype) {
grade = this.tags.tracktype.charAt(5);
}
if (this.tags.tunnel) {
shp_tunnels.add(this.geom, {
id: this.id,
type: this.tags.highway,
level: this.tags.layer,
ref: this.tags.ref,
grade: grade
});
} else {
shp_roads.add(this.geom, {
id: this.id,
type: this.tags.highway,
ref: this.tags.ref,
grade: grade
});
}
} else if (this.tags.waterway) {
shp_lines.add(this.geom, {
id: this.id,
type: this.tags.waterway,
temporary: this.tags.intermittent
});
} else if (this.tags.railway) {
shp_lines.add(this.geom, {
id: this.id,
type: this.tags.railway
});
} else if (this.tags.aerialway) {
shp_lines.add(this.geom, {
id: this.id,
type: this.tags.aerialway
});
} else if (this.tags.aeroway) {
shp_lines.add(this.geom, {
id: this.id,
type: this.tags.aeroway
});
} else if (this.tags.route) {
shp_lines.add(this.geom, {
id: this.id,
type: this.tags.route
});
}
}
Osmium.Callbacks.area = function() {
if (this.tags.amenity) {
shp_areapois.add(this.geom, {
id: this.id,
type: this.tags.amenity,
name: this.tags.name
});
}
else if (this.tags.tourism) {
shp_areapois.add(this.geom, {
id: this.id,
type: this.tags.tourism,
name: this.tags.name
});
}
else if (this.tags.landuse) {
shp_background.add(this.geom, {
id: this.id,
type: this.tags.landuse
});
}
else if (this.tags.natural && this.tags.natural.match(/^(beach|heath|water|wetland|wood)$/)) {
shp_background.add(this.geom, {
id: this.id,
type: this.tags.natural
});
}
else if (this.tags.waterway && this.tags.waterway == 'riverbank') {
shp_background.add(this.geom, {
id: this.id,
type: this.tags.waterway
});
}
}
Osmium.Callbacks.end = function() {
shp_pois.close();
shp_areapois.close();
shp_places.close();
shp_tunnels.close();
shp_roads.close();
shp_trails.close();
shp_lines.close();
shp_background.close();
}
Since each shapefile dataset can only hold one datatype (point, line or polygon), we need at least three shapefile sets. I decided to further break down linear objects into roads, tunnels and other linear objects not directly related to the road network like rivers. There is also an additional polygon layer to hold the points of interests (POI) that are areas instead of points.
Multipolygons
Luckily Osmium has the ability to convert multipolygons properly. To achive this, osmjs is executed like this
./osmjs -2 -m -l sparsetable -j config.js data.osm.pbf
where config.js is the name of your configuration file and data.osm.pbf the osm data in protobuf format.
Relations
Using the 2pass option, it is possible to process relation information. The above configuration file saves ways that are part of hiking relation and later on when the ways are process builds a separate shapefile for them. Note that a) this is likely to be really, really far from the most elegant or most efficient way, but it works sufficently on the 13 MB pbf file I am working with. And b) the approach is solely suitable if you just want to highlight the ways, as the information that a way is part of 2 or more relations is overwritten and thus only one relation stored in the shapefile.
Credits
Big, big thanks to User:Joto for the tool as well as for the article in the German IT magazin "Ix" that finally got me going! I actually overlooked the javascript examples until recent which were very helpful as well.
Import into QGIS
One important thing is to get the projection right. I found it easiest to add a WMS layer first, e.g. the free osm tiles from http://irs.gis-lab.info/. Set the projection to EPSG:3857 as listed there, and activate on the fly reprojection.
Then you can import the generated shapefiles into QGIS (not necessarily under linux, QGIS is available for windows and mac as well). If they align well, you may safely disable or remove the WMS layer.
Add the layers in the order you would like them to be drawn, or move them around later and start to assign styles to the different highways, areas and so on. Note: If you have non-ascii characters e.g. in name fields, make sure the character set is set to UTF-8 when importing.
To define different line styles within one layer, you would open the layer properties and set the style to rule based. After that you can add a filtering rule for each line type you want and set its style indivdually.
Icons for the POI layers are as well easy to import, you only need to drop svg files into the QGIS subdirectory apps->qgis->svg of your installation.
If you want your icons, line width etc. to be resized when zooming (i.e. relative to the map size), you need to choose "Map unit". Note that this works best with a projection that uses metric units instead of degrees (EPSG:3857 does, among others), otherwise a map unit is huge and you often cannot set reasonable small sizes because the number of decimals allowed in the forms is limited.
To add icons to the area POIs change the default "Simple fill" to "Centroid fill", then in the change marker dialog switch from "Simple marker" to "SVG marker" to have all icons available.
I admit that it is a lot of work to set up all the line styles, area fillings and icons, but the good things about it:
- updating the map is easy, as QGIS only references the shape files. Thus, you can generate new ones and simply replace the old files
- making a map from another area should be similarly easy (but I guess every new map will reveal something that is not perfect yet)
- and you can share your style definitions with others (best along with the osmjs configuration file because the data structure must match)
Credits
The styles used for the ways are largely based on User:Mayeul's work which he kindly shared.
Additional data
When making maps of islands or coastal areas, objects with natural=coastline are not always helpful as creating proper polygons from them may fail. Luckily, the coastline file mapnik uses for higher zoom levels is available from http://tile.openstreetmap.org/processed_p.tar.bz2. To cut the huge coastline shapefile down to the region of interest, select the coastline objects you would like to keep and use "Clip" from QGIS Geoprocessing tools. This works well at least for islands. Openstreetmapdata.com provides sea polygons, land polygons and coastlines as well.
Contour lines can be generated from SRTM or ASTER data. When using the latter, don't forget correct attribution. Gdal tools for conversion should be already installed if Osmium works. You can use gdal_translate to create geotiffs from SRTM hgt files, gdalwarp to combine two or more geotiffs, and finally e.g.
gdal_contour -a height infile.tif outfile.shp -i 20
to create a shapefile with contour lines every 20 m.
Exporting from QGIS
QGIS' print composer allows to define areas to export into svg or pdf files. Version 1.7.4 throws a warning about svg export being buggy, but I encountered some issues with the pdf export as well. One is easy to solve: the background color from the normal view is missing in the print composer, you need to set it separately (tab "item").
Then I had trouble with the labels, the spacing between the letters was completely broken. This does not seem to be related to the font and happened both under windows and linux, rather some weird resizing effect when using very, very small font sizes. My current workaround is to export the labels separately to svg, convert it (after some editing) to pdf with Inkscape and then combine both pdfs to one. Not too convenient as it requires some additional manual steps to be executed whenever the map is updated.
Similarly, very thin lines (e.g. 0.5 map units with units set to meter) look fine within QGIS, but they are considerably thicker after exporting.
The labels I use for the contour lines behaved somewhat strange as well: defined in map units, they get smaller when zooming out (just as expected) until a certain zoom level when they suddenly get huge. This happened in the print composer as well until I changed the resolution to 72 dpi. This problem is not reproducable anymore though, right now the labels simply disappear instead.