Geobase/NRN - OSM Map Feature
This page contains outdated information. |
This page describes the status of and method for importing the Geobase, Canadian National Road Network (NRN) into OpenStreeMap.
The Geobase NRN Product (Data description) can be found on their website: http://www.geobase.ca/geobase/en/data/nrn/description.html
Import Process Overview
The first stage of the NRN import is to import all of the roads in the Geobase NRN that don't seem to conflict with roads already in OSM. Doing this will give OSM a more complete road network without having duplicate roads or requiring bulk deletes of user contributed data.
The general procedure for each area is
- Obtain the NRN data files from Geobase as a shapefile (SHP).
- Generate a shapefile for the NRN data in an area (converted to WGS84), and a second shapefile based on existing OSM roads in the area. Ensure that both the OSM and NRN data are using the same geographic coordinate system. This can be done in a number of ways (See below).
- Import the shapefiles into RoadMatcher and run the AutoMatch routine using the NRN as the base data set.
- Export the results
- Run the geobase2osm script against the NRN GML file for the area passing the RoadMatcher results with the -e flag. This will generate a .osm file with the complete NRN road network in the area and a second .standalone.osm file that only contains roads that RoadMatcher has determined are not already in OSM
- Import the resulitng .standalone.osm file using bulk_upload.py
This procedure will get many roads into OSM but some will be missed because RoadMatcher isn't able to determine if the road is in OSM or not. Mappers will have to manually add these roads in.
Setting up the tools
- Create a Postgresql database. This page uses "gis" for the database name.
- Add spatial references, e.g. on Linux
psql gis < /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql
- Create functions in postgis for extracting the OSM and NRN data in the same geographic coordinates. These functions could look similar to
DROP TYPE nrn_data; CREATE TYPE nrn_data AS ( gid integer, the_geom geometry, nid text, rtename1en text, l_stname_c text, l_placenam text ); CREATE OR REPLACE FUNCTION select_nrn_roadtile(coordinates text) RETURNS SETOF nrn_data AS $$ SELECT gid,ST_Transform(the_geom,3348),nid, rtename1en, l_stname_c,l_placenam FROM nrn_roadseg WHERE ST_Intersects(ST_Transform(the_geom,4326), ST_GeomFromEWKT($1)) $$ LANGUAGE SQL; CREATE TYPE osm_data AS ( way geometry, osm_id integer, name text ); CREATE OR REPLACE FUNCTION select_osm_roadtile(coordinates text) RETURNS SETOF osm_data AS $$ SELECT ST_Transform(way,3348),osm_id,substr(name,0,80) FROM planet_osm_line WHERE ST_Intersects(way,ST_GeomFromEWKT($1)) AND highway is not null $$ LANGUAGE SQL;
Note that there can be problems with accented characters, for example in Québec and other francophone areas. I've applied the 'translate' function in Sam's script, which translates the accented characters to their non-accented counterparts. By using this version of select_osm_roadtile the OSM.shp file can be opened with OpenJUMP. As far as I can tell, this is harmless. It won't impact the import process, because OSM.shp is only used as a mask for RoadMatcher. --Fsteggink 02:03, 30 August 2009 (UTC)
- It's actually SteveS script, so i dont know if that error has been fixed in the geobase2osm script. But for the canvec2osm script that error has been fixed to show the accents. --acrosscanadatrails 18:19, 30 August 2009 (UTC)
CREATE OR REPLACE FUNCTION select_osm_roadtile(coordinates text) RETURNS SETOF osm_data AS $$ SELECT ST_Transform(way,3348),osm_id,substr(translate(name, E'\303\200\303\201\303\202\303\203\303\204\303\205\303\206\303\207\303\210\303\211\303\212\303\213\303\214\303\215\303\216\303\217\303\221\303\222\303\223\303\224\303\225\303\226\303\230\303\231\303\232\303\233\303\234\303\235\303\237\303\240\303\241\303\242\303\243\303\244\303\245\303\246\303\247\303\250\303\251\303\252\303\253\303\254\303\255\303\256\303\257\303\261\303\262\303\263\303\264\303\265\303\266\303\270\303\271\303\272\303\273\303\274\303\275\303\277','AAAAAAACEEEEIIIINOOOOOOUUUUYSaaaaaaaceeeeiiiinoooooouuuuyy'),0,80) FROM planet_osm_line WHERE ST_Intersects(way,ST_GeomFromEWKT($1)) AND highway is not null $$ LANGUAGE SQL;
On Linux, it is possible to just pipe the above definitions to "psql -d gis". The above SQL will generate SHP and OSM files in the epsg:3348 geographic datum. This is a meters based coordinate system used in Canada.
Creating OSM and NRN Shapefiles with PostGIS
- Download GeoBase NRN data as both GML and SHP from GeoBase
- Obtain OSM data for the area. One can use canada.osm (or planet.osm). It is also possible to use JOSM to download desired area of interest (make sure it includes the chosen NTS area, e.g. josm --download=49.25,-123,49.5,-122.5). Save as .osm, e.g. 092g07.osm.
- Load OSM data into postgis ("gis" database):
cp /path/to/osm2pgsql/default.style . osm2pgsql -l -d gis 092g07.osm
- Load GeoBase NRN shapefile into postgis. Example for BC:
shp2pgsql -s 4326 NRN_BC_6_0_ROADSEG nrn_roadseg |psql -d gis -f -
- Generate shapefiles for the area you are interested in
pgsql2shp -f OSM.shp gis "select * FROM select_osm_roadtile('SRID=4326;POLYGON((-122.5 49.25,-123 49.25,-123 49.5,-122.5 49.5,-122.5 49.25))')" pgsql2shp -f NRN.shp gis "select * FROM select_nrn_roadtile('SRID=4326;POLYGON((-122.5 49.25,-123 49.25,-123 49.5,-122.5 49.5,-122.5 49.25))')"
RoadMatcher
RoadMatcher is a plugin to the JUMP GIS system that will examine two datasets of roads and match roads between them. Roadmatcher is used as part of the GeoBase import to avoid duplicating roads that are already in OSM. Note: This RoadMatcher version works with OpenJUMP, further developed JUMP variant: RoadMatcher for OpenJUMP
- Open the NRN and OSM data in JUMP
- Create a RoadMatcher session with the NRN data set the reference data set. Leave auto match/conflate to off. If there are errors in OSM geometry that prevent RoadMatcher from working, fix those in JOSM, upload and repeat OSM import. It may take a couple iterations to fix the errors. One possible workflow is to use "Feature Info Tool" in OpenJUMP to find out OSM ID of the offending feature, then use Ctrl-F (Find feature) to locate the feature in JOSM.
- Adjust AutoMatch settings (menu RoadMatcher->ConflationOptionsPlugin). The proper settings depend on the quality of the OSM data and the urban vs rural mix of the roads (see below).
- Run RoadMatcher->AutoConflate, and run it with auto adjust set to off. "Find Matched Roads" can be useful, but it sometimes chokes on larger datasets, so if it happens to you, leave that off. Standalone segments are what matters in this process.
- Review the output, and work through the road segments that RoadMatcher was not able to automatch (go through "unknown" segments and mark them as standalone, if appropriate - see the RoadMatcher manual). Keep in mind that the current procedure only imports the "standalone" segments!
- Enable exporting the NID to the result layer (menu RoadMatcher->Result->Options, select the NRN dataset, move the NID field to the right column). If you don't do this, geobase2osm.py will not find any standalone segments, and your import file will be empty.
- Then generate a result layer.
- Save the result layer to a file (i.e. Result.jml)
The following settings tend to work well for many areas when you convert your dataset to meters
Minimum Segment Size | 1.0 |
Standalone Distance | 50 |
Match Distance | 50 |
For areas with many roads mapped from low-res imagery you might want to change the 50's to 100's.
When working in wgs84
Minimum Segment Size | 5.0E-7 |
Standalone Distance | 5.0E-4 |
Match Distance | 5.0E-4 |
have produced reasonable results in some areas
Exporting and uploading to OSM
- Create a file with a WKT with bounds for the chosen area. E.g. 092g07.txt:
POLYGON((-122.5 49.25,-123 49.25,-123 49.5,-122.5 49.5,-122.5 49.25))
- Run Geobase2osm, e.g.
geobase2osm.py -o out.osm -b 092g07.txt -e Result.jml -i NRN_BC_6_0_GEOM.gml
It's a good idea to look at the resulting .standalone.osm file in JOSM before uploading.
Finally, use bulk_upload.py to upload resulting data:
bulk_upload.py -i out.osm.standalone.osm -u username -p password -c 'geobase nrn import'
After the Import
The newly imported ways need to be connected to existing OSM ways. The Validator plugin in JOSM provides a good way to catch common problems. Intersecting ways will need a new node placed at the intersection. Ways that end close to another way can use a "Join node to way" tool in JOSM. Bridges may have to be created where omitted previously. Also, the roads that are oneway need to marked as such (GeoBase import does not provide this info). There are many other problems that validator plugin can find.
Road Names from StatsCan
GeoBase only has road name and street address information for a handful of provinces. Statistics Canada has a road network data set that can be imported into OSM that has names for most roads. However the positional accuracy of the StatsCan road network isn't very good. The Canadian OSM community has decided to use the geometries from GeoBase because they are far superior to the StatsCan 2007/2008 data.
Download
Road names can be extracted from the StatsCan dataset and attached to the GeoBase derivied roads during import. The following process is being used.
- Generate a shape file from the StatsCan data containing the geometry, roadname, nid and address data
- Import the GeoBase data and the StatsCan data into RoadMatcher and generate a match (with no auto-adjust) between the roads. 100m match distance seems to work okay.
- Save the results to a file
- Pass this results to geobase2osm.py with the -n option when converting the GeoBase roads to OSM format. The script will use the StatsCan name if no name exists in the GeoBase data.
It is also possible to add road names to .osm file (that is tagged with GeoBase uuids) with the addnames_statscan.py script
Example SQL for extracting StatsCan data:
-- -- For creating shapefiles with StatsCan road names from the -- StatsCan road network data. -- DROP TYPE statscan_data CASCADE; CREATE TYPE statscan_data AS ( ngd_id int, the_geom geometry, name text, type text, direction text, addr_fm_le bigint, addr_to_le bigint, addr_fm_rg bigint, addr_to_rg bigint ); CREATE OR REPLACE FUNCTION select_statscan_roadtile(coordinates text ) RETURNS SETOF statscan_data AS $$ SELECT gid,ST_Transform(the_geom,3348),name,type,direction,addr_fm_le,addr_to_le,addr_fm_rg,addr_to_rg FROM statscan.road_network08 WHERE ST_Intersects(st_transform(the_geom,4326),ST_GeomFromEWKT($1)) $$ LANGUAGE SQL;
Status
Important notice: Due to the HUGE volume of data to be imported (12,538 canvec tiles, 62 GeoBaseNRN files, and 5,756 GeobaseNHN tiles) where each needs to be uploaded and recorded with the latest status of import, listing what was imported and when, and who imported it. A Google Docs spreadsheet was created. Only Gmail users have access to view this, and to gain access to edit it, just ask on the talk-ca discussion list, and someone can add you to the list of editors, or contact Sam directly.
The bulk of this import is actually being done by only a few people. Where there is alot of OSM activity, the converted .osm files are (and will be) made available, so users can download the converted .osm file and use it to manually copy over whatever data that is needed.
The following areas have been imported (this chart is now outdated as of August 23 2009) but will remain below to ensure that everything was copied over correctly.
Area | Status | Date | Link to complete NRN OSM data for area' | Notes |
---|---|---|---|---|
Alberta portion of NTS Tile: 074 (near Fort McMurray) | Imported | Jan 17 2009 | 074.jan16.osm.gz | imported by? Comments? |
Alberta portion of NTS Tile: 084 | Complete | Jan 24 2009 | Imported By: Stevens | |
Quebec NTS Tile: 021E05 | In progress | Jan 30 2009 | Imported By: Jeanseb | |
Quebec Témiscamingue (Abitibi-South) | Complete | May 09 2010 | Rectangle "-77W 46N -80W 48N" from NRN_QC_2_0_GEOM.gml | Imported By: Yan Morin |
Quebec Hautes-Laurentides | Complete | Jul 30 2009 | Rectangle "-74W 45.96N, -77W 48N" from NRN_QC_2_0_GEOM.gml | Imported By: Yan Morin |
Quebec - Côte-de-Beaupré, NTS Tile: 021M2 (north side of St-Lawrence only) | Complete | Aug 18 2009 | Imported By: Pierre-Luc Beaudoin with help from Yan Morin View on OSM | |
Alberta portion of NTS Tile: 083 | Complete | Feb 6 2009 - Mar 9 2009 | 083M excluded 083M complete
083NOP complete 083NOP excluded 083LKJI complete 083LKJI excluded 083EFDC complete 083EFDC excluded 083H 13,12 excluded 083H 13,12 complete 083H 5,4 excluded 083H 5,4 complete 083H 2,3,6,7 excluded 083H 2,3,6,7 complete 083H 15,16,10 excluded 083H 15,16,10 complete 083H 11 excluded 083H 11 complete 083H 14 excluded 083H 14 complete 083H 1,8 excluded 083H 1,8 complete 083G excluded 083G complete 083A excluded 083A complete 083B excluded 083B complete |
Imported By: Stevens |
Ontario NTS Tile: 040I05 (Ridgetown) | Complete | Feb 22 2009 Jun 28 2009 | geo_040I05.osm.gz excluded | Imported By: jdp1120 |
Ontario NTS Tiles: 040I09-long point 040I10-port burwell 040I15-tillsonburg 040I16-simcoe | Complete | Aug 2009 | geo_040I09_scn_alone.osm.gz 040I09 excluded
geo_040I09_scn.osm.gz 040I09 complete geo_040I10_scn_alone.osm.gz 040I10 excluded geo_040I10_scn.osm.gz 040I10 complete geo_040I15_scn_alone.osm.gz 040I15 excluded geo_040I15_scn.osm.gz 040I15 complete geo_040I16_scn_alone.osm.gz 040I16 excluded geo_040I16_scn.osm.gz 040I16 complete |
Imported By: jdp1120
View 040I09-long point on OSM View 040I10-port burwell on OSM View 040I15-tillsonburg on OSM View 040I16-simcoe on OSM |
Ontario NTS Tile: 040I11 (Port Stanley) | Complete | Feb 7 2009 Jun 27 2009 | geo_040I11.osm.gz excluded | Imported By: jdp1120 |
Ontario NTS Tile: 040I12 (Bothwell) | Complete | Feb 12 2009 Jun 27 2009 | geo_040I12.osm.gz excluded | Imported By: jdp1120 |
Ontario NTS Tile: 040I13 (Strathroy) | Complete | Feb 20 2009 Jun 28 2009 | geo_040I13.osm.gz excluded | Imported By: jdp1120 |
Ontario NTS Tile: 040I14 (St. Thomas) | Complete | Feb 22 2009 Jun 28 2009 | geo_040I14.osm.gz excluded | Imported By: jdp1120 |
Ontario NTS Tiles: 040J 01 Wheatley 02 Essex 03 Amherstburg 06 Windsor 07 Belle River | In Progress | March 02 2009 Jun 28 2009 statscan G15 J01 J02 | geo_040Jse.osm.gz | Imported By: jdp1120 |
Ontario NTS Tiles: 040J (excluding 01,02,03,06,07, see above) | Complete | June 2009 | 040J excluded | Imported By: Stevens |
Ontario NTS Tile: 040O01 (Brights Grove) | Complete | Sept 9 2009 | geo_040O01_scn_alone.osm.gz excluded | Imported By: jdp1120 |
Ontario NTS Tile: Ontario NTS Tile: 040P (north from London) | Complete | Feb 14 2009, Jun 2009 | 040P excluded | Imported By: jdp1120, User:stevens |
Alberta portion of NTS Tile: 082 | Complete | Mar 13 2009 |
082 PIH excluded 082 PIH complete 082 NO excluded 082 NO complete 082 JG excluded 082 JG complete |
Imported By: Stevens View on OSM |
Alberta portion of NTS tiles: 072,073 | Complete | Mar 21 2009 | 072MLE excluded
072MLE complete 073ED excluded 073ED complete 073ML excluded 073ML complete |
Imported By: Stevens |
British Columbia NTS tile: 082E | Complete, except for subtiles 13, 14 | Jun 13 2009 | http://drop.io/bvhw44o/chronological (files with a date of June 12 2009) | All subtiles except 13 & 14 will are done and uploaded. Because there's been a reasonable amount of mapping done around Kelowna, and I need to talk to those mappers first, these will wait for a bit. Being imported by: Austin Henry |
British Columbia NTS tiles: 092b | In progress | July 2 2009 | Manually Importing from converted file. | Imported By: Mappers In Greater Victoria Area |
British Columbia NTS tiles: 092c | Complete | April 23 2009 | 092C excluded 092C complete | Imported By: Stevens |
British Columbia NTS tiles: 092g07 | Complete, uploaded | May 30 2009 | Imported By: Mbiker | |
British Columbia NTS tiles: 092g02 | Complete, uploaded | June 5 2009 | Imported By: Mbiker | |
British Columbia NTS tiles: 092G | In progress | Being Imported By: Mbiker | ||
British Columbia NTS tiles: 092H | Complete, uploaded | September 19 2009 - September 24 2009 | excluded/standalone | Imported By: Adam Dunn |
British Columbia NTS tiles: 092N | Complete, uploaded | September 24 2009 | excluded standalone | Imported By: Adam Dunn |
British Columbia NTS tiles: 102I, 92L, 92K, 102H, 92E, 92F (except 92F01) | Complete, uploaded | Sept 09 2009 | http://drop.io/bvhw44o/chronological (files with a date of September 9 2009) | Imported By: Austin Henry |
Ontario 031 | Complete | April 23 2009 - June 2009 | 031E06 excluded 031E06 complete 031E (except 06) excluded 031E (except 06) complete
031B excluded 031B complete 031C excluded 031C complete 031D excluded 031D complete 031ML excluded 031ML complete 031G(2,3,6,7) excluded 031G(2,3,6,7) complete 031G(4,5) excluded 031G(4,5) complete - Ottawa 31G(1,8,10,11) excluded 31G(1,8,10,11) complete
|
Imported By Stevens
View on OSM |
Ontario 030 | Complete | April 23 2009-June 2009 | 030M05(Hamilton-Oakville) excluded
030M05 complete 030M13 excluded 030M13 complete 030M 15,16 excluded 030M 15,16 complete 030N excluded 030N complete 030M11 complete 030M03,030M04 excluded 030M03 030M04 complete 030M14 complete 030M14 excluded 030L13,030L14 excluded 030L13,030L14 complete | |
Ontario 042, 043 | Complete | May 2009
043,042 excluded 043,042 complete
|
Imported By Stevens
[View on OSM]
| |
Ontario 041 | Complete | May 2009 | Imported By Stevens | |
Ontario 05(2,3,4) | Complete | June 2009 | Imported By Stevens
| |
Prince Edward Island 011 | In Progress | June 2009 | Currently fixing OSM errors as highlighted by Road Matcher |
Areas to not receive the bulk import
Areas that won't be bulk uploaded (If you don't want GeoBase roads imported into your area please add the 6 digit NTS tile reference here)
Area | Link to complete NRN OSM data for area' | Notes |
---|---|---|
030M12 | 030M12 complete | Multiple mappers decide to delete data as the import was progressing |
Geobase Road Segment class
Geobase filename: (NRN_provinceName_version_edition_ROADSEG.shp)
Geobase Attribute | Attribute Value | OSM tag |
---|---|---|
created_by=?? source=geobase | ||
Address Range Digitizing Direction Flag (r_adddirfg, l_adddirfg) |
Same direction (1) Opposite direction (2) Not applicable (3) |
based on oneway = opposite geobase:r_adddirfg = opposite default = same; blank = same |
Address Range NID (adrangenid) |
a uuid |
geobase:adrangenid = |
Exit number (exitnbr) |
IdNumber (char 10) |
geobase:exitnbr hopefully this is a node joining the exit to the way. highway=motorway_junction ref = exitnbr |
First house number (r_hnumf, l_hnumf) |
House numbre address |
see Karlsruhe? :-) |
Functional road class (roadclass) |
Freeway (1) |
|
Expressway / Highway (2) |
||
Arterial (3) |
||
Collector (4) |
||
Local / Street (5) |
||
Local / Strata (6) |
||
Local / Unknown (7) |
||
Alleyway/Lane (8) |
||
Ramp (9) |
Need local context. Depends on the way type. | |
Resource / Recreation (10) |
||
Rapid Transit (11) |
||
Service Lane (12) |
||
Winter (13) |
?? | |
NID (nid) |
A UUID |
nat_ref=NID geobase:nid = nid |
Number of lanes (nbrlanes) |
nbrlanes |
lanes=nbrlanes geobase:nbrlanes = nbrlanes lanes = nbrlanes |
Official place name (r_placenam, l_placenam) |
Place name |
name=l_placenam,r_placenam ?? |
Official street name concatenated (r_stname_c, l_stname_c) |
Concatenated street name, street type ... |
name=l_stname_c, r_stname_c ?? |
Paved road surface type (pavsurf) |
Unknown (-1) |
|
None (0) |
||
Rigid (1) |
||
Flexible (2) |
||
Blocks (3) |
surface=cobblestones | |
Pavement status (pavstatus) |
Paved (1) |
surface=paved |
Unpaved (2) |
surface=unpaved | |
Road segment id (roadsegid) |
Dataset segment unique identifier |
|
Route name english (rtename1en, rtename2en,rtename3en,rtename4en,) |
English national or sub-national route name |
name:en= ?? |
Route name french (rtename1fr, rtename2fr,rtename3fr,rtename4fr,) |
French national or sub-national route name |
name:fr= ?? |
Route number (rtnumber1, rtnumber2,rtnumber3,rtnumber4,rtnumber5) |
Numbered route (up to 5 numbers possible) |
ref=rtnumber1, rtnumber2 ... |
Structure Id (structid) |
A national unique identifier assigned to the Road Segment or the set of adjoining Road Segments forming a structure |
?? |
Structure name english (strunameen) |
English structure name |
name:en= ?? |
Structure name french (strunamefr) |
French structure name |
name:fr= ?? |
Structure type (structtype) |
None (0) |
|
Bridge (1) |
bridge=yes | |
Bridge covered (2) |
bridge=yes | |
Bridge moveable (3) |
bridge=yes | |
Bridge unknown (4) |
||
Tunnel (5) |
||
Snowshed (6) |
New tag snowshed=yes ?? is required. And proposed. | |
Dam (7) |
bridge=dam ?? | |
Unpaved road surface type (unpavsurf) |
Unknown (-1) |
|
None (0) |
||
Gravel (1) |
||
Dirt (2) |
||
Acquisition technique (acqtech) |
Unknown None Other GPS Orthoimage Orthophoto Vector Data Paper Map Field Completion Raster Data Digital Elevation Model Aerial Photo Raw Imagery Data Computed |
source=gps ?? |
Coverage (metacover) |
Unknown Complete Partial |
Not required |
Creation date (credate) |
YYYYMMDD or YYYYMM or YYYY |
|
Dataset name (datasetnam) |
Province or Territory name |
|
Planimetric accuracy (accuracy) |
||
Provider (provider) |
Other Federal Provincial / Territorial Municipal |
|
Revision date (revdate) |
YYYYMMDD or YYYYMM or YYYY |
|
Standard version (specvers) |
Not required |
Geobase Blocked Passage class
Blocked passage information has not yet been imported into OSM.
Geobase filename: (NRN_provinceName_version_edition_BLKPASSAGE.shp)
Geobase Attribute | Attribute Value | OSM tag |
---|---|---|
created_by=?? source=geobase | ||
Blocked Passage Type (blkpassty) |
Unknown (-1) Permanently Fixed (1) Removable (2) |
|
NID (nid) |
A UUID |
nat_ref=NID |
Road Element NID (roadnid) |
A UUID (The NID of the Road Element on which the point geometry is located) |
Not required |
Acquisition technique (acqtech) |
Unknown None Other GPS Orthoimage Orthophoto Vector Data Paper Map Field Completion Raster Data Digital Elevation Model Aerial Photo Raw Imagery Data Computed |
source=survey |
Coverage (metacover) |
Unknown Complete Partial |
Not required |
Creation date (credate) |
YYYYMMDD or YYYYMM or YYYY |
|
Dataset name (datasetnam) |
Province or Territory name | |
Planimetric accuracy (accuracy) |
||
Provider (provider) |
Other Federal Provincial / Territorial Municipal |
|
Revision date (revdate) |
YYYYMMDD or YYYYMM or YYYY |
|
Standard version (specvers) |
Not required |
Back to GeoBase Import page