CHSEGURA import
About
The import process refers to hydrography dataset available on the administration of the Segura Hydrographic Confederacción. Segura basin is on the South-East quadrant of Spain
Among the hydrographic data available on the Confederacción web, we have chosen only those that are useful and have equivalence with entities supported in OpenStreetMap, i.e. the layer of rivers, streams and canals (waterway=river and waterway=stream) and reservoirs layer landuse=reservoir.
Target
Our target is to add OpenStreetMap data supplied by this government agency which are not in OSM database yet. In case they exist in OSM but only partially (eg. rivers not completely mapped), merge current OSM data with data from Confederacción, giving precedence to what is currently in OSM over the new data.
The following images show a comparison between current data in OpenStreetMap (information obtained from Overpass_API, with those that will be added with the import. Only the main hydrographic network (rivers and streams) is compared and the improvement is noticeable.
Import Data
Links
Data Source site: C.H.Segura
Data Licence: Legal Notice
Type of license: CC BY 3.0
ODbL Compliance verified: Yes
It has been asked by email about permission to import the data, in order to ensure license compatibility with ODbl. The answer form Confederación Hidrográfica del Sagura is reproduced below:
Received: by 10.59.8.229 with HTTP; Sun, 16 Feb 2014 23:30:50 -0800 (PST) Date: Mon, 17 Feb 2014 08:30:50 +0100 Delivered-To: taborda.barroso@gmail.com Message-ID: <CAGwCOQOVsjEn4ans3szMoYWEYa+UaSK6jOunxZQawWzfBko9Uw@mail.gmail.com> Subject: Compatibilidad de licencia CC By con ODBL From: =Matias_Taborda_Barroso?= <taborda.barroso@gmail.com> To: segura@chsegura.es Hola, buenos días. Participo en un proyecto, desarrollado a través de Internet, llamado OpenStreepMap.org [1]. La idea es crear un mapa que esté disponible libremente y que pueda ser utilziado por todo el mundo, al estilo de la Wikipedia. Me gustaría preguntarles si podemos utilizar los datos cartográficos disponibles en su portal, en formato SHP, evidentemente haciendo caso a su Aviso Legal y siempre citando su origen, para incorporar esos datos al mapa de Openstreetmap. Por lo que puedo leer en su portal, sus datos estarían liberados bajo licencia CC By 3.0. En un principio, puede parecer que no debería haber ningún problema en la utilización de estos datos, pero la gente que trabaja en el mantenimiento de la base de datos de OSM es muy cuidadosa con el tema de las licencias de datos y me ha pedido que pregunte expresamente si se podrían utilizar esos datos y liberarlos bajo la licencia de Openstreetmap, es decir, una Licencia ODbL [2], lo que significa que : (A) todo el mundo puede compartir (es decir, copiar, distribuir y usar para cualquier propósito) los datos, (B) crear nuevos trabajos a partir de los datos, y (C) adaptar, modificar, transformar y construir sobre esos datos, pero (A) indicando su procedencia, (B) liberando los datos derivados bajo esa misma licencia, y (C) manteniendo los datos abiertos y accesibles para todo el mundo. En otras palabras, se utilizarían sus datos junto con los ya existentes en un gran proyecto conjunto para agregar valor a los datos. La licencia ODbL es equivalente a la licencia Creative Commons Atribución-Compartir bajo la misma licencia que usa el proyecto Wikipedia [3], sólo que es específica para datos. Muchos de los datos que figuran en su mapa ya están incluidos en OSM, pero sin duda, con su colaboración, lograríamos enriquecer el mapa. Atentamente, Matías Taborda. [1] http://www.openstreepmap.org [2] http://opendatacommons.org/licenses/odbl/summary/ [3] http://creativecommons.org/licenses/by-sa/3.0/es/ ---------------------------------------------------------------------- Disposition-Notification-To: "Vera Gomis Jose Antonio" <joseantonio.vera@chsegura.es> Subject: RE: Compatibilidad de licencia CC By con ODBL Date: Tue, 18 Feb 2014 14:17:20 +0100 From: "Vera Gomis Jose Antonio" <joseantonio.vera@chsegura.es> To: <taborda.barroso@gmail.com> Entiendo que por nuestra parte no debería haber ningún problema, pero no tengo claro que tenemos que hacer desde aquí. Solo debemos indicarlo en nuestra web?. Hay que inscribirse en algún sitio? Un saludo ################################################ José Antonio Vera Gomis Jefe de Servicio de Sistemas de Información Oficina de Planificación Hidrológica Confederación Hidrográfica del Segura Ministerio de Agricultura, Alimentación y Medio Ambiente Plaza de Fontes 1, 30001 Murcia Tlf. +34 968 35 88 90 / +34 968 965 753 Fax +34 968 21 46 05 joseantonio.vera@chsegura.es http://www.chsegura.es <http://www.chsegura.es/>
OSM Data Files Example
- Reservoir File. [1]
Import Type
Import will be done by humans (no bots) loading small files (up to 1.5 MB) in JOSM. Manual merging with already existing data will be done, giving priority to what is currently in OSM database and checking possible errors with the aid of aerial photography (Bing, PNOA).
Processing the Data
Data available at Confederación website are in SHP format and ED50 datum, Zone 30N (EPSG:23030), so it will be necessary to reproject the data and transform it to .osm format. Two layers will be used: Hydrographic network (Rivers and streams) and Reservoirs. The treatment will be similar for both of them, but we will separate their processing.
Hydrographic network
Hydrographic network shp file contains some values that need to be parsed and other ones that won't be used, as they are of no interest for OpenStreetMap.
Preprocessing
QGIS software has been used for preprocessing. Using the tools included in QGIS the following tasks have been achieved:
- Fixing spelling errors: e.g.: Rio --- Río
- Main name capitalization, according with Toponymy Guidelines from Instituto Geográfico Nacional. (e.g.: ARROYO DE LOS ENDRINALES ---- Arroyo de los Endrinales)
- Adding a new field (TYPE), to be used later for tags assignation, differencing between rivers (waterway=river) and streams (waterway=stream).
- Using the above field, two layers with streams and rivers entities are obtained separately. This step is also used for reprojection, saving the layers in datum EPSG: 4326.
- A polygon layer from Confederación website is loaded, which divides the basin in 32 subareas following administrative criteria from the Hydrologic Plan of the basin. These subareas are not sub-basins, but they serve for the purpose. Subareas layer.
- Intersection of lines hydrographic network layers (rivers and streams) and polygon subareas layer.
- Multilayer division taking as split field the identifier of the subarea.
Tagging
Among the different fields in the original .shp file, only the name has been used, as numerical identifiers don't add any useful information. The field TYPE has been added to differentiate stationary waterways (streams) and permanent ones (rivers) and so have them ready for next step.
SHP ATTRIBUTE | OSM TAG | DESCRIPCTION |
---|---|---|
OBJECTID | Not used. | Numerical identifier used by C.H.Segura. |
NOMBRE | name=* | Name of the waterway |
SHAPE_len | Not used. | Entity lenght. |
NIVEL | Not used. | Numerical category for internal CHS classification. |
TIPO | waterway=river, waterway=stream | Field added with QGIS. Type of waterway. |
Finally, according with import guidelines, the following tags about the import will be used in each Changeset:
- source=* = Confederacción Hidrográfica del Segura.
- source:license=* = CC-By-3.0
- source:date=* = 201311
- import = yes
Conversion to osm format
shp-to-osm.jar has been used to convert the files from previous steps to osm format. version 0.7.3 has been used, as the one on the wiki, 0.8.2, doesn't seem to work.
Rules file to convert dbf fields to osm tags is as follows:
line,NOMBRE,,name,- line,NOMBRE,N.C.,, line,TIPO,Rio,waterway,river line,TIPO,Arroyo,waterway,stream
Data simplification
Before importing the data, way simplification has been achieved, using GPSBABEL, with the following command:
gpsbabel -i osm -f file.osm -x simplify,crosstrack,error=0.001k -o osm -F file-F.osm
Hydrographic network Import Process
Import will be done manually using JOSM. Process will be as follows:
- Open *.osm file for a given area.
- Depending on the file, there will be streams or rivers of different length. One of them will be selected and zoomed [3] on it.
- OpenStreetMap imagery layer is activated. This way a quick check is easily done to see if the given waterway is already in OSM database, total or partially.
- If it is partially in OSM, the area is downloaded from OSM in a separate layer. The object is then selected in the osm file to be imported and merged with downloaded layer (Ctrl+Shift+M) and deleted from the original file.
- It is then checked if the way is correctly traced, using aerial imagery (PNOA, Bing).
- Ways downloaded from OSM and obtained from the import are connected to complete the waterway.
- Process will be similar in cases where a given waterway is not in OSM but the main waterway it is connected to is already in OSM database. Corresponding nodes will have to be merged.
- Changeset will be tagged as described above.
- Finally data is uploaded to OSM.
Postprocessing
Once data is imported, a check is done to detect places where the waterway crossed other ways: roads, tracks, etc. Bridges/tunnels are created where necessary and uploaded to OSM with a regular user account.
Reservoirs
The available reservoir layer, as the one of rivers, is in shp format. It is a polygon layer, representing a total amount of 36 reservoirs of the basin.
Preprocessing
QGIS has been used for preprocessing in the same way than for hydrographic network layer. Using QGIS Field calculator the following edits have been done:
- Spelling normalization of the field containing reservoir names. (e.g.: EMBALSE DE LA RISCA ----> Embalse de la Risca).
- Reproject the layer to EPSG:4326.
Tagging
Only the field containing reservoir names has been used, as the rest are considered useless for OpenStreetMap.
SHP ATTRIBUTE | OSM TAG | DESCRIPTION |
---|---|---|
OBJECTID | Not used | Object numerical identifier. |
NOMBRE | Not used | It contains the name, but field NOMBRE_COM is more complete. |
ID_CHS | Not used | Internal C.H.S. numerical identifier. |
NOMBRE_COM | name=* | Full name of the reservoir. |
SHAPE_Leng | Not used | Reservoir boundary length. |
SHAPE_Area | Not used | Area occupied by the reservoir surface. |
USO | Not used | Use of the water contained in the reservoir. |
According to guidelines from discusión the following tags are added:
Changesets are tagged as follows :
- source=* = Confederacción Hidrográfica del Segura.
- source:license=* = CC-By-3.0
- source:date=* = 201311
- import = yes
Conversion to osm format
Conversion to osm format is done with version 0.7.3 of Shp-to-osm.jar.
Rules file to convert dbf fields to osm tags is as follows:
outer,NOMBRE_COM,,name,- outer,NOMBRE_COM,,natural,water outer,NOMBRE_COM,,water,reservoir
Reservoir Import Process
Import process for reservoirs will be similar to the one described for hydrographic network, but as the number of entities is smaller (36) it is not necessary to split the file and the work will be done directly on the file obtained in previous transformation.
- Open .osm file with JOSM.
- Openstreetmap imagery layer is loaded.
- A reservoir is selected and zoomed on it [3].
- It is checked whether the reservoir is already in OSM or not, using OpenStreetMap imagery layer. If it is in OSM database, then is removed from the .osm file and the next object is processed.
- If it doesn't exist in OSM, the area is downloaded in a new layer and the reservoir is merged with it (Ctrl+Shift+M). Nodes of the waterway that feeds the reservoir are connected where appropriate. Note that if hydrographic network has been previously imported, all reservoirs will have their corresponding waterway.
- The quality of data is assessed activating some aerial photography layer (PNOA, Bing)
- Finally data is uploaded. Import tags are added to the changeset.
- Above steps are repeated for the next reservoir.
User account
import_chsegura user account, will be used for the import.