Import/Catalogue/AlberiMonumentali-RAFVG-opendata

From OpenStreetMap Wiki
Jump to navigation Jump to search

About

This page is about importing Alberi Monumentali (natural monument trees) dataset for regione Friuli Venezia Giulia (RAFVG).

In this page we consider dataset downloadable at RAFVG catalogue page fetched searching "Alberi Monumentali".

Dataset shall be adapted in order to generate OSM files suitable to be imported in planet.osm. It shall not be a blind import: source data shall be checked by mappers thru an audit support map.

The import is being discussed on the national OSM mailing list and will be the result of consensus there.

Goals

This import aims to have a complete and updated set of trees denoted as "natural monument" for the italian territory. Survey took in consideration historic, ecological and dimensional values. This import has been set on a regional size, as a pilot for other italian regions (admin_level=4).

Schedule

First import will be performed after community audit on shared json preview files. Audit progress will be trackable in project page. Import limited size should take 10-30 days to be accomplished.

Import Data

Background

Source dataset contains <200 punctual objects (mostly provided by "Regione FVG") which quality is good and spatially accurate. Other sources have less or basic details for tagging. During a pre-audit, few minor spatial errors has been detected (<20m).

Resource

Dataset is presented as ESRI shapefiles compressed zip in two proections; UTM 33N has been choosen.

Metadata

  • Language: ita
  • Date: 24/04/2018
  • Subject: RAFVG - Direzione generale - Servizio paesaggio e biodiversità
  • Role: originator
  • Web site: www.regione.fvg.it
  • Classification: unclassified
  • Other constraints: public

Record format and tagging plan

RAFVG dataset table structure will be adapted and pruned thru OpenRefine.

RAFVG Alberi Monumentali - record format
Field Name Description:it Description:en Example tagged as
1 numero_sch codice univoco scheda albero unique tree code 01/D014/GO/06 ref:mipaaft
2 genere genere genus Larix genus(redundant)
3 specie specie species Larix decidua Mill. taxon
4 nome_volga nome volgare common name Larice species:it
5 circonfere circonferenza (cm) circumference (cm) 340 circumference/100
6 h_stimata altezza stimata (m) estimated height (m) 24 height+note
7 h_misurata altezza misurata (m) measured height (m) 0 height
8 tipo candidato per candidate for monumentale/notevole denotation=natural_monument/landmark
9 vincolo vincolo protected yes protected=yes
10 data_rilie data rilievo survey date 2016/03/13 2016-03-13

Legal

Import Type

Prior to uploading, the dataset will be prepared and checked by an audit process.

Data Preparation

Source data is presented as shp shape ESRI file in a collection of punctual elements, one for each tree, projection is UTM zone 33N (ETRS89). The following operations has been performed thru QGIS application:

  • reprojection using WGS84 datum
  • creation of lat & lon columns in DD.DDDDD format

Refining

Prior to OSM JSON conversion, some issues require refining operations, documented herein. A summary of actions performed thru OpenRefine:

  • Text to number for geo and dimensional columns
  • Text to date for timeline columns
  • Rename column numero_sch to ref:mipaaft
  • Rename column specie to species
  • Rename column genere to genus
  • Rename column nome_volga to taxon:it
  • Rename column circonfere to circumference
  • Create column height based on column complementary fields h_stimata + h_misurata
  • Create column note if h_stimata has value
  • Rename column fonte to source
  • Rename column data_rilie to survey:date
  • Rename column tipo to denotation
  • Mass edit cells in column denotation
  • Rename column vincolo to protected
  • Mass edit cells in column protected

Exporting

Conflator input requires json format. Dataset conversion to json is performed thru OpenRefine template documented herein.

Further validation (ie: typos detection) on output json files can be performed thru jsonlint (npm -g install jsonlint).

Up to 2.8 version, Openrefine doesn't manage null values; workaround to remove lines containing nulls: 

pi@raspberrypi:~/OSM sed -i -e '/ : null/d' <Openrefine-output-file>.json

Conflation

Conflation is performed by OSM Conflator. Objects tagged ad natural=tree will be extracted from OSM in a bounding box defined by source dataset. Existing OpenStreetMap data within a range is merged and tags will be added/replaced accordingly to conflator parameter file profile.py. Prior to import, 9 objects has been detected in AOI querying landmark or natural_monument trees.

Conflator output example

pi@raspberrypi:~/OSM conflate -i AlberiMonumentali-FVG-csv.json  -o AM.osm -c previewAM.json profile.py
12:02:28 Read 139 items from the dataset
12:02:40 Downloaded 2457 objects from OSM
12:02:42 Matched 14 points
12:02:42 Adding 125 unmatched dataset points
12:02:42 Deleted 0 and retagged 0 unmatched objects from OSM
pi@raspberrypi:~/OSM

Dedicated upload account

The account attilaimport will be used to upload community revised .osm files.

Changeset Tags

Changeset will be tagged with:

Team Approach

Import will be managed by the following OSM users:

  • Cascafico

Workflow

Step by step operations:

  1. dataset download
  2. Qgis reprojection and csv export
  3. OpenRefine operations
  4. OpenRefine json export & null's removal
  5. run conflator
  6. audit map announcement & publication
  7. wait for community validation
  8. conflation re-run
  9. Upload changeset(s) in OSM

In case of import problems, changeset(s) involved will be reverted using proper reverter

Uploaded

changeset objects notes
65116794 177

QA

In case some problems will be detected after upload:

Widespread:

  • TBD

Limited:

  • TBD

Possible improvements

This import has been performed using wikidata reconciliation service to better set "species" tag. KEW specific botanical service should be tested.

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Import/Catalogue/AlberiMonumentali-RAFVG-opendata&oldid=2109948"