ONS Postcode Import

This project dates from 2013.

About

This project outline is intended to help ensure that your Import Plans covers as many of the common questions about imports as possible. Just create your own page and cut and paste the editing information from this page into it.

Please! If you identify ways that this outline didn't meet the needs of your import (key evidence of this: tons of questions or alarm bells on mailing lists!), please return and fix this page.

Goals

Link the postcode information released by the Office of National Statistics with buildings in OpenStreetMap

Schedule

The intention is to run this for the first time when ONS release their February 2013 dataset, not sure whether it is worth running on a regular basis after that.

Import Data

Background

Provide links to your sources.

Data source site: http://www.ons.gov.uk/ons/guide-method/geography/products/postcode-directories/-nspp-/index.html
Data license: http://www.ons.gov.uk/ons/guide-method/geography/beginner-s-guide/licences/index.html
Type of license (if applicable): The ONS postcode directories are subject to the Open Government Licence, and Ordnance Survey OpenData Licence.
Link to permission (if required): N/A
OSM attribution (if required): N/A
ODbL Compliance verified: Yes - http://lists.openstreetmap.org/pipermail/talk-gb/2012-October/013996.html

OSM Data Files

Link to your source data files that you have prepared for the import - e.g. the .osm files you have derived from the data sources.

See Data Transformation Results

Import Type

Initially it will be a one time import, although there may be value in having it as a recurring import whenever ONS release their data. It will be generated from a script. The source of the script will be referenced here when completed.

The data will be output to sets of files in osmChange format with approximately 1000 ways in each. upload.py will then be used to upload all of these using the ONS Postcode Import account

Data Preparation

Data Reduction & Simplification

The following reduction to the data will be done for every import:

Filter the following from the ONS Postcode data:

Postcodes which have a date of termination set
Postcodes whose centroid is shared with other postcodes
Postcodes which have a quality indicator other than "Within the building of the matched address closest to the postcode"

Then match these filtered centroids to ways from Openstreetmap that have the following criteria:

The way is closed
The postcode centroid is inside the way
The way does not already have a "addr:postcode" tag
The way is tagged building=*

Tagging Plans

The only tag being added is the addr:postcode tag on some ways in the UK.

I'm not planning on adding a source tag directly on the ways, the source information will be contained in the Changeset comments.

Changeset Tags

The full URL for the source file will be included in the changeset e.g. As of Feb 2013 the most up to date source of data is: http://data.statistics.gov.uk/ONSGeography/PostcodeProducts/ONSPD/ONSPD_FEB_2013_txt.zip

   <tag k="created_by" v="ONS Postcode Import"/>
   <tag k="comment" v="Bulk import of ONS Postcode data (see http://wiki.openstreetmap.org/wiki/ONS_Postcode_Import) from http://data.statistics.gov.uk/ONSGeography/PostcodeProducts/ONSPD/ONSPD_FEB_2013_txt.zip"/>

Data Transformation

Describe the transformations you'll need to conduct, the tools you're using, and any specific configurations or code that will be used in the transformation.

Preprocessing

Original ONS Postcode File 
   ▼
Filter Terminated Postcodes
   ▼
Filter Postcodes sharing centroid
   ▼
Filter Postcodes with centroid outside of a building
   ▼
Create Filtered ONS Postcode File

Setup

Install local version of Overpass with just UK data

Processing

for each postcode in Filtered ONS Postcode File
 call to local overpass to get ways (and their nodes) in area surrounding postcode centroid, 
 Filter out ways that have no building tag 
 Filter out ways that have a postcode tag
 Filter out open ways (Check if first node equals last node)
 Finally if a way encloses a postcode centroid put it in the output

Code for the above can be found on github: https://github.com/mcgin/osm_postcode

Data Transformation Results

Post a link to your OSM XML files.

In progress

Data Merge Workflow

Team Approach

This is a solo effort

References

List all factors that will be evaluated in the import.

Import has been discussed on Talk-GB mailing list http://lists.openstreetmap.org/pipermail/talk-gb/2013-January/014358.html

Workflow

Detail the steps you'll take during the actual import.

Information to include:

• Step by step instructions: Use upload.py for each generated change file
• Changeset size policy: Approximately 1000 way's per changeset
• Revert plans: Revert changesets

Conflation

Identify your approach to conflation here.

QA

A random selection of way/postcode combinations that are identified in the output will be manually verified

Open to other suggestions.