Importing Government Data
The intent of this page is to share the in the discussion of Importing Government Data
This page needs to be tidied up. discuss
_-_The_Noun_Project.svg){kind=small}
See Import/Guidelines and Automated Edits code of conduct for more information. Imports/automated edits which do not follow these guidelines might be reverted!
Importing Government data
So the big questions on everyones mind are these;
- If the government information is available for free, how do we use the data?
- What state is this data available in?
- A static 1-time copy?
- A WMS feed?
- How do we use the data?
- How can this import update what is already on the map?
- How do we deal with updates?
- What is the significance of this data?
Summary
Here is a summary of the current options, this also applies to the more generic page import.
- Create a full map of the country, making it as an openlayer.
- Have this layer overlayed on top of the OpenStreetMap slippy map that we work on, (just as we
currently have yahoo imagery)
- Since this map overlay has high resolution imagery available, it is also part of this slippy map.
- Trace over roads, and place markers or draw shapes where we like.
- Access to the most current dataset as actual shape files, and can import them as we see fit.
- Show the map background WMS feed as an openlayer. It gets updated on a yearly basis.
Learning from others
The GeoBase (Canada) /Tiger (USA) /AND (The Netherlands)/LINZ (New Zealand) all have common threads: dealing with government data.
Corine
The OSM Import Database idea is directly derrived from the Corine Import using the http://clc.openstreetmap.fr/cgi-bin/index.py website, is a great example a way to hand the ever changing source database. And lets the community import the data as they like.
GeoBase
- GeoBase is designed as a complete map of Canada, with the primary purpose of integrating all levels of government and all government agencies at all levels. So that, for city planning, and Parks Canada and boundaries and features are all known to everyone at each level of government.
Have access to the full Atlas of Canada, created by Natural Resources Canada (NRCan) where the information is available as a WMS feed. It is technically possible to have this whole map (with everything on it) shown as an OpenLayer, as a slippy map inwhich users can trace, as well as import specific map features, as well as import bulk data.
See GeoBase Import
Tiger
- Tiger (USA)
Project Summary:
Method:
Conclusions:
AND
- AND (The Netherlands)
Project Summary: (Although this was not a government import, the method still applies.)
Method:
Conclusions:
LINZ
- LINZ (New Zealand)
Project Summary: See http://wiki.openstreetmap.org/wiki/LINZ_Trial_Import
Method: The method is a VERY good example of the community approach.
- Members of the OSM community in New Zealand hosted a LINZ Import meet-up (in person) and discussed the process.
- There is a list of Key people on the wiki (those who are the main contacts & would be the best to get information from
- The Data was in the form of .mp format. So to convert it, they created a conversion tool with python
- Then converted the mp files, and made them available.
http://joerichards.dev.openstreetmap.org/files.html
- There is also a link to download the shp files directly from the source provider
- Download as shapefiles from koordinates.com http://koordinates.com/maps/linz/
- there is a todo list which includes
- the creation of as designated API http://linz.dev.openstreetmap.org/api (the new zealand LINZ import dev server), it isn't working yet, but when it does, you can point JOSM at it & download the data
- The wiki needs to have all the issues in the current import fleshed out and addressed LINZ_Trial_Import
- agree on an approach for tagging (e.g. the link to the attribution page is quite long, we could do with a tinyurl)
- find out if the upcoming license change (to odbl) will affect the use of the data
- ask linz if our attribution method is sufficient
Conclusions:
--acrosscanadatrails 21:52, 29 March 2010 (UTC)(Also a WMS layer could be set up) so people can trace. --acrosscanadatrails 21:52, 29 March 2010 (UTC) This community is a 'OpenGIS' discussion list, which combines the General OSGeo users with the OSM Community, rather than 2 separate discussion lists for the same region. nzopengis@googlegroups.com
Naga
- OSM_Philippines_Data_Import From the Naga city Government
Using gml2osm to convert the files to usable for OSM Project Summary: Method: Conclusions:
plan.at
An example of how an Import should not be: very bad accuracy due to improper coord-transformations, lots of MapSpam (thousands, if not millions of orphaned nodes), no cooperation with local mappers....
Project Summary: Method: Conclusions:
In contrast, OpenStreetMap is a project which is created at the public level, where this free editable map of the world all contributions and technical know-how is delivered by volunteers, where with working together -in person, on wiki, on discussion lists, on message boards, on blogs and more. Through this collaborative effort, known as 'OpenSource' we can/are create a pretty awesome map.
Government Maps
The first map, we have an entity where it is useful only for government employees at every level, so to make decisions based on government sourced data. Slowly, this information will become editable by all levels of government in real time.
After all, the true value of any map, is in the date-time stamp, along with its accuracy to show the real world. Information like 'lot numbers and official building plan numbers & and extra detail on roads - this is only needed for government use.
In contrast, having all this information on a public use map, is not needed. It just takes up space. If city planners were to use it, they would enter in that information themselves (on there free time)
Update Frequency
So we have this contrast: 2 separate maps, both being updated in (for government almost) in real-time, and on a planet scale, government maps are updated all the time. OpenStreetMap: The free map at the public level, which is also updated in real-time. This map is updated at a pace which is directly proportional to the technical knowhow and the volunteer contributions of the map users.
Government vs. Public Technical knowhow
On the Government side, there is a vast about of funds available to hire staff to create / maintain & update the maps. This technical knowhow is based on the ability for the hiring managers to find those technically literate people, who can make the project work.
On the Public Technical knowhow side, we have whats known as 'croudsourcing' and 'OpenSource' software, where through with community involvement, and sharing codes, and ideas. From both the back-end of the software (java programming for example) and from the fron-end, where users submit their comments and sujestions for improvement. The program.. such as JOSM gets better and better with each new release. As the release itself is available as open source, so those people who know how to fix problems can do so. Using the Wiki approach, problems (such as GeoBase Import)can be solved, where as, the different users, are contributing their own ideas. With communicating to various people who have all different backgrounds, can all share their ideas and insights.
The profit Aspect
The products to make the maps are purchase/ leased from actual (for-profit) companies. So the maps, which are produced on a local level, there is no incentive for these agencies to give away for free to anyone (openstreetmap community), the most current data available. .. After-all, when someone wants to adjust property lines (for example) they would need to pay the government so to re-map that area after it has been approved. In contrast, if that information was provided publically, someone can just as easily go onto openstreetmap, and make that adjustment.
Updates of Government data
Because the maps that the government(s) use, are constantly changing. What is done, is release versions are made, along with update packages. These update packages, only contain the list of information which as been; Modified from original, Removed from original and added to the original.
Modified
This means that if the original import was fully complete, and posted to OpenStreetMap. The Shapes\ways\ points that were imported would need to be either; Removed and replaced, or a clever piece of software (plugin) would be able to detect what that NID was, and be able to receive the modifications, and adjust the tags and location to show the modifications.
Removed
This means that if the orginal import was complete, and posted to openstreetmap; These shapes\ways\points would need to have some kind of way to tag what was imported with a note saying 'these needs to be removed' and a clever script to be able to physically remove these items.
Added
This would be the easiest, as the import script would not need to care about the previously imported data, however, when it is importing something in which a OSM user has already placed that same map feature, it would be a duplicate. So this clever script would need to be able to detect existing identical map features
Summary: Options
Areas of high OSM work
For areas of a great deal OSM Community work, there is little need to be importing everything that is available to import. So the best course of action would be to
A - Do nothing, and keep up the great mapping work
B - Use the Slippy map layer to find things which have not already been added to openstreetmap.
C - Go through the list of available feature downloads (extracts), and just import those features that you would like to see on the map.
Areas of Medium OSM work
For areas of Medium OSM work, these would be where 50-100% of the roads have been mapped, and many features have already been added, but of course, there are many features which can be added. For these areas, a great deal of care needs to be taken. The best course of actions would be to;
A - Do nothing, and keep up the great mapping work.
B - Use the slippy map layer to find things not already on OSM and trace
C - go through the download list and download features that you want to see.
D - Also download map features that do exist (say parks) but you would not mind manually doing through the map and adjusting those dublicated parks. care needs to be given so to not make too many duplicates that you cannot handle. So the best course of action would just be to import 1 feature type at a time, and in the smallest area at a time.
Areas of Low OSM work
For the area of Low OSM work 25-50% OSM Coverage
A - Do nothing, and keep mapping (but since it's available, why not import?
B - Use the slippy map layer and trace
C - go through and download the features you want to see
D - download features that do exist, but you don't mind the manual adjusting work. .. still great care is needed to avoid duplicates
E - For these areas, there is the option of remove and replace. .. however, it is not the preferred method. (it would really be up to the local mapper, if they want to do that.
Areas of little to none
For area do little to no OSM work 0-25% OSM Coverage
A - Do nothing (it would be advised to import, as it's available)
B - use slippy map trace (but since there's very little, its better to..
C - go through and download features you see (but since its so much, better too..
D - download all the map features
E - the option of remove and replace might be better, to avoid the manual work.
F - make sure you don't download the placenames (if done) and the land tracing already done.
Summary: Dis-approved options
Blanket Import
- Blow away data and replace en masse
- Slap on imported data & create 'ghost lines' with existing data
- Import data as nodes with reference to what shape/line/point it is, keeping original source designations
Why?
- We want to protect the existing osm data, and keep it as a
higher priority
- We don't want to fill up the database with duplicate nodes
Summary: Pending options
-coordinate locally, copy select elements with a custom script layer
-use wms layer (rendered openlayer slippy map and trace
-write protect imported data (nodes? If a host can be found to donate resource?
Summary: Approved
- tracing over josm wireframe
- tracing over imagery
- tracing over free openlayers
- import in empty areas
Next steps Government import
1 - Complete the charts showing the comparisons of the data to be imported, and how it should be tagged. (these are country specific, but should be used for reference)
- Canada see GeoBase Import
2 - Complete the script (and work off of what has already been created) to make the process of converting the government data to OSM, easier) See Government Data Import Script
3 - Use OpenLayers to host all the available government data (from WMS Feeds). For areas in which users would prefer to trace, than to have more work on fixing)
3 - Agree within the community on the best courses of action See talk discussion lists
How OSM can to deal with updates
As was mentioned with the update frequency, and Updates of government data, this is a tricky issue. We need to keep in mind the following
- OSM is a free and editable map of the world, and therefore, it is not confined to actually use the updates that are available.
- If the update is available in the form of a WMS Openlayer, then users can trace over the map, for the areas not done (just as we currently do with Yahoo imagery)
- If it's possible to use OSM Inspector where these updates can be available on a list. What this would mean, is;
- That the Government Imported data would not empedede on the natural growth of the OpenStreetMap project
The OpenStreetMap advantage
Because OpenStreetMap is not limited to the types of features that can be used, it is possible to create things which the government cannot. For example;
- WikiTravel's guide maps are created by picking out certain information that are useful for a travel map.
- CycleMap is created by looking at those features in which are useful for those on a bicycle, so showing features such as Bike Parking (racks), and where bicycle lanes are, and where bicycle route are.. as well as contour lines, are all available.
Back to GeoBase