Import Borno eHealth Africa Smaller Settlements Workflow

In this wiki we will:

present the general import rules in OSM, then the data itself, the tool set to make the import, and the extra data that will be added as a useful complement.

describe step-by-step a workflow to make the import, with screenshots and tips.

Quick Reference

In case you have read and understood this workflow, you can refresh all steps here.

0. Check you have installed the Todo list and UtilsPlugin2 JOSM plugins, and that you have the remote control and the Download object to new layer enabled.

1. Load the data. Choose a task of the project and load the OSM data and the eHealth Africa data, into 2 different layers.

2. Add road access to all settlements. We will do this in the Data Layer 1. Remember to follow the Highway Tag Africa and the Nigeria roads tagging schema. Correct roads already in OSM when needed. Tag bridges and fords when river crossings, and create some meters of river/stream when it isn't mapped yet, with seasonal=yes when seasonal (most cases). And never forget to add the surface=* tag to all roads.

3. Make sure you finished creating/modifying/deleting roads and other objects before proceeding to the next step, as you aren't allowed to add this kind of objects during the importing steps

4. Upload these roads to OSM, using your regular OSM account, with the following changeset tags:

comment=eHealth Africa Borno Smaller Settlements import, #hotosm-project-937
source=Bing or source=MapBox or source=Bing;MapBox, depending the imagery provider/s you've used.

5. Merge the layers. Merge the eHealth_Africa_Smaller_Settlements_BO.osm layer with the Data Layer 1 layer.

6. Add eHealth nodes to the Todo list. With the new filter, select all eHealth nodes and add them to the Todo list. Unselect all nodes, disable the new filter and double-click on the first node of the Todo list. Zoom out a little to have a better view of the settlement surroundings.

7. Check the location of the node. Normally nodes shouldn't be moved. If in the middle of nowhere, delete it and report its name in the TM task commment box.

8. Check the spelling of the node.

9. Two nodes over the same settlement. If names are different, set one as name and the other as alt_name and merge both nodes into one. In case the names are the same, merge them into one.

10. Set the type of place. Set the place=* to its appropiate value, and delete the fixme=* tag.

11. Merge the node with OSM data. If there is a GNS node around, there are 3 scenarios:

a) If the name of the GNS node is the same than the eHealth one, we will add the GNS source to the source tag of the eHealth Africa node (source=ehealthafrica.org;GNS), delete the GNS:dsg_code, GNS:dsg_string , fixme and is_in tags of the GNS node and then merge the GNS node with the eHealth Africa one, so we keep the node history. We suggest doing this with the Replace Geometry tool of the UtilsPlugin2 JOSM plugin (shortcut Ctrl+Shift+G)

b) If the GNS node name is similar to the eHealth counterpart, we will change the GNS name=* tag to alt_name=* (example alt_name=Sansani and the source=GNS to source:alt_name=GNS, delete the GNS:dsg_code, GNS:dsg_string , fixme and is_in tags of the GNS node, and then merge the GNS and eHealth nodes into one (Ctrl+Shift+G).

c) In case the name of the GNS node is too different from the eHealth one, we will just delete the GNS node. The limit on when the name is similar enough to add it as an alternative name or different enough to dismiss it, it's difficult to define. In case of doubt, we suggest to proceed as in the preceeding point, and leave a comment on the comments box when you save the task in the Tasking Manager, so it will be carefully reviewed during the validation process.

GNS nodes that are far enough from any of the eHealth Africa nodes that we are importing and that are at the same time over or near a settlement or group of small hamlets, will be left where they are. They would be eventually replaced in future imports of eHealth Africa or other provider's data, or by anyone's own survey.

12. Mark the node in the Todo list. If there are more nodes still to check, go to step #7. Else, proceed to next step (#13).

13. Upload (import) the nodes to OSM, using your special OSM account, with the following changeset tags:

14. Switch back to your regular OSM account now, to avoid using it for purposes other than importing

Need some inspiration?

You can get some further insight on road tracing and classification from some of the project tasks. We would recommend checking Task #11 for the mountainous Southern Borno - with seasonal rivers all around -, and Task #74 for the Northern barren and semidesertic areas of the state.

Presentation

Import rules in OSM

The data from eHealth Africa, considering its high accuracy, is to be imported into OSM and consequently, needs to follow the guidelines of any to-be-imported dataset.

The dataset has been analyzed, the attributes have been converted into tags, presented and validated to the imports mailing list.

Any import must be done through a specific OSM user account. So the advice is to create a new OSM user account (eg the usual name with _import at the end) from a secondary email address.

Only the eHealth Africa data will be uploaded using this account, so any other edit must be uploaded with the everyday OSM account. As the import is the opportunity to add objects around it, the proposed workflow will integrate this to make the whole job easy and efficient.

The eHealth Africa data for North Nigeria

These data has been originally made through field surveys conducted by tens of field collectors during the last 2 years within the 10 Northernmost states of Nigeria (Kebbi, Sokoto, Katsina, Zamfara, Kano, Kaduna, Bauchi, Jigawa, Yobe and Borno).

During the import we will put changeset tags that indicates the source. Nonetheless, it is useful to keep the source tag for each object as the changeset information is not forcefully got from any OSM extract tool and eHealth Africa could be disappointed not to be mentioned for this data + the metadata can disappear easily once the information is shared and shared, and the source can be a very useful information for people who will work with these data.

An import through the Tasking Manager

Rather than a single, blind import, it has been decided to import the data with a control check of each object. An specific HOT Tasking Manager project has been created for this purpose.

They all share the same grid and a specific feature: they propose to load in JOSM the extract of the data to be imported over the area of each task (square). Consequently, the normal OSM layer is loaded from the Tasking Manager + one layer containing only the eHealth Africa data for that task.

Other data to be added

Although not part of the import itself, we will use this import to trace the roads that access each of the imported settlements, so we will improve the road coverage for Borno state in OSM. For the classification of the roads, it's very important that you read and understand very well the roads section in the Nigeria Wikiproject page, that is mainly based in the Highway Tag Africa wiki with some small adaptations to the Nigerian particularities.

You can also improve the existing roads accuracy and their tagging at the same time.

Workflow

The workflow will show:

first the common steps to start

next, we will show some tips on how to add road access to all settlements, with your regular OSM account

last, we will show the final step about how to check and import the eHealth Africa place nodes, with your special OSM import account

Load the data

Open JOSM. Before you do anything, check that you have the remote control enabled and the Download object to new layer box selected in the JOSM preferences:

We also have to install the following JOSM plugins (if we don't have them yet in our JOSM): Todo list and UtilsPlugin2.

In case you needed to change any of these preferences, restart JOSM. Load Bing imagery and, if needed, MapBox imagery too.

Go to the import TM project, choose a task, click on Start mapping...

and then click on Edit with JOSM button...

Once the OSM data is loaded, you need to load the eHealth Africa settlement nodes into a second layer. To do so, click on here in the sentence Download the eHealth Africa settlement nodes into JOSM by clicking here in the Extra instructions:

A new eHealth_Africa_Smaller_Settlements_BO.osm layer will be loaded in JOSM, containing the settlement nodes to be imported for this task:

Preparing the import

The import will consist of 2 main steps:

1. Adding road access to all the settlements that will be imported, and uploading them with your regular OSM account.

2. Importing the eHealth Africa nodes: Checking the correctness of the nodes (spelling, presence of duplicates, etc.), choosing the correct place type (village, hamlet, etc.), merging them with the OSM data (specially GNS nodes) and uploading them with your special OSM account.

Adding road access to all the settlements

This is the step that is not actually part of the import, but that we consider essential for this HOT activation, as it will make a big difference in the state of the map for Borno.

Before starting tracing roads, you have to:

a) Read and understand well the Highway Tag Africa wiki (if you haven't done it yet). All of it. Specially, pay attention about the restricted use of highway=track and highway=path. You will find some hamlets/isolated_dwelings that are accesible only through a path, but except for these situations, you will be tracing highway=unclassified and up.

b) Read and understand the (small) adaptation of the Highway Tag Africa wiki for the particularities of Nigeria. In general, you won't have almost any chance on tracing any new trunk nor primary roads, as they are mostly completed for Borno state (they are the A (trunk) and F (federal) roads). Most of roads will be either unclassified or tertiary roads. Most secondary roads will be eventually edited in another job out of the TM. As a rule of thumb, tertiary roads generally interconnect wards (municipalities), and unclassified roads are lesser roads that give access to villages and hamlets (normally with connecting purposes).

We will trace roads in the OSM data layer (Data Layer 1 if we have our JOSM localized in English), never in the eHealth_Africa_Smaller_Settlements_BO.osm layer!

If we find roads that have an incorrect geometry or wrong tags, we will also correct them.

Another particularity, specially for North Borno State, is that many roads are composed by several paths going in parallel. You can see this [example]. When this happens, we will just trace one way only: in some cases we can trace the most used of all tracks, and in other cases we will just map a way going through the middle of all tracks.

As usual, pay attention for river crossings, tagging bridges and fords accordingly. It the stream/river is not mapped yet, you can just map some meters up and downstream from the crossing. Tag it as natural=river or natural=stream. The huge majority of rivers and streams are seasonal, so please add the seasonal=yes tag.

And please, don't forget to add the surface=* to all roads!. The huge majority of all unclassified and tertiary roads in this area are not paved, so either surface=unpaved or surface=ground will be appropiate. Remember that unclassified/tertiary roads and up, are assumed to be paved by default.

Note that we have the majority of Borno State covered by Bing and/or MapBox imagery.

This step will be by far the one that will take longer.

Remember that you aren't allowed to create/correct/delete or any other object in the next step of importing the eHealth Africa nodes. So any road or other object that you want to add or touch, make sure you do it now before you proceed further!

Once we have finished tracing roads and other objects, we will upload the changes using our regular OSM account.

To do so, first make sure you have the Data Layer 1 active:

Next, check that you have your regular OSM account user in the preferences (F12):

Finally, you can click on upload. If you get some warnings/errors correct them, and then upload using the following changeset tags:

comment=eHealth Africa Borno Smaller Settlements import, #hotosm-project-937
source=Bing or source=MapBox or source=Bing;MapBox, depending the imagery provider/s you've used.

Importing the eHealth Africa nodes

WARNING!

This is the actual import step. So we won't touch (create/modify/delete...) any object except for the settlement nodes (both the eHealth and the OSM).

At the end of this section we explain what to do in case you commit that mistake.

Now it's time to work with the eHealth settlement nodes, to prepare them for importing.

The firts thing we will do is to merge the eHealth_Africa_Smaller_Settlements_BO.osm layer with the OSM data layer (Data Layer 1). Make the eHealth_Africa_Smaller_Settlements_BO.osm layer active, click on the merge button...

... and then choose the Data Layer 1 as target layer:

We will normally have many nodes for each single task, so we will use the Todo List JOSM plugin to avoid leaving any unchecked node behind.

To add the nodes to the Todo List, the best way is filtering the eHealth nodes out. We can do that using the filter new:

To add the nodes to the Todo List, first select all nodes...

... click on the Add button of the Todo List...

... click on any point of the editing area to unselect all those nodes and disable the new filter:

Double-clicking on the first node of the Todo list (in our case Nguluri) will automatically zoom in to that settlement node:

Zoom out a bit, so we can easily check about the presence of other place nodes around (specially the GNS ones), and also to see how big the residential area is, changing the place=unknown to its appropiate value.

This zooming out is not only useful for those purposes, but in some cases it may show that a node apparently out of any settlement is actually over one settlement that moved between two dates. In the next example, we show that Sabon Sara hamlet was located on 5th February 2012 to the North of its location just one year after, on 7th March 2013 (those dates are the imagery dates for different zoom levels):

Sabon Sara on 5th February 2012

Sabon Sara on 7th March 2013

Now, we will follow these steps:

1. Check the location of the node. The location of the eHealth nodes is generally correct. So, unless we have strong reasons to move the node, we will leave it in its original position. One example when it could be interesting to move a node is the following (Golgwlama village):

It may be moved to a more centric position, beside the road.

In very rare cases (0.5% - 1%), a node is by error in the middle of nowhere, far from any settlement. In those cases, we will set a fixme=Location approximate on the node, we will write down the name of the node and we will write a remark about this in the task comments box. We will do the same when the imagery is cloudy and we can't double-check that the position is correct (also very uncommon).

2. Check the spelling. We will check if the name of the place is spelled correctly and respect the cartographic writing conventions. With very few exceptions, each word of the place name must have its initial letter in capital and subsequent letters non-capital, like for example Kurje, Unguwar Abdu and Gwarmai Cikin Gari. And we will keep a space before and after a slash "/", and before the opening of parenthesis and after its closing.

In case of doubt, we won't make any change to the name and we will leave a remark on the comments box of the task when we mark it as done in the Tasking Manager.

3. How to handle two nodes over the same settlement. In quite rare cases (1% - 2%) we may find 2 nodes over the same settlement.

If names for each node are different, they are alternate names for the settlement. We will put one of the name as name=* and the other as alt_name=*.

If both nodes have the same name, we will just merge the nodes into one node, and place it in the centre of the settlement.

You can see 3 screenshots of duplicated nodes for one settlement:

4. Setting the type of place. All 4,556 settlement nodes for this import come from two different files: RuralVillage.osm and Villageplacename.osm.

RuralVillage.osm consists of 2,241 nodes. Roughly 62% of them fall under place=hamlet, 33% are place=village, 3% place=isolated_dwelling and 2% place=neighbourhood (inside towns/cities).

Villageplacename.osm consists of 2,346 nodes. These settlements are generally smaller than the RuralVillage ones. Approximately 84% of them fall under place=hamlet, 11% are place=isolated_dwelling, 4% place=village and 1% place=neighbourhood

Note: we got these numbers checking 100 random nodes for each file.

For this reason, we couldn't set a fixed place=* tag for all the nodes, but had to tag all of them with a generic place=unknown. So we will check the size of the settlement for each node and decide what type of place is the most appropiate: village, hamlet, isolated_dwelling or neighbourhood. To help a bit more, we set two different fixme=* tags, depending the file that the node comes from: fixme=Bigger places set. Please, set place tag to hamlet/village/isolated_dwelling/neighbourhood if it comes from the RuralVillage.osm file, and fixme=Smaller places set. Please, set place tag to hamlet/village/isolated_dwelling/neighbourhood if it comes from the Villageplacename.osm file.

What we have to do is to change the place=unknown to the most appropiate value and remove the fixme=* tag.

To help on choosing the place type, we remember here what those types of places mean and some screenshots taken from settlements of the import area. Note in any case that choosing one type or another can be sometimes controversial.

place=village: A smaller distinct settlement, smaller than a town with few facilities available with people traveling to nearby towns to access these. Populations of villages vary widely in different territories but will nearly always be less than 10,000 people, often a lot less.

Examples:

place=hamlet: A smaller rural community typically with less than 100-200 inhabitants, few or no infrastructure. Most of the settlements for this import are hamlets.

Examples:

place=isolated_dwelling: Smallest kind of human settlement. No more than 2 households. . Note that in Africa, many huts aren't households, but used for storing stuff, so you can see an isolated dwelling that is actually composed of several huts.

Examples:

place=neighbourhood: A named part of a place=village, a place=town or a place=city. Smaller than place=suburb and place=quarter. Easy to classify.

Examples:

5. Merging with OSM data. Most of the place nodes we will encounter in the OSM database are from the GNS 2009 import.

In 2009 a huge set of GNS data was imported into OSM, an import that is now assessed as poor. In that dataset we have many duplicated, misspelled and badly located nodes, having many nodes being located in the middle of nowhere or just in the non-inhabited area between several settlements. Worse, many of the nodes don't have any relation to the village where they are located (if they are located over a settlement), nor any of the villages nearby. And, except for the name, we aren't interested in the other GNS tags, that shouldn't be imported to OSM in the first place.

In case of finding a place node already in the database and quite near the eHealth Africa node (in almost all of the cases it will be a 2009 GNS imported node), we will proceed the following way:

If the name of the GNS node is the same than the eHealth one, we will add the GNS source to the source tag of the eHealth Africa node (source=ehealthafrica.org;GNS), delete the GNS:dsg_code, GNS:dsg_string , fixme and is_in tags of the GNS node and then merge the GNS node with the eHealth Africa one, so we keep the node history. We suggest doing this with the Replace Geometry tool of the UtilsPlugin2 JOSM plugin (shortcut Ctrl+Shift+G)

If the GNS node name is similar to the eHealth counterpart, we will change the GNS name=* tag to alt_name=* (example alt_name=Sansani and the source=GNS to source:alt_name=GNS, delete the GNS:dsg_code, GNS:dsg_string , fixme and is_in tags of the GNS node, and then merge the GNS and eHealth nodes into one (Ctrl+Shift+G).

In case the name of the GNS node is too different from the eHealth one, we will just delete the GNS node. The limit on when the name is similar enough to add it as an alternative name or different enough to dismiss it, it's difficult to define. In case of doubt, we suggest to proceed as in the preceeding point, and leave a comment on the comments box when you save the task in the Tasking Manager, so it will be carefully reviewed during the validation process.

GNS nodes that are far enough from any of the eHealth Africa nodes that we are importing and that are at the same time over or near a settlement or group of small hamlets, will be left where they are. They would be eventually replaced in future imports of eHealth Africa or other provider's data, or by anyone's own survey.

About the GNS nodes inside big towns and cities, we will proceed similarly as we do in rural areas. The general goal is to merge the GNS nodes with the eHealth Africa nodes.

After these 5 steps, this node is ready for import, so you can go to the next settlement node by clicking on the Mark button of the Todo list:

After finishing with all nodes, you are ready to import them to OSM. As we will import these nodes with the special OSM import account, we need to change the user in the preferences (F12):

Finally, you can click on upload. If you get some warnings/errors correct them and then upload using the following changeset tags:

WARNING!

If you see that in the list of objects to upload (see last screenshot) there is anything different from settlement nodes, then that means that you have created/modified/deleted other objects. Sorry for that, but in this case the safest is to proceed the following way:

1. Delete the Data Layer 1

2. Go to the Tasking Manager and load again the OSM data and the eHealth nodes in two layers as before.

3. Go directly to this section of Importing the eHealth Nodes and start again with the checking of the settlement nodes, setting of the place type, merging with GNS, etc.

Good! Now go to the Tasking Manager, write any issues you might have encountered, and mark the task as done.

To avoid using the special OSM import account (example: johndoe_import) for other non-import changesets, go now to the JOSM Preferences (F12) and switch back to your regular OSM account (example: johndoe).

If you missed adding some roads or other objects, you can do it now, as you did in the Adding roads section, and using the same changesets tags at the end of that section.

If you are happy with what you did, you are now ready to start another task!