Mechanical Edits/Peter Elderson

Replace note=* with ref=* in node network routes in Nederland and Germany

Dutch mappers have decided to replace note=* with ref=* for tagging the references of node2node route relations in recreational node networks. See OSM-forum discussion (Dutch) at https://forum.openstreetmap.org/viewtopic.php?id=69373

A few objections addressed the issue of it being a mechanical edit. This document aims to meet the requirements for a selective and regional mechanical edit.

When no further objections (show stoppers) arise, the operation will begin on 2020-06-08.

I will start with walking node networks in Nederland, which is the bullk of the work, and has been started by many mappers already. This saves them the painstaking work. Others may use the method for cycling node networks in Nederland, other types of node networks in Nederland, or node networks in other countries.

Before 2020-06-08, many mappers were already doing smaller scale edits for the same purpose. No action has been be taken to stop this in Nederland; we expected mappers to spare themselves the trouble and to simply wait for the mechanical edit.

Progress

2020-06-09 Peter Elderson All walking node network routes in Nederland now use ref=* for the route reference. The note tag is now available for notes, conform wiki documentation. (talk) 16:13, 9 June 2020 (UTC) 2020-06-10 Started conversion of note->ref for cycle node networks in Nederland. Peter Elderson (talk)

2020-06-09 22:30 Finished! All cycle node networks in Nederland now use ref for the node2node route references. Left2do: check with overpass if I missed some. Then use knooppuntnet to see if any errors have been introduced.

2020-06-11 I have asked on the OSM-forum Germany if they want to convert as well, and offered to help. So far 4 of 4 reactions were positive. If no objections arise, I will start this work on 2020-06-15.
2020-06-15 I have looked at some of the cycle node networks. The action as described on this mechanical edit page is simply not possible, because the tagging is not consistent enough. The node numbers are in the name tag, the description tag, the note tag, and/or the ref tag, sometimes in three tags at the same time so very redundant; often mixed with other information and special punctuation. Often there is also note:de and there can also be other information in the description and note fields. Sometimes the way this is done is consistent within a network, but sometimes it differs even within the network. I have posted this on the German OSM-forum.

2020-06-11 I have asked on the riot forum OpenStreetMap Belgium if they want to convert as well, and offered to help. So far, one reluctant reaction, no positives, no rejections. I will of course not convert Belgian node network routes without a positive consensus of the Belgian community.

Rationale

The decision in the past to use the note field was not conform the wiki documentation of the note tag. Notes are meant for comments about an object from mapper to mapper. The routes are referred to as mm-nn where mm and nn are the reference numbers of the two interconnected nodes. The wiki documentation of the ref tag does encompass this kind of reference. In other countries where node networks are deployed, mappers have preferred the ref field from the start, though some have used note=mm-nn to comply with the Dutch!

This operation will correct the faulty use of note=* for node networks in Nederland, thereby enabling mappers to use the note tag for real notes.

Impact

• Since ref was actually the preferred tag and note was the anomaly, tools and applications already support the ref tag for this purpose.
• A minor change in JOSM was performed to show ref=* instead of note=* in a style for editing walking networks.
• One cycle map renderer reported that actual notes could appear on their map, if that happens a lot then a style change would be necessary.
• OsmAnd never supported note=* for walking routes but always supported ref=*. This change now makes the route-references visible on the map.
• Osm's cycling layer never showed note=* but always shows ref=*. This change makes the route-references visible on the map.
• Waymarkedtrails did not show notes for the routes; the routes were displayed in the side panel as a list of object id's. When ref is used instead of note, a list ordered by reference is shown.
• Wiki pages on node network tagging are being adapted.

Why mass edit?

To avoid massive mixed use of ref and note for actual notes and for the route references, the operation needs to be done in a relatively short time span, say 3 months. There are a huge amount of these small routes. It cannot possibly be done by hand without any kind of repeated or mechanical edit. I have tried, and found that it would take years to edit them all one by one in small changesets.

Mea culpa

I became a human bot, and still it took me several hours to get one small network done. I did the routes one by one so I did not think it was a mechanical edit. Then I tested some tricks to speed it up. I downloaded a network of say 200 small routes, did a manual check for possible problems, saved it locally as a .osm file, and edited that in a text processor to replace 'note' by 'ref', then reloaded the file into JOSM and checked the result in the list of relations. Then searched for possible problems such as a missing ref=* tag. Then uploaded. But then, though everything had been individually checked, it had become a mass edit. A few mappers had warned me and they were right. In hindsight, I was wrong to do it like that without proper documentation.

Rather than reverting the edits which will eventually be done anyway, I am now documenting the action. Note that other mappers had already started manually editing walking node networks and other types of node network, e.g. bicycle networks, before I started. Walking node networks, however, contain by far the most routes.

Description of the method; workflow

Most of it is still manual, just the tag replace is done on a selection in one edit.

Only the ref and note tags of node network route relations are affected by the mass modification.

1. Fetch all node2node routes of one named node network in Nederland into JOSM, using the remote control option in https://www.knooppuntnet.nl/nl/networks/nl/rwn (for walking) or one of the other transport modes. Or use an overpass query and feed the result to JOSM.

2. Use search to find all routes having note=* AND ref=*. Check the list visually to make certain ref and note contain the same value. If not, check what's in them and resolve the issue. Then remove the note tag.

The following steps ensure that there are no problems with the note and ref fields that would stand in the way of the tag replacement.

3. Save JOSM to an .osm file

4. Outside JOSM, use the python script* provided by JeroenHoek to detect any remaining problems, and solve them in JOSM. Then save again and recheck.

Now do the actual conversion:

5a. Option a: within JOSM. Select all the relations with the reference number in the note tag, and having no ref tag. In the attributes window, doubleclick on the note tag. Replace note with ref and click OK. Now the ref tag holds the values previously stored in the note tag.<br>

5b. Option b: outside JOSM using the python script*. Once the check is faultless, run the script for conversion. Then clear JOSM and open the converted .osm file. <br>

6. Check the entire list of relations again for anomalies. (I have set JOSM to display ref, note, name in the relations list).

7. Make sure nothing is selected (click an empty space on the map). Then upload to OSM as one changeset. Refer to this page as the source of your mechanical edit.

8. Check the result e.g. with https://www.knooppuntnet.nl/nl/networks/nl/rwn (for walking), achavi or OSMcha

9. Next network!

The whole thing takes 5 - 15 minutes, depending mainly on how many actual notes are in the note field, and on the length of the list as it has to be completely checked visually.

A walking node network can vary from 10 to over 1500 routes. The analyser tells me: There are 120 networks, with a total of 20800 nodes and 28360 routes

So it's still a lot of work, but I do not want to perform a fully automated edit.

* Python script for step 4

JeroenHoek has provided a python script to perform a check, then if no problems are detected replace note with ref. Any problems need to be resolved in JOSM until the check finds no more problems, before the conversion is run. The script works for all node networks.

See https://github.com/jdhoek/osm-note-to-ref

To use this script, python3.x.x needs to be installed on your PC. Make a work directory where you put the .osm fie and the .py script. Perform the check and/or convert from your linux or Windows command prompt as indicated in the README file.

After running the convert, load the resulting .osm in a cleared JOSM and upload the changes. Please refer to this page as the source of your mechanical edit.