JOSM/Plugins/Wikipedia/GSoC 2018
Over the course of the Google Summer of Code 2018, the JOSM plugin for Wikipedia got extended, especially with regards to Wikidata. On this page the progress of this project is documented.
Work product
Here is a quick overview over my work on the Wikipedia plugin for JOSM over the course of Google Summer of Code 2018. A more detailed summary can be found below in the #Timeline section, for a git changelog listing each and every commit see the before-after comparison on GitLab (and the same comparison for the gradle-josm-plugin).
One of my first steps was to move the project to Git (hosted on GitHub, since July also on GitLab) to make collaboration and working on different features at once easier.
The build system was switched to Gradle, using the gradle-josm-plugin (which I created and maintained already before GSoc), which encapsulates common build logic and especially in combination with git reduces the setup required by developers to git clone the code and run Gradle (only requirement to be installed is Java).
Translations are now managed in Transifex instead of Launchpad, since Launchpad is regularly slow (see 
#8645).
The first coding period had the main theme of adding validation rules for wikipedia=* and wikidata=*:
- check if the item in wikidata=* is linked to the article in wikipedia=*
- check if the item in wikidata=* actually exists (not only that the format is valid)
- if wikipedia=* contains a redirect, suggest the page to which the redirect leads
To implement these validator checks I generalized the classes that create and execute API queries, so the class hierarchy can be easily extended to cover additional APIs or other queries. Also there is now an abstract class for validation tests, that makes it easier to create new rules that make network requests, it deals with splitting the list of objects into chunks and then testing each chunk separately.
Another important improvement was the new website https://josm.github.io/wikipedia that allows to always get the latest development version of the JOSM plugins wikipedia and pt_assistant. In the meantime this has moved to https://josm.gitlab.io/wikipedia where now only the wikipedia plugin is served.
In the second coding phase I implemented the wiki layer that displays Wikidata items and Wikipedia articles, with clustering of markers where there are many of them. Also I cleaned up the interface (buttons in Wikipedia dialog, merge properties panels).
During the third coding phase, I added a Wikidata info panel that displays labels, statements and sitelinks for each Wikidata item that you select (either an OSM object with wikidata=* or an item from the Wikipedia panel). Also I added a new validator check, that checks for Wikidata items that have an unusual type for an OSM object.
Timeline
Community bonding
April 23rd — April 29th
- mail to the josm-dev mailing list
- create project JOSM/wikipedia on GitHub as a git-mirror of https://svn.openstreetmap.org/applications/editors/josm/plugins/wikipedia
- add Gradle build
// e1bbc2c d111008 - fix unit test that only succeeded if i18n language was English
// a495b53
(all commits to the repository this week)
April 30th — May 6th
May 7th — May 13th
Coding Ⅰ
May 14th — May 20th
- the plugin is now translated on Transifex
- migration to GitHub complete: source code repository, CI builds, code coverage, release v1.0.0
- add JOSM plugin update site that installs the current state of the master branch to your JOSM instance: https://josm.github.io/wikipedia/
- add validator check that the item from the wikidata=* tag really exists. If the item is a redirect, an autofix is offered.
(all commits to the repository this week)
May 21th — May 27th
- Implement validation check: Validate wikipedia=* with respect to wikidata=* #12219
// f443e9a d945333 - Release version
v1.0.1containing the new validation checks and updated translations from Transifex - Create JOSM update site, from which you can always get the current development state of the plugins wikipedia and pt_assistant
(all commits to the repository this week)
May 28th — June 3rd
- test the API queries with WireMock
// d388cc3 eaedcd0 - Generalize the classes that make API requests to allow POST requests with the query not in the query string but in the request body (doing batch queries as GET requests could soon reach URL length limits)
// 90525b0 71ba157 - for the gradle-josm-plugin:
- try to get required JOSM plugins from the Nexus repo, currently not possible due to this Gradle issue (or the corresponding JOSM ticket #16347)
// db4d9f7 d413efd - fix a long standing bug, where the JOSM preferences were always reset between two invocations of
./gradlew runJosm.// 74cc640
- try to get required JOSM plugins from the Nexus repo, currently not possible due to this Gradle issue (or the corresponding JOSM ticket
(all commits to the JOSM-wikipedia repository this week)
(changes to gradle-josm-plugin this week)
June 4th — June 10th
- add sitematrix query, that checks against the Wikidata API, which Wikimedia sites exist
- add caching for API queries (at the moment only used for the sitematrix)
// e2cd12c 136205e - opened tickets #16373 (pre-fill component field of bug report) and #16374 (don't always advise to disable plugin) for the bug report process
- adapt gradle-josm-plugin to changes in Gradle 4.8, use jGit to determine plugin version, #movingtogitlab. Versions
v0.4.8was released// d413efd...c3e7fdc
First evaluation
Coding Ⅱ
June 11th — June 17th
- make the ApiQueryClient more generic (for future reuse with other APIs than the Wikidata Action API)
// b0bc2d0 5eebcb4 - move the source directories to directories in Maven/Gradle style (
src/main/java)// 1bdde16 - create
WikipediaSitefor easier usage of the sitematrix query// 963dce4 1d48578 - add API query to find Wikipedia articles that redirect (will become a validator test later)
// dcde48f 4e22aa0 - add WikiLayer, that displays Wikipedia articles and Wikidata items
// eb6a0e4 - add Notification when download of Wiki elements fails (e.g. area too big)
// 0b374eb - combine the download buttons of the WikipediaToggleDialog into one single button
// 18f2414
June 18th — June 24th
- fix the Jenkins build, that was broken by moving the source directories and using WireMock 2.18.0 for unit tests
// 23f42a7 - add validator check for Wikipedia redirects
// cdd972f fedfa8a - gradle-josm-plugin: Run CI also with Java 10, move the remaining tasks from Java to Kotlin
// a5f7307...88e7e42
June 25th — July 1st
- cluster the markers of the wiki layer
// 43e7c34 - highlight selected element in wiki layer
// fb534a8 - move the settings to one common tab in the JOSM preferences
// 07bda9d
July 2nd — July 8th
- move preferences (Sophox API URL and Wikipedia language) to one common preferences panel
//
07bda9d - switch from CircleCI to GitLab for continuous integration
// d8ecbe4 - add API query to find instances of a Wikidata "class" from a given list of Wikidata items (using Wikidata Query Service)
// 1cad5e6 - add validator check that uses the API query for unusual Wikidata "classes"
// 5659dca - use the correct API names in logging messages of the
ApiQueryClient(which is reused for the several APIs)// 4430bf7 d4b09e9 - fix repaint issue of the preferences panel when switching expert mode on or off
// 83ff387
(all commits to the repository this week)
Second evaluation
Coding Ⅲ
July 9th — July 15th
- fix the validator check for unusual types
// 6c4ff3d
(all commits to the repository this week)
July 16th — July 22th
- add API query to get all labels for a Wikidata item
// 6ac45e9 - reorganize API queries to allow them to return a different type than the root type of the deserialization schema
// f4d4ed4
(all commits to the repository this week)
July 23rd — July 29th
- add API query to get all claims for a Wikidata item 252efeb
- add new dialog, which will display info about the selected Wikidata item
// 99ae410
(all commits to the repository this week)
July 30th — August 5th
- improve defaults for the "unusual types" validator check
// 12dcd29 - fixes for i18n strings
// 3490226 0edc4dd - pull new translations from Transifex
// f30bdf5 - put Docker images in gitlab.com/JOSM/docker-library where they are auto-updated and can be reused across projects
// bd70707
(all commits to the repository this week)
August 6th — August 14th: Code submission & Final evaluation
- fix
- fix
- #15597: Add Wikidata Toolkit to plugin classpath, use Ivy to get the dependencies when building with Ant 902f222
- show sitelinks for selected Wikidata item 9bb3e5a 268dee7
- show statements for selected Wikidata item d0bf68b
- show labels for selected Wikidata item adeaa86
(all commits to the repository this week)
Ideas for the GSoC project (from before it started)
This list contains some ideas for improvements over the course of the GSoC.
If you have ideas not on this list that you would like to see implemented, you are very welcome to suggest them On the Talk page.
Validation
- Validate "type" (highway, building, tree, …) of objects with wikidata=* against P31 (instance of) on Wikidata
- make suggestions to use subkeys instead (e.g. brand:wikidata=* when wikidata=Q-ID of some brand)
- Validate that the Wikidata item used in wikidata=* on an OSM object has coordinates #11541
- Validate that the coordinates of the OSM object and the Wikidata item are close (relative to the size of the object: say, 5km for a city, 10m for a small building)
- check that the website=* matches the one on the Wikidata item
- check that IDs in OSM match those on Wikidata (uic_ref=* — 954, ref:bag=* — 981, ref:whc=* — 757, …)
Visualisation
- Add Wiki layer
- show Wikidata items
- show images from Commons #8472
- Show aliases, description and properties from Wikidata in JOSM #9886 #14518
- filter Wikidata items by type (tumuli, runestones, events, things that no longer exist, …), as suggested on the talk page