Overpass API/GDPR
With regard to the GDPR, the OSMF developed the policy that several categories of data related to the user accounts on osm.org are personal data. From the data processed in Overpass API this affects the association of a user name, a user id, and a changeset id to a node, way, or relation.
In line with this OSMF policy, the next version of Overpass API will grant access to these objects only to users with an account on osm.org. To avoid piling up data from the other direction, this is implemented in a way such that Overpass API only knows that a query stems from a known user and not who.
A prototype is now available. At this point of the development feedback is highly appreciated and will be used to improve the final version.
For Users
In the final implementation, you need to get an API key (string of 40 hex digits) from osm.org for Overpass API. You then store the returned key and add it to a special field of your query.
All requests not involving the aforementioned user meta data continue to work without API keys. In particular, you can still download data from Overpass API for editing (e.g. into JOSM) without API key.
The current prototype pair
As the feature is not yet implemented on osm.org, please go to the mapper login of the prototype. I do stand behind the policy that you should not enter your osm.org credentials elsewhere, thus you need an account on the master test instance to login to the prototype.
Once you are logged in to the prototype then you can request an API key per service. Please use dev.overpass-api.de for the tests.
Please then go to your favourite Overpass API tool, e.g. Overpass Turbo. You need to use HTTPS, the prototype does not run on HTTP and the final instance will not either.
Make sure that your API endpoint points towards the test instance https://dev.overpass-api.de/api_keys_test/. In Overpass Turbo you find the setting in Settings > General Settings > Server; insert it without https:// please.
You then can try the following three example requests:
This one should work fine. It returns only version numbers and timestamps on elements, no user metadata:
nwr[name="Aldgate"];
out geom meta;
This one is forbidden because it asks for user metadata without an API key:
nwr[name="Aldgate"];
out geom attribution;
This is the fixed variant of the previous. You must replace APIKEY with your API key from the key dispenser prototype:
[api_key:"APIKEY"];
nwr[name="Aldgate"];
out geom attribution;
The prototype only has data for a sample around London that is not updated. This is essentially due to disk space constraints.
API details
Query interface
There are two ways to add API keys:
- You can prepend your query with [api_key:"APIKEY"].
- You can add an URL parameter apikey= (no quotation marks) to your query.
The query language is affected in two ways:
- out meta is pruned back such that it is still good for editing but does not reveal personal data. out attribution takes its place.
- out attribution, the user and uid filters, and the user(), uid(), and changeset() evaluators trigger that the query engine insists for an API key.
All other parts of the query language remain unchanged.
The query engine syncs its API key database whenever it is queried with an unknown key. This behaviour may get more sophisticated in future versions. But it ensures now that API keys are conveyed perciveably immediate.
API key dispenser
POST /apikeys/login_mapper
POST /apikeys/login_serviceStart a session for the mapper's portal resp. the service mainteance portal. Is expected to be replaced by /login in the productive version. These API keys return on success a suitable website.
The following API calls all need a session, a user id, and a service name parameter:
POST /apikeys/announce_serviceReserves the given service name for use by the logged-in user. The service name must not be already reserved for someone. Returns a session key for the newly created service.
POST /apikeys/make_apikeyCreates an API key for the given service. It is allowed for anyone, not only the service owner. As a side effect, revokes all existing keys for the given user for the given service. Returns the newly created API key.
POST /apikeys/make_servicekeyCreates a new service key for the given service. As a side effect, revokes all existing service keys for the service. Returns the newly created API key.
The following API call needs a service name, the corresponding service key, and an integer for the start of the delta:
GET /apikeys/retrieve_keysReturns for the given service all keys that have changed since the event numbered by the beyond parameter. The service can get the next fitting delta by using the returned event counter.
The following API calls need a parameter for the user id and an existing key:
POST /apikeys/renew_apikeyReturns on success a new API key that is valid for the same service. Does not revoke the existing key.
POST /apikeys/revoke_apikeyRevokes the given API key.
POST /apikeys/revoke_servicekeyRevokes the given service key.
Installation instructions
The source code is available from the branch api_keys of [1]. The Installation has not been smoothened yet, and the best is to wait for it to settle in the next release.
If you want nonetheless: Read both the short and the long instructions.
You need to have libcurl installed, e.g. from package libcurl4-openssl-dev. Please add the parameter --enable-curl to configure.
You need to setup your own service base URL: Use the service portal of the prototype to setup such an URL.
Make a file api_keys.source.url in the database directory with the content https://olbricht.nrw/apikeys/retrieve_keys?service=BASEURL&key=SERVICEKEY&beyond=0 where you need to replace BASURL with your service base URL and SERVICEKEY with the received key. Downoad a temporary XML file from that URL, e.g. api_keys.xml. Then replace DBDIR in the following and run
cat api_keys.xml | bin/update_database --api-keys --db-dir=DBDIR
Finally, you need to start an extra dispatcher along with the one or two you have to start anyway, with parameter --api-keys.