Talk:OSM Mobile Binary Protocol

Existing protocols

See also the rather flexible design of OpenDMTP (Open Device Monitoring and Tracking Protocol) protocol for minimizing the traffic when communicating to OpenGTS (Open GPS Tracking System) --Stefanb 15:41, 27 September 2007 (BST)

Interesting this takes things to a lower level, tcp/udp packets, maybe for a future binary api... Also they scale lat/lon values to fill the integer range, giving higher accuracy than the 6dp in the suggested binary api, but only claim it to be good to 5dp perhaps you get more rounding errors? --Adamboardman 20:15, 28 September 2007 (BST)

Diff

Stefanb: How compact is the osmosis diff format? You indicated that it could just be adopted as a binary protocol?

It is bloated XML (*.osc), and should be somehow close to .osm file (in terms of tags...), so same principles could be employed for shrinking the diff. Grab an example from [1] --Stefanb 16:05, 28 September 2007 (BST)

Ok have improved diff description, osmosis probably not particularly useful as we work from OSM API not planet files. But yes a binary diff is definitely something we need. Perhaps all we need to add for the output is a list of deleted poi's and deleted ways (any mentioned will be added). Though more complicated things could be done in terms of only sending changed parts of nodes/ways etc. --Adamboardman 19:56, 28 September 2007 (BST)

bmap.php

Erik: In revision 807 (and earlier) I have a problem. The area which I request (tile=436829705 ts=29; tile=1747318821 ts=31) is not complete. My client is receiving half of the requested area. After changing the value $remaining=1024 to $remaining=4096 I receive the complete tile.

Not something I've noticed, that value should only configure the chunk size for downloading content, so it would indicate that there are issues around adding two chunks together? --Adamboardman 14:24, 12 October 2007 (BST)

Latest bmap.php (now only 0.5 API is supported) has an extra check that removes a lot more nodes taking the tile you mention from: 49421 to 17093 --Adamboardman 15:04, 12 October 2007 (BST)

I don't understand...

Hi, I'm trying to implement this but I find the spec a bit confusing. Quote:

To enable noticing deleted items, ways that have been omitted due to use of the have parameter are listed in an omitted field listing just the id's.

Node - length n id lon lat nodepropenumval {propkeyid (enumval | valsignedint | packedtime | strlen str)}

BlnBnBidBBlonBlatE
BlnBnBidBBlonBlatEILString Of Text
BlnBn...

What are all those Bs? Double Bs? E? EIL? BlnBwBidBBnuBBloBBatB...IEILA9? Literals? Or are the ln, w, id, ... literals? Neither of them are? --SLi 00:05, 20 October 2007 (BST)

I've added further details to the spec, you may find it easier to look at the php example code (and php.net/pack to find out what the coding letters mean), but B merely indicated Binary data, and the the idea was to occupy 4 bytes so 'Binary Block Length' => 'BlnB'. The IEILA9 you query is 'Property Key Id Highway (I), Enum value for primary(E), Property Key Id Ref (I), Length (L), A9 (utf8 text)'. Hope that helps. For interest what are you working on? --Adamboardman 16:47, 21 October 2007 (BST)

Analysis : Viability of Mobile protocol

The alternative to a mobile protocol is a "download (partial) planet to flash" approach.

Which is best ? The important questions are :

• Which is more up to date and does it matter ? Mobile protocol will be just as up to date as the database, while your flash disk will be on average 1 week older if you download it every 14 days. The OSM contributors will take several years (decades ?) to (re)visit all the important roads, so the OSM database will on average be at least 1 year behind the physical world. So it that extra week (or month) it takes for the data to become available for flash download doesn't really matter.
• Which is cheaper ? Downloading 1MB with a mobile handset(GPRS / EDGE / 3G) has always been more expensive than downloading it with ADSL AND writing it to flash.

So IMO, although a mobile protocol will represent an extraordinary achievement, it will not be extremely useful. -- Nic 22:11, 21 October 2007 (BST)

• In the limited (connected) devices you don't always have the room for whole (and growing) planet, or even the subset that one might someday be interested in.
• Even if you do have plenty of room or free bandwidth, it's probably wise to compress it efficiently - as this binary protocol does.
• Even with unlimited GSM data plans or broadband it probably is desired to have the needed data downloaded as fast as possible (imagine driving and waiting for the new section of the map to appear, or waiting for the maps in 100km radius to be downloaded so that you can do your routing).
• With compressed (=cheap) downloads you can also easily delete whatever you didn't need recently to save space, or do frequent updates on the road (more convenient than thinking about it in advance at home every week).

--Stefanb 09:51, 22 October 2007 (BST)

Reduced Data OSM

A discussion started by bvh on #osm suggesting removal of ways got me thinking about the best way to store the minimum data to represent what we want. Whilst I dont agree with bvh I think there is room for improvement in this area.

Stored data:

• Nodes - lon+lat only used for POI's with meta data
• Ways - directional list of lon+lat meta data only if the way is not shared with any other concepts (to avoid db changes main db would keep a list of node ids)
• Relations - adding meta data to groups of ways

Representing reality:

• POI - one node entered
• Road - one relation giving name/ref + links to ways that make up that road
• Park - one relation giving name + links to ways that border it (some ways would be only used by this park, some would be shared with neighboring roads or footpaths
• Footpath in countryside - one way with meta data
• Bus route - one relation giving number + links to other relations/ways that make up the route

Editor interface:

• POI - click on poi to see its properties
• Linear - click on a section of it to see a list of what it is used for, user is protected from details of relation/way data minimisation
• Area - clicking on part of its linear perimeter will list it as part of what that segment is used for, then clicking on the area item will highlight all parts of that area
• Items list - list of items related to currently selected object
• Properties list - edit properties for current items list item

Automated migration:

• If two ways are tagged ref=A7 that share and end point then editors should automatically remove the ref tag from both ways, create a relation to represent the common feature
• If a park boundary uses the same nodes as a road for part of its edge then the editor would cut out that section from both the road way, and the park way and then create relevant relations for both the park and the road.

Backwards compatibility:

• From an editor point of view backwards compatibility could be maintained by keeping any tags applied to a relation also applied to the ways it connects (eg ref=A7). The mobile API would probably need to filter out tags on ways already mentioned in relations.
• From a renderer point of view they would have to use a BC layer that would manufacture additional ways for each relation that would otherwise mean losing information, probably most important would be areas.

Prerequisites: Full support for relations (currently unsupported by many editors/renderer's including WhereAmI/PHPProxy)

PHPProxy(osmgetbmap.php) now supports relations, and has the ability to filter out properties applied to linked relations from ways/nodes, currently disabled until WhereAmI supports relations (other users could enable it for their usage though) --Adamboardman 09:43, 29 December 2007 (UTC)

protocol comments

• this is my first "talk" page edit -- be kind if i screw something up, please. --Pgf

• i don't see any version number referenced in the protocol. did i miss it somewhere?

Nope don't have any yet... with adding things at the end of enums and not doing things like you request below we'd not need them... --Adamboardman 19:07, 15 November 2007 (UTC)

• while the data is stream-oriented, i still think there's value in presenting friendly alignment for consumers of the data. for example, putting a one-byte 'type' field in front of a string of longs means that there's no chance that the data can be referenced in-place -- it will almost always have to be copied. i think all fields should have 4 byte alignment -- there should probably be more room left for growth in some of the one-byte fields, anyway, and some of the one-byte fields could be combined -- two 2-byte fields in a 4-byte space, for instance.

That would be more of a storage format, and feel free to parse it and save it with padding, but why pay for the extra bytes if you dont have to, were talking x pounds per meg downloads on mobile phones here! The cost you mention is minimal assuming the data is in RAM on the device, the other end of the process of writing the data to a db in persistent flash in the device, now that is costly. --Adamboardman 19:07, 15 November 2007 (UTC)

• what's the BwiB entity? it's not defined in the list. i'm guessing this is "way id". but isn't it simply the same as "BidB"?

An omission, no particular need but just indicated that it was a list of id's relating to ways --Adamboardman 19:07, 15 November 2007 (UTC)

Servers?

Is there a URL for getting recent data in this format? Ojw 18:21, 18 December 2007 (UTC)

The next release of WhereAmI (v0.08) will be offering this click able tile selector: http://www.symbianos.org/post.php?nav=osmtileselector (don't click too many tiles unless you intend to use them - even then importing more than 400k of tiles in one go may not be possible depending on phone memory) --Adamboardman 09:27, 29 December 2007 (UTC)

Error or feature?

Im working with this protocol. All was fine but some days ago it has changed. The length of UTF8 strings(i.e. ways names) become incorrect as i understand. In english all fine but in other languages there are some problems.

In english language string with 10 characters occupies 10 bytes in UTF8. But i.e. in russian language string with 10 characters may occupy 15 bytes in UTF8. BUT!!! The length of the ways name, which i receive, is 10(not 15). And i think this is error. I repeat - couple of weeks ago all was fine. Name of UTF8 was send count of bytes(15) and now it sends count of characters(10).

That why even ur program whereami works incorrect with russian names.

I hope u will fix it.

Thank u very much, big respect for ur job.

Vadim. vkassin@mail.ru

ps. sorry of my english.

This should be fixed now, also new server url obf.symbianos.org/osmgetbmap.php --Adamboardman 23:27, 18 October 2008 (UTC)

Lon/Lat representation

I would prefer to use the whole range of the 32bit integer to store lat/lon. It would be some kind of binary coded angle -> (0xffffffff + 1) = 360deg. The multiplication factor is as good as the current one because 10000 also gives no speed improvement and you get the advantage that invalid angles are impossible. In my opinion it makes no sense to use floating point values for storing angles, you just loose precision the bigger the angle gets. Also you never need nanometer precision at zero degrees...

That would be a different format, this format wishes to maintain forwards and backwards compatibility as much as possible. The * 1000000 was chosen as its considered to be sufficiently accurate for the purpose at hand and because its human readable without the use of a calculator for debugging purposes. Elements of the main OSM database and newer binary forms tend to use your suggested method, had they happened first or someone suggested it before initial deployment... --Adamboardman 15:06, 9 January 2009 (UTC)

software using this format

I would like to add a section "usage" to this page. Does anyone have an overview of the programs that support this protocol as a client? --MarcusWolschon 12:00, 6 January 2009 (UTC)