User:Harry Wood/hack ideas

From OpenStreetMap Wiki
Jump to navigation Jump to search

Harry's ideas for OpenStreetMap hacks.

Where we've attempted to have shared lists of ideas on the wiki, we've always run into problems. One of the problems has been that we've had too many different lists. I appreciate this is yet another list. Apologies for that. So why am I putting this here?

This is a list of ideas which makes sense to me. For each of these, I feel like I personally have a good handle on the problem and would have at least some idea of how to tackle it myself. In some cases they're quite simple things which would make for a good hack day/weekend event activity. They're not over ambitious. They're "doable by Harry" hack ideas. They may also be ideas which I would be reluctant to punt onto a shared list because I think they would be shot down by other developers as duplicitous or stupid for some other reason. Some of them are ideas I've kept secret, because I want to do them, and don't want somebody else doing them. But they're things I've been meaning to do for years so... clearly time to share.

The list is also not complete. I've have many ideas. I just blurted some out from the top of my head.

dev

MediaWiki maps extension

The Slippy Map MediaWiki Extension is a bit pants. I think Grant was looking into swapping in the better maintained Extension:SlippyMap. So... either

A) Help with that. Set up a mediawiki install with that to test it, and customize it. Ask grant. What needs improving? Is it too feature rich for the purposes of this wiki? We probably don't want google maps features for example.

or

B) Make the old extension a bit less pants. I think grant said he's cheffed it, to get deployed from github so I can make little improvements e.g. the credits are positioned too high up. Using frederik's lowzoom map or this one would be a neat trick (good for country level wiki pages)

MediaWiki Captcha

Think of a Mediawiki captcha plugin which supports an audio captcha alternative for blind people. Got it? You're thinking of reCaptcha right? Now imagine one which isn't operated by google, and feeding address data into google maps :-) This is in the operations issue tracker: https://github.com/openstreetmap/operations/issues/19

Update: As mentioned on the issue there, google's newer version of reCaptcha doesn't do the address numbers things any more. Best solution is probably to just update to that (?)

The original reCaptcha has also been systematically circumvented by spammers somehow, so there's wide demand for an alternative.

I actually got a long way through integrating secureimage into a mediawiki plugin. If you ask me to set up a user account for you, you can see it in action, on my test wiki. But it's not working yet. I couldn't understand Mediawiki's weird cookie storage scheme for the right answer. Got a bit stuck. Maybe I should publish that code somewhere.

OpenStreetMappy captcha? This doesn't need to be anything to do with OpenStreetMap. OSM techy types will quickly get waylaid by the enticing idea that we could somehow contribute to OpenStreetMap as part of the captcha. That would be neat, but these things are actually quite complex to design in a robust spam-proofing way... all of which is a distraction from the simple goal of having a captcha which isn't run by google!

...having said that I see Stefan Keller has been working on ReMAPTCHA. Could wrap that as a MediaWiki plugin

Wiki - community index cross-checks

The original idea was to do some improvement work on this tool: http://usergroups.openstreetmap.de , which scrapes some data out of wiki pages where a particular template is used, however...

The system which does well at the user groups list idea these days is the OSM Community Index. The impetus which has lead to better maintenance of lists, is the integration causing it to link user communities within the iD editor on submitting every changeset. So this idea is obsoleted/nailed pretty nicely these days.

However... I still think it would be good if the wiki carried good up-to-date information about this stuff on city pages, plus there may be gaps in the community index data which could be plugged with info from the wiki, and if people do find the wiki info they may find it easier to contribute to. So the idea becomes...

a scraping/matching script which cross-checks and encourages us to add missing information from one to the other. I'd probably suggest being flexible about wiki page layout, and merely search for and check inclusion of links.

Open google charts API

Re-implement google charts maps API as open source to offer a better answer to questions like this. Could use a static map images service. Could use OpenStreetMap boundaries data for the colourable/clickable zones, perhaps via mapit

Temporary floodwater extents collaborative editing platform

An idea for FloodHack

Because floodwater extents are temporary and change over time, the data doesn't really belong in OpenStreetMap, but where does it belong? And could a platform allow people to collaborate on mapping floodwater extents? How could this system handle the temporal aspects. Imagine the ultimate output, a map with a time slider, letting you see where the floodwater was up to, and when. This is quite different proposition to OpenStreetMap style collaboration, and belongs on a different system stored in a separate database. It could take account of satellite and aerial snapshots in time, where the entire flood extent can be mapped at a point in time, but how can crowd contributions be used to fill in the gaps in that picture? e.g. someone knows that the on this particular road, the floodwaters had risen to this particular coordinate, at this particular time. Allow input and storage of that data, and then feed it into the display somehow.

Look at tools like http://www.ushahidi.com which layers data on top of OpenStreetMap in a community contributable way, and on a timeline. Perhaps extend this tool to do community editable polygon areas. Sahana Eden already allows community-editable polygons on top of OpenStreetMap (& other) layers. Also look at OKFN's time mapper.

Open Source Flood modelling

An idea for FloodHack

Modelling floods (and more broadly Hydrology) is a sophisticated area of GIS. People pay good money for proprietary software to do this. General hack idea would be to look at the state of the art within the open source arena. The QGIS community have listed some Open Source hydrology projects. Deploy it for the UK, perhaps using OSM river data. This generally requires a digital elevation model, which OpenStreetMap doesn't offer directly. Where the OSM community have used DEMs extensively already, has been producing relief maps. Often these have used free SRTM data. For a UK focus it may be better to use Ordnance Survey Opendata#Terrain 50.

OpenStreetMap POIs and road data could combine in an interesting way with the outputs of this project. e.g. Which schools/shops/etc get flooded, and which roads close, if river X rises by 10 metres?

Another interesting thing (which has little to do with UK floods) which could help OpenStreetMap would be to model hydrology worldwide, and figure out where the most significant missing rivers are in OpenStreetMap (or create a public domain rivers dataset which we can compare with OpenStreetMap rivers data) For example Hydrological analysis of SRTM data could tell us there's likely to be a river flowing northwestwards here in the Congo (it's not on OpenStreetMap)

BASINS is a hydrology modelling tool developed by the US Environment Agency, largely built with open source technology. I looked at some of the requirements for using OpenStreetMap data with this tool. It was basically feasible, but some other data is needed (from memory, profiles of rivers and other hydrological features). Other data are flow rates, precipitation etc. SK53 (talk) 18:53, 16 February 2014 (UTC)

Note that landuse is a very important parameter for modelling surface runoff in most hydrology models.

Static maps

Fix the staticMapsLite wizard. It includes osmarender. Obviously out of date. Various other obvious fixes needed.

Hi-res rendering for printing

Set up a high-resolution tiles server for double-resolution OpenStreetMap tiles. This is just a config option in Mapnik, so more of a Operations & hosting hack than a development hack. This could be the "standard" style, or perhaps the Humanitarian Style.

The motivation for doing this would be to offer these tiles into tools which make printing dead easy, so that suddenly printing with good dots per inch would magically become dead easy. Tools which make printing dead easy include Field Papers and BigMap. The latter in particular suffers from resolution problems. You can use it to generate beautiful high complexity maps (maps which would overload any vector handling system) and you can make a very big raster image as a way of cranking up the resolution, but only by making the size of things like text labels really tiny.

A double resolution rendering would also be useable on smartphones as a "retina" version of the map I think.

Having said that... for pure printing use cases one nice simplification is that tile server performance is not all that important. If you have to wait a few minutes for BigMap to run, then so be it.

geofabrik offer hi-res images for this purpose, but only via a paid-for WMS.

I mentioned the Humanitarian Style, and this could be a good HOT initiative, because aid organisations in particular seem to like big printouts!

Update: Mapbox are making this work well within Mapbox studio. Likewise MapTiler

downscaling tileserver / many-marker raster renderer

I mentioned frederik's lowzoom tileserver above as a potential mediawiki maps improvement. That server could do with an update, plus there's quite a few other maps I'd like to see available as lowzoom in this way.

Actually I mostly think of this idea when I see tools which show markers, and then the markers disappear or become a bit crap at lower zooms. A raster layer could be used within these tools while at low zooms, and when you zoom in it goes backs to dynamic clickable markers.

  • Keep right has been screaming for this feature for years. Imagine a worldwide raster rendering of Keep Right bugs. Ideally broken into different raster layers I suppose (some of the more rare bugs would be more interesting while zoomed out). Likewise other Quality Assurance tools like Osmose. (ITO World's Maps and OSM Inspector and OSMlint already use this lowzoom trick to good effect)
  • Notes - These still use the slightly weird trick of showing a random selection of markers as you zoom out. Kinda lame I reckon.
  • Wheelmap - Likewise showing a random selection of markers as you zoom out. Does that not bug anyone else?
  • Great British Toilet Map needs this. Currently the web interface just kills the API at low zooms :-)
  • commons-on-osm
  • ...and many more

Some existing tools I know little about:

  • Tiletool - For low-zoom generating from high-zoom (raster only)
  • tippecanoe by mapbox (demos page) does a lot of this, but via vector tiles.

I guess I'd like to have my own script to make it easy to experiment. But maybe also a set of scripts for different server-side languages PHP/rails/python etc to make it easy to bundle into websites in a fairly lightweight way. Ultimately it'd be nice to allow developers to visualise a tonne of marker data without ever needing to know that they're handling raster tiles. Google's fusion tables integration does this in a really nice way. In OSM-land this used to be a key offering of CartoDB.

Wiki content scraping

The wiki is a good platform for allowing the community publish content. Well... sometimes it sucks. It allows everyone to muck in, which is bad for certain kinds of content, but usually good. In any case, there's various situations where keeping things in the wiki is great, but presenting those things elsewhere on different websites would also be great. Scraping!

I created the SOTM sessions program page for 2014, implementing scraping from the wiki editable list at State Of The Map 2014#Program (and linked session pages). I ported the code for this into the ruby on rails site as a WikiMirrorController (on github). This was after demonstratin the principle in php demonstration of the principle of wiki content scraping for SOTM schedule pages. The idea is to specify a certain constrained set of wiki content, and proxying into onto another website (and cached locally) This might be "full page" type things, but the destination site can have it's own style and header and footer elements, or even bring in very small snippets or lists from the wiki, presented within just a small. "portlet" on the page. The idea could be used for...

Making a special more glossy styled website for a regional OSM group. I'm thinking of OSM London here. I like maintaining the list of events on the London wiki page, so if were to build an OSMLondon website to present the identity more boldly / stylishly, then it would be great to bring in the events list content from the wiki. There are many other regional website which already exist, and could maybe benefit from this approach.

Making a special more glossy styled beginners guide website, but with the Beginners Guide remaining on the wiki. It somehow doesn't look very cuddly and welcoming at the moment. With a large dollop of CSS, the wiki beginners guide content could look more like http://learnosm.org (in fact it could look exactly like learnosm.org and be made available within that website. Not sure if that would make sense but it's all possible)

Making something js code example snippets website. like my set of js code examples, but with the code snippets coming from the wiki. Not sure if this is a great idea. Obviously we'd have to watch out for security implications on that one (someone could edit the wiki to put in a javascript redirect to a porn site) Might need some permissions/change moderation features. Another problem is getting the community to agree on a good stripped down example snippet isn't always easy. I could try the idea for presenting snippets on my website, but what I was actually thinking about was http://switch2osm.org

Making HOT project lists on HOT's website. http://hot.openstreetmap.org has been designed to be glossy and appeal particularly to funders, but people end up on the site for all sorts of reasons. The really juicy information about current projects and activations is something I firmly believe should be live on the wiki and wiki editable. This works well in terms of community editing, but I think people sometimes have a hard time finding the info. So... use a wiki scraping approach to bring a list of HOT projects from the wiki onto the hot website. For this list it could simply be scraping a wiki category. The HOT website is in drupal, so a wrapping scraping logic in a drupal plugin might be the way to go (or does generic scraping exist as a drupal plugin already?? investigate!)

Making HOT project descriptions appear within Tasking Manager. I'm not sure if Pierre Giraud would like the idea much, but the descriptions appearing in the task manager, describing the project and describing the workflow. These things get refined over time during an activation response, but often rather slowly, or sometimes they get left in quite an short state. Ideally there'd be good pithy descriptions and compelling images, and screenshot images to illustrate example tasks. To make this happen better, throw it open to the wiki editing community with a wiki scraping approach?

fix the funmap editor

http://funmap.co.uk/editor/ It was a nice simple php OAuth client... until something changed with OSM oauth. It was also an editor. Not really suitable for anything serious, but proving a concept, and it did form the basis of the first version of toiletmap.org...

narrow focused editor like toiletmap.org

Some time ago gregory made toiletmap.org which primarily showed a map of toilets (a narrow subject focus) but also allowed adding and editing toilet data (php OAuth). Now the idea has since been folded into some other great british toilet map stuff which... I may be being unfair, but it seems to have become much less OSM-helpful. This app could (should?) be driven directly from OSM's open data, using it as the central repository, and instantly writing users contribution back to OSM.

Of course this idea works just as well for some other narrow vertical concept e.g. pub map. clothes shop map. ...or any endless number of things.

Couple of pre-existing narrow focused maps: coinmap.org for bitcoin places. Code?. osmcamera.tk shows CCTV cameras github.

narrow focused editor like COFFEEdex

Recently Tmcw's done this kind of thing with COFFEEdex. A nice further narrowing of the simple editor concept, is to only add a new tag to existing cafes. Saves needing to add nodes or ways. Adding tag is dead easy (in code terms). This could be for adding any kind of sensible tag to OpenStreetMap so its a shame he's chose the not very sensible cost:coffee=* tag.

It could be fun to just fork COFFEEdex and do something different with it. The tech stack seems a bit alien to me though, so I'd be tempted to re-implement in php for example. There's certainly plenty of room for more projects doing this kind of thing.

I came across "yardmap": http://content.yardmap.org It's google-encumbered. I know there a people interested in mapping ecological type info in OSM, but mostly this is just vague inspiration. check out the nice video, and the way it's designed to appeal to kids.

Pizzacompass

PizzaCompass? Can I get some OpenPizzaCompass?? Nice short hack weekend sized idea. I actually started work on this at the London Hack Weekend. Here it is so far: http://harrywood.co.uk/pizzacompass/ ...but only spent an hour or so on it, so it's not actually querying OSM yet.

Obviously substitute in different types of POIs for an equally valid idea, but I think a important and fun aspect of this is that it's mega narrow focussed and simple.

Having said that... extensions to the idea of basic POI searching apps are also cool and obvious and not all that well exploited at the moment. I did come across a nice new example here of the kind of thing I'm thinking of: http://thenextis.com

Notes dedicated app

An app for viewing, commenting on, resolving, and adding Notes

A dedicated app for low-bandwidth use on the move, which lets you view nearby notes as a list (Bonus features: Notes on a map, or in "compass" mode)

Then given phone's built in cameras I think it would be neat to make it dead easy to reply to a note (or create a note) with a photo. By this I mean upload a photo somewhere like twitpic/flickr and bung a link in the comment text. In future we might look to support photos more natively in OSM notes if this proved popular.

Maybe we can modify existing notes apps. Certainly we need to acknowledge prior work in the area:

  • OSMAnd and Vespucci both have full note handling functionality which I'm yet to play with much.
  • "OSM Note" is a dedicated notes app (as a website). Think it doesn't show existing notes, only lets you post. Could be a good starting point to develop.
  • When it comes to showing notes I recently did some hacking to load notes into MAPS.ME.

full Notes#Applications using notes

Show me where emailer

A site which is for posing a question "show me where". The site has two user journeys, one for the question asker. One for the question responder.

  • Question asker: Setting up a question involves generating a unique link which can be emailed or otherwise passed on to the question responder. Set up might also (optionally) involve stating the name of the thing you want to locate. Or perhaps a more long text of a question e.g. "Hi mum. Show me where exactly the restaurant is", and selecting a startpoint for the map (you may know approximately where something is, but you want to know exactly).
  • Question responder: Is presented with the question, and they can locate the thing exactly. Crucial thing about this is that it should be super-dooper-easy. The idea of this is that we could present this question to somebody who is really clueless about how to use websites etc. Another important thing is that website should guide them to locate the thing exactly. I see systems like this (OSM Notes for example) which confuse people who don't really get the idea of zooming and placing a marker really precisely, so it ends up being in the wrong place. As such, I think it would ideally force people to zoom in and in and in, before logging the response. Maybe a single click interaction for zooming would help noddy users get it. ...although ...

(bonus idea). Detect if a user has "dropped-off" perhaps because they found it confusing or didn't figure out how to zoom, or just forgot click 'submit' and email the question asker with details. The site could actually track what exactly the responder does. (Did they open the link? Did they click on the map at all?) and maybe their IP address

With this bonus idea in place, the interaction could be further complicated, and it wouldn't matter if the user didn't make it to the end. For example it could ask them to locate the thing, and then pose a further question. Perhaps generically a textbox saying "Add any further details you like here then click submit". But this could be customisable by the question asker. e.g. "...and what is the address of the restaurant?".

Note: I see this as a very generically useful app which would just happen to use OSM as the basemap. ....but if course it would also be useful for posing a question to a non-tech-savvy local to gather details about something we want to add to the map.

Group tweeting tool

This is not really an openStreetMap hack, but a twitter hack. We're paying for the use of http://grouptweet.com for CWG to manage @openstreetmap, but we now need it for @hotosm, and I'd also like to have it for @OSMLondon, and @OSMUK, and I'm sure there's other community accounts where people would like it.

I've actually started building something, and making some progress based on this sample code.

The other thing to be aware of is twitter themselves introduced 'Tweetdeck Teams'. It looked like this would obsolete this idea. However CWG experimentation revealed it's missing a view of who has sent what tweets. So that's a major bummer. It's quite a new thing though. Maybe they'll add this.

So anyway, onto the idea. It can be outrageously simple:

  • A website where individual people authenticate (somehow... the logical mechanism might be twitter, but could be OpenStreetMap or anything)
  • A list of people who are allowed access. This could just be in a fixed configuration file for the first cut.
  • For those who have access, a textbox for sending a tweet!
  • Tweets would be sent from the managed account (that's the whole point). Details of the account could be in a fixed config file.
  • A view of who has sent what tweets.

I'm pretty close to achieving the above with my code actually.

Bonus points:

  • There's probably lots of gnarliness around previewing what a tweet renders like, shortening URLs, etc and displaying how many characters you have left, so bonus points for getting that working as well as possible.
  • Other standard tweeting stuff: Ability attach an image. Ability to do a native retweet somehow
  • Ability to schedule a tweet for e.g. 1 hour's time.
  • Requirement to schedule a tweet for 1 hour's time for some lesser permission level users.
  • Email notification going out to everyone else (or IRC or other notification) when a user first schedules a tweet. This could create a useful "check eachother's tweets" workflow and would allow for less trusted users to join the fun.
  • Get proposed tweets from a wiki page instead?

non dev

Do these count as "hacks"? More like a TODO list.

wiki


documenting
  • Write a tutorial like this one on google 'my maps' but on using uMap. (That might give rise to some feature idea for uMap or maybe a hacked version of it especially for itineraries)
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=User:Harry_Wood/hack_ideas&oldid=2349787"