Humanitarian OSM Team/Working groups/Technical/meeting 2013-07-15

Meeting to discuss HOT technology on IRC on Monday 15th July 2013

IRC log:

17:59 mkl1: right on
18:00 mkl1: hi everyone
18:00 mkl1: thanks for joining again
18:00 mkl1: to recap, last week we just got started with this chat.
18:01 mkl1: we basically went around and did check ins and discussed what came up
18:01 mkl1: decided to start documentign our tools, our needs, and our ideas
18:01 mkl1: and resolved to do it again
18:02 mkl1: and somewhere along the way, the idea for the Birthday Sprint was born
18:02 mkl1: http://wiki.openstreetmap.org/wiki/Humanitarian_OSM_Team/Working_groups/Technical/meeting_2013-07-01
18:02 mkl1: so why don't we start off with a round of check ins, get into some discussion, and then make some plans for the next few weeks, including the sprint
18:03 : pierzen|2 [~kvirc@205.151.117.55] entered the room.
18:03 mkl1: how bout we go clara, then boris_m, then ybon, then AndrewBuck, then me. and anyone else, just wave
18:03 mkl1: ok?
18:03 AndrewBuck: souds good
18:03 pierzen|2: hi
18:04 mkl1: hi pierzen|2
18:04 clara: hi, I'm a Drupal site builder and offered to help with the hot.osm site
18:05 mkl1: hi clara. have you been set up with access? what are the priorities to help with there?
18:05 clara: no access yet
18:05 clara: I would mainly need access to a development site
18:05 clara: both an account and preferably ssh
18:05 clara: <end>
18:06 mkl1: currently the hosting is on, pantheon?
18:06 mkl1: i think it will be moved though
18:06 mkl1: waiting on the new sysadmin to start
18:07 mkl1: one thing that would be helpful for the website, would be a simple issue tracker
18:07 clara: I can also set up a dev site on my server to start working on things
18:07 mkl1: i think there was some talk of moving to drupal 7
18:07 pnorman: waves
18:08 clara: yes - it would make sense to move now instead of setting things up in drupal 6 and then upgrade in a few month
18:08 mkl1: i think setting up a dev site, as well as figuring out a place to stash issues would be a good start
18:08 AndrewBuck: We should also have some pages on the site that list the current activations, the current projects we are mapping on that aren't activations, and the countries where we have ground teams currently working.
18:08 mkl1: indeed!
18:09 mkl1: pierzen|2 had some other suggestions as well
18:09 pierzen|2: a hierarchical menu that let go rapidly through various infos
18:09 mkl1: does it make sense to just create an empty github project, to use as a place to stash these feature requests?
18:09 clara: i've got mantis as an issue tracker if that's of any use
18:10 pierzen|2: We should have such menus on our various tools including the TM
18:10 mkl1: pierzen|2 like a consistent toolbar across HOT dev projects?
18:10 mkl1: nice idea
18:10 pierzen|2: exacltly
18:11 clara: sounds like this could be a never ending list :)
18:11 pierzen|2: we call this a work in progress :)
18:11 mkl1: right, that's why i'm suggesting we get an issue tracker for it
18:11 clara: I would propose to first fix the RSS feeds, upgrade to D7 - and then take all the other ideas on board
18:12 mkl1: yup, makes sense to start with the upgrade
18:12 clara: and get the issue tracker of course
18:12 mkl1: let's take this to mike and felix, and figure out next steps.
18:12 pierzen|2: it should be discussed at the tech level before it is brought to the board.
18:12 AndrewBuck: github has issue tracker as well, if the site code is on there it makes sense to track issues there as well.
18:12 mkl1: clara, can you continue on the thread
18:12 clara: k
18:13 mkl1: AndrewBuck: it's drupal, i haven't seen drupal sites managed through github?
18:13 AndrewBuck: ok, I know nothing about it so I don't know.
18:13 mkl1: pierzen|2 i think we just implement that great idea and tell the board about it ;)
18:13 AndrewBuck: so drupal is hosting as well as language, etc?
18:13 pierzen|2: mkl1: yes
18:14 mkl1: it's code+plugins. but mainly the customization is through configuration in the database
18:14 mkl1: should we move on with checkins?
18:14 mkl1: boris_m is next
18:15 mkl1: or maybe ybon
18:15 ybon: ok
18:15 ybon: not many new things for me since last meetup
18:16 boris_m: Ok , I'm mainly here only for listen ...
18:16 ybon: Maybe the legend for the HOT style
18:16 harry-wood: hello all
18:16 harry-wood: ybon: yeah the legend is nice
18:16 ybon: http://hotosm.github.io/HDM-CartoCSS/
18:16 mkl1: looking
18:16 ybon: thanks harry-wood :)
18:17 ybon: the worldwide service is on its way, waiting for a new SSD disk for our server
18:17 mkl1: sweet, how it zooms to an example feature
18:17 : hjart [~hjart@37.128.215.244] entered the room.
18:17 mkl1: ybon: have you thought of collapsable sections, so you don't need the scroll bar?
18:18 ybon: yeah, I have it in mind
18:18 mkl1: k :)
18:18 mkl1: AndrewBuck, any updates?
18:18 AndrewBuck: does the server cache the images for the key somehow, or are they just live map tile calls cropped by js?
18:18 harry-wood: Made me think of an idea I had a while ago. I've suggested it somewhere. ...that the taginfo system should have examples of tags in use, so picking an instance of the tag at random from the DB to just to you see it
18:19 AndrewBuck: I guess the only things from me would be to repeat my request for a server to run mumble on, and that we use that a bit more.
18:19 AndrewBuck: But that has to wait on the sysadmin obviously
18:20 AndrewBuck: oh, there was some stuff about the hot export tool
18:20 AndrewBuck: so Currently the shapefiles it produces follow the osm_point, line, and polygon layers that osm2pgsql uses.
18:20 harry-wood: Frederik's had a spate of github interactions I see
18:20 AndrewBuck: This is a very bad way to export shapefiles, and it is very much not the way most people do shapefiles.
18:21 pierzen|2: thematic approach woul be better
18:21 AndrewBuck: It leads to bloated tag column lists making the files big, cumbersome, slow, and hard to query, and it also means that most people's normal workflow needs to be adopted to use them.
18:21 AndrewBuck: yes as pierzen|2 mentioned they should be broken down by theme.
18:22 harry-wood: Well... I suppose it's bad unless you're using Mapnik which likes its shapefiles that way
18:22 mkl1: what would they be based on then? only the tags mentioned in the presets file?
18:22 AndrewBuck: You make one file for say water, and that has all the waterways, dams, locks, etc, and then nothing else in it.
18:23 AndrewBuck: This means you can cut down the number of columns to the ones appropriate to those (you don't need a highway=* column for example on water features).
18:23 mkl1: what's the status with HOT export tool anyway?
18:23 AndrewBuck: And then you make similar files for roads, poi's, etc.
18:23 mkl1: i mean, i think Frederik is implementing to a spec. not sure how flexible that is
18:23 AndrewBuck: Also, each file would have points , lines, and polygons in the same file, rather than a layer for each.
18:24 AndrewBuck: I think there was some work being done towards something like this but we don't know the status.
18:24 mkl1: anyone tracking this? harry-wood?
18:25 AndrewBuck: Also, there are some standard schemas from orgs like the UN, red cross, etc, on how they interchange data (column names, what goes in each file, etc). We should have preset options that match these schemas, as many people will have workflows to use data in exactly this format.
18:25 mkl1: these are good ideas. at least, i don't want to lose them. can you create some issues AndrewBuck?
18:25 AndrewBuck: That way our data becomes a drop-in replacement for their current data source and they can keep their existing workflows, stylesheets, etc
18:26 mkl1: https://github.com/hotosm/hot-exports/issues
18:26 mkl1: or
18:26 mkl1: on the hot-exports wiki, create a page for "new ideas"
18:27 AndrewBuck: We can try to creatue issues on github
18:27 mkl1: https://github.com/hotosm/hot-exports/wiki
18:27 mkl1: that's the way to start organizing feature requests
18:27 mkl1: and giving some focus for the birthday sprint
18:28 mkl1: ok. pierzen|2, you're up
18:28 mkl1: any updates?
18:28 mkl1: (then harry-wood, then me)
18:28 pierzen|2: To support field teams, activations and various humanitarian requiring our help, we need various flexible tools
18:28 pierzen|2: - way to take account of bad internet connection
18:28 pierzen|2: - instance of data store on a server to take care of humanitarian tags while contributing also to add tags to OSM (a way to work closely with humanitarins)
18:28 pierzen|2: - ftp server
18:28 pierzen|2: - have the ability when necessary to implement rapidly other tools.
18:28 pierzen|2: - Develop Statistics
18:29 pierzen|2: that's all
18:29 mkl1: what is #2, this data store?
18:29 mkl1: is this like the private data store?
18:29 pierzen|2: yes
18:30 mkl1: why handle the humanitarian tags sperately?
18:30 : awwright [~awright@ip-64-134-243-251.public.wayport.net] entered the room.
18:30 mkl1: they're designed to be stored in osm, i presume
18:30 AndrewBuck: There is an MSF team in Central African Republic interested in working more cloosely with us, and a lot of the data they will survey does not make sense for direct inclusion in the DB.
18:30 AndrewBuck: Things like how many people live in a building, etc.
18:31 pierzen|2: this is a way for humanitarians to take care of confidential info while also contributing to osm by adding info.
18:31 AndrewBuck: Not necessarily private data but just not appropriate for our DB. And some might be confidential as well.
18:31 pierzen|2: yeap
18:31 mkl1: ok
18:31 mkl1: that's the private data store
18:31 AndrewBuck: If they could easily use something like the private data store, they could do all their work right in josm and then put all the OSM relevant stuff on public and keep theirs separate.
18:31 AndrewBuck: yeah.
18:31 mkl1: it doesn't need to be private, that's there call. but seperate
18:32 mkl1: have the links for that one?
18:32 AndrewBuck: We know it exists, the question is what is its status, etc.
18:32 AndrewBuck: Does HOT stand by it as a good way to do things for humanitarian orgs, etc.
18:32 mkl1: Kate was the client for that development. I think it's Frederik that implemented
18:32 harry-wood: yeah
18:32 : pierregiraud [~pierregir@ver73-2-88-160-44-40.fbx.proxad.net] entered the room.
18:33 harry-wood: I think it worked well for a very similar kind of task in indonesia
18:33 mkl1: AndrewBuck: we can try it for this case
18:33 AndrewBuck: do we have an API server running anywhere that they could use as their DB?
18:33 mkl1: https://github.com/hotosm/sds-server
18:33 AndrewBuck: I know they are a mess to set up
18:33 mkl1: and there's a josm plugin i guess
18:34 pierzen|2: yes
18:34 AndrewBuck: oh, interesting, so it is not a whole notehr API server then.
18:34 mkl1: for the time being, someone would need to step up and offer server space, and set it up
18:34 mkl1: yea, it *should* be easier than getting openstreetmap-website going
18:34 AndrewBuck: if it is a simple to set up server, they might set up that themselves then so they control it.
18:34 AndrewBuck: I thought it took a whole rails port setup.
18:34 mkl1: i doubt they have that capacity
18:35 pnorman: AndrewBuck: and running with multiple API servers is a pain - because if you ever mix them up, yo0u're screwed
18:35 AndrewBuck: or at least the API anyway.
18:35 AndrewBuck: pnorman: yeah, this only puts the tags on the second server and it does it automatically.
18:35 mkl1: sds-server looks like a simple rails app. not sure what it takes to get it running, and how you get your seperate data out of it
18:35 AndrewBuck: pnorman: you give it a list of tags that should go to the private data store, rather than to the actual OSM api.
18:35 mkl1: yea, i guess that's configured in JOSM
18:36 mkl1: pierzen|2: what is ftp server?
18:36 mkl1: i mean, what is it needed for?
18:36 pierzen|2: this is to take account of poor internet line, to facilitate ground team work.
18:37 pierzen|2: we prepare the files, zip it and place on a ftp server.
18:37 mkl1: to get them what?
18:37 pierzen|2: it is then easier to downloa
18:37 pierzen|2: d
18:37 mkl1: what files?
18:37 AndrewBuck: pierzen: the SDS server has nothign to do with the internet access that is a separate thing.
18:37 mkl1: AndrewBuck: i asked about another point pierzen brought up
18:37 pierzen|2: we talk about ftp andrewBuck
18:37 AndrewBuck: ok, yeah sorry.
18:37 pierzen|2: files, any commands from the team
18:38 AndrewBuck: I missed the transition.
18:38 pierzen|2: the support team does the work to gather these and prepare for the ground team
18:38 AndrewBuck: FTP is more tolerant of disconnect and reconnect and resume than most other file transfers.
18:38 flavour: rsync best
18:38 AndrewBuck: So each day the plan is to put the data they need on the FTP server in a single zip file that they can have a machine download.
18:38 clara: or torrents
18:39 flavour: right
18:39 flavour: But FTP is easy :)
18:39 mkl1: so this would be a request to the sysadmin, once they're started, and HOT gets some servers
18:39 mkl1: in the mean time
18:39 AndrewBuck: probably, yeah
18:39 mkl1: i could probably offer an account just for ftp on my dreamhost account
18:40 mkl1: is it pressing pierzen|2? or can it wait?
18:40 pierzen|2: I think that we also have ftp on our dev osm accounts.
18:40 AndrewBuck: I would say it could wait. I think the systadmin is soon anyway.
18:40 mkl1: ah
18:40 pierzen|2: it is pressing with the 4 new deploys in africa
18:40 mkl1: but if you can use your dev account, then ok?
18:40 pierzen|2: yes
18:40 mkl1: if there's a quota issue, can maybe ask TomH
18:41 AndrewBuck: yeah, that should work for now. But having a more permanent solution eventually is good.
18:41 mkl1: rapidly implement? pierzen|2, this is a need for on call devs?
18:41 pierzen|2: yes it should not be a problem. no big files, we erase regularly.
18:41 harry-wood: well the issue with dev accounts is that FTP users are server users (osm develeopers)
18:41 AndrewBuck: The quota shouldn't likely be an issue, it just gets uploaded once and downloaded once.
18:42 harry-wood: We can't be asking the sysadmins to set up new accounts for lots of humanitarian field operatives
18:42 harry-wood: or maybe we're talking about giving the password for one account
18:43 harry-wood: to a handful of people.
18:43 AndrewBuck: that could work for the interim. We probably won't need it in the next few days anyway, so depending on when the sysadmin starts we might be able to just wait.
18:44 mkl1: pierzen|2: anything else from your list you want to delve into?
18:44 AndrewBuck: By the time all 4 teams are up and running though we will defnitely want something, but that is a while from now I think.
18:44 harry-wood: when I said 'sysadmins' I mean TomH and others who sysadmin the dev server
18:44 mkl1: what is Develop statistics?
18:44 harry-wood: setting up FTP with permissions is easy on our own server of course
18:44 pierzen|2: mkl1: statistics, this is a request from Nicolas, to follow-up training groups.
18:45 AndrewBuck: harry-wood: yeah, that is what we are really looking for. If you could do that it solves the whole issue.
18:45 mkl1: what kind of statistics?
18:45 AndrewBuck: mkl1: stats on how many objects added/changed by each user, etc.
18:45 pierzen|2: osm contribution
18:45 harry-wood: AndrewBuck ...just need "our own server" :-)
18:45 mkl1: we're actually talking about groups right now on #osm-dev
18:45 AndrewBuck: harry-wood: ok, I thought you were impying we already had one.  :)
18:45 mkl1: we could maybe experiment with stats features
18:46 mkl1: to roll out to all potentially groups takes substantial infrascture
18:46 AndrewBuck: mkl1: is this as per your SOTM us talk on site redesign?
18:46 mkl1: but if someone wants to experiment just on a few groups, would be awesome to start generating stats
18:46 mkl1: yea, this is based of my talk, mvexel's talks, and a few others at SOTM
18:47 AndrewBuck: when would the site infrastructure be in place for something like that (the code and whatnot I mean, not necessarily the hardware for a full rollout to everyone).
18:47 mkl1: we're talking aobut getting a first version of groups pushed for the Birthday Sprint
18:47 AndrewBuck: ok, so soon enough the africa teams could try it out then.
18:48 mkl1: we could look at adding a hook to grab stats from a 3rd party site, for testing purposes
18:48 : awwright left the room (quit: Ping timeout: 480 seconds).
18:49 mkl1: wow, hour almost up
18:49 mkl1: um, harry-wood, want to check in?
18:49 harry-wood: Well I can report that we interviewed a couple of candidates for the sysadmin
18:49 harry-wood: that went well. very capable folks
18:50 harry-wood: so we can get somebody on board pretty soon I think
18:50 : pierzen left the room (quit: Read error: Connection reset by peer).
18:50 mkl1: awesome
18:50 mkl1: how soon is that going to be?
18:50 harry-wood: We'll need to structure our ideas, and reporting channels a bit
18:51 harry-wood: or we bombard the new sysadmin with requests for EVERYTHING, and see how he copes :-)
18:51 mkl1: perhaps they will have some ideas
18:52 mkl1: on communication
18:52 mkl1: ok, i'll go for checkin
18:52 mkl1: basically, what i want to share is that i met the folks behind BRCK
18:52 mkl1: last week in Nairobi
18:52 mkl1: they are interested in supporting OSM related development
18:53 mkl1: meaning, they'd be supportive, not pay for it
18:53 : pierzen|2 left the room (quit: Read error: Connection reset by peer).
18:53 mkl1: but that is something, since it's still in development
18:53 mkl1: meaning, BRCK could be a platform for pre-caching osm data and tiles for a field mission
18:53 mkl1: and build an method to do some remote updating
18:54 mkl1: so if anyone is excited about that idea, we can talk more
18:54 mkl1: it's a 400 MHz processor for v1, so can run a webserver, but nothing hefty
18:54 : pierzen [~kvirc@205.151.117.55] entered the room.
18:54 pnorman: checks in
18:54 mkl1: ...
18:55 mkl1: pnorman, yes
18:55 AndrewBuck: how much storage is it likely to have? Gigabytes?
18:55 pnorman: mkl1: is this related to the cache mentioned on the list?
18:55 mkl1: expandable, microsd
18:55 pierzen: Just back, lost connection and history of the last 10 minutes, had to change computer
18:55 AndrewBuck: ok, so you could even have a few SD cards that get swapped out periodically.
18:55 mkl1: 64 GB max i think
18:55 : awwright [~awright@ip-64-134-243-251.public.wayport.net] entered the room.
18:56 mkl1: yea. and they're stackable, so you can add additional ones for additional storage
18:56 mkl1: pnorman: maybe, not sure which list
18:56 pnorman: hot@
18:56 AndrewBuck: load one full of tiles, let everyone connected downlaoad all the tiles they want, then pop in the 'normal' one for the servers, etc.
18:56 mkl1: it's maybe related to pierzen's item about bad internet connection
18:56 mkl1: pnorman: probably :)
18:56 pnorman: "Here in Indonesia we're thinking of developing some kind of cache server, since network bandwith is a major issue almost in every workshop that we have."
18:56 mkl1: yea, something like that
18:57 AndrewBuck: yeah, if the brck has up to 64 gb storage then that could largely do what we are thinking of for the cache server.
18:57 mkl1: oh yea, my response to that email was thinking of BRCK
18:57 pnorman: mkl1: of course it doesn't talk about cache for *what*, it could be cache for imagery, tiles, api, whatever
18:57 mkl1: right
18:57 AndrewBuck: pnorman: it is cache for all those things.
18:57 pnorman: not on a 64GB SD card it isn't
18:57 mkl1: we'd want to build the tools to do different kinds of data at least
18:58 mkl1: and also, manuals and software installers
18:58 pierzen: cache is one of the many solution to take care of internet connection.
18:58 AndrewBuck: api is hard but we had suggested a overpass mirror kept up to date with augmented diffs for just that region.
18:58 mkl1: maybe it takes a few sd cards
18:58 AndrewBuck: tiles are the easiest to cache though and would offer the biggest benefit, so for now, just that would be enough.
18:58 mkl1: i was thinking it would be tough to have an offline api. but it could work easily to push Notes back up
18:58 AndrewBuck: And a squid cache should handle the tiles.
18:59 pnorman: tiles and imagery are fairly easy - in fact, you should find a tile cache recipe in chef-public using squid, which is what tile.osm.org uses for the CDN
18:59 AndrewBuck: mkl1: no point in making the write API cached and offline, it is mostly reads they have trouble with anyway.
18:59 AndrewBuck: Reads are big and common, writes tend to be small and infrequent.
18:59 mkl1: hmm, yea, gotcha. wonder if this processor could handle it.
18:59 AndrewBuck: pnorman: it is less for osm.org tiles and more for imagery, etc, but the same idea applies.
18:59 pierzen: we even had problem with 100k writes in haiti.
19:00 mkl1: chef recipe to get BRCK sorted makes sense
19:00 pnorman: AndrewBuck: augmented diffs are eeeh for this purpose for technical reasons
19:00 harry-wood: A simple intercept of requests to tile.openstreetmap.org would probably be handy, but maybe the BRCK will do that *anyway* for any kind of web traffic
19:00 AndrewBuck: pnorman: yeah, then maybe that is something we don't bother with. Like I said the imagery is 90 % of the issue.
19:00 pierzen: yes
19:00 AndrewBuck: Once you remove the imagery load, the other problems might go away on their own.
19:02 : awwright is now known as Guest204
19:02 : awwright [~awright@ip-64-134-243-251.public.wayport.net] entered the room.
19:02 mkl1: we can experiment
19:02 mkl1: i think there may be folks out there who just want maps on their brck
19:02 : Guest204 left the room (quit: Read error: Connection reset by peer).
19:02 mkl1: in fact, i know there are. but they are different from the hot field teams perhaps
19:02 AndrewBuck: What is the timeline on the brck? Next month or two?
19:03 mkl1: more like fall i gather
19:03 AndrewBuck: ok.
19:03 mkl1: before end of year for sure
19:03 mkl1: but we might be able to start building for it sooner
19:03 mkl1: i've asked if they are setting up a test suite
19:03 AndrewBuck: If we know the hardware and OS specs it would be possible to put the software side together sooner.
19:03 mkl1: and about api docs
19:03 mkl1: right
19:04 mkl1: it's WRT
19:04 AndrewBuck: Ahh, ok. That is trickier, but probably still doable.
19:04 mkl1: not sure what will come installed on it
19:04 mkl1: yes
19:04 mkl1: yet
19:04 mkl1: well it's 5 after the hour
19:05 harry-wood: time to head home for me
19:05 mkl1: want to wrap up with the birthday sprint
19:05 AndrewBuck: might as well.
19:05 harry-wood: is the birthday sprint a HOT thing? or a general OSM mega-hack weekend thing?
19:06 mkl1: i love the idea of working on stats and groups
19:06 mkl1: we're definitely working on groups, would be awesome to have something to connect them to
19:06 mkl1: i think it's general
19:06 clara: when is it?
19:06 mkl1: though only a few of us are discussing it in osm-dev
19:06 mkl1: August 10
19:06 harry-wood: 10th Aug is the official party day
19:06 harry-wood: normally spent in drinking all afternoon
19:06 mkl1: we'd gather wherever there's a few folks, and sprint online
19:06 harry-wood: at least that's what we do in London :-)
19:06 mkl1: then party
19:06 mkl1: what are you doing int he morning?
19:07 harry-wood: Last year I baked an OSM cake in the morning
19:07 harry-wood: this year I'm going to miss the whole thing :-(
19:07 mkl1: there's nothing sweeter than a code push
19:07 mkl1: we're also thinking about opening it up to other non-tech needs
19:07 mkl1: make it a big work sprint
19:07 mkl1: then the drinking
19:08 mkl1: clara: could be a good day to target to get HOT website moved to drupal 7
19:08 clara: could try
19:08 clara: give me a login and I check the status of the modules
19:08 clara: but should be possible
19:09 mkl1: please follow up with us on the thread with mike and felix, will help make this happen
19:09 clara: k
19:10 mkl1: ok. i'm going to sign off myself and get some dinner
19:10 mkl1: thanks, another interesting chat
19:10 clara: thanks
19:10 AndrewBuck: Yep, thank you all.
19:10 mkl1: please do keep documenting, filing tickets, and making plans for the sprint
19:10 mkl1: can someone throw the IRC log up on the wiki?
19:10 harry-wood: I gotta go too. I shall take a grab of the IRC log
19:10 harry-wood: in 5 ... 4... 3...
19:10 mkl1: thx harry-wood
19:10 pierzen: bye all
19:10 mkl1: bye