User:Geekasylum/Gentoo Linux Virtual Tile Server

From OpenStreetMap Wiki
Jump to navigation Jump to search

Gentoo Linux Virtual Tile Server

I'm building a virtual tile server to become part of my local network. I figure that a guide to setting up an up-to-date tile server on Gentoo will be useful, so here it is. This HowTo will also work for a real server - just skip over the section on building a virtual guest operating system, and use the Gentoo Handbook directly to install Gentoo on a real machine.

Note: While this page is still in my user space, its incomplete and being worked on. Please bear with me.

The virtual machine that I'm building is set up specifically for my network. Ie: It's registered in my DNS, and uses IP addresses that have been allocated to me, but its fairly simple to customise your own system to your needs while you install the base operating system.

The Virtual Tile Server described here will make use of the following applications:

  • Apache (latest)
  • Postgresql 9.1 and Postgis 2.0
  • Mapnik 2.1
  • Mod_tile and renderd (latest)
  • Osm2pgsql (latest)

Prerequisites

VirtualBox

You will need VirtualBox, running on your physical server to create and run virtual machines. This is easily installed on either Linux or Windows (and yes, you can run your Gentoo Linux Virtual Tile Server as a virtual machine under Windows if you like).

To install VirtualBox on a physical Gentoo Linux server, run the following command: 

 emerge -av virtualbox

It comes with command-line utilities to manage your virtual machines, but also has a GUI which makes common tasks easy.

Gentoo Virtual Guest

Follow the instructions here to create a Gentoo Virtual Guest. It's entirely up to you if you want to install X and a desktop such as KDE. I chose not to install a desktop on my virtual machine.

Optional Software

For my system, which will be part of a larger network, I added the following to the base installation. You probably wont need most of these - the Gentoo Handbook gives simpler alternatives to syslog-ng, and joe is our preferred editor. You can use the default editor (nano) or install app-editors/vim to edit the config files.

  • app-admin/syslog-ng
  • net-misc/ntp
  • app-admin/logrotate
  • sys-process/vixie-cron
  • net-fs/nfs-utils
  • app-editors/joe

Preparation

Use Flags

Set the following global use flags either by editing /etc/portage/make.conf directly, or installing and running app-portage/ufed

Note: On older distributions make.conf is in /etc. On my system, ufed expected it to be in /etc, but that may just be because I didn't reinstall ufed after moving make.conf. 

+postgres +postgis +threads +icu +bidi +geos +gdal

The only reason you would need to install X and a desktop is to run a web browser on the virtual machine. Since I already have a web browser on another machine I decided not to install X and unset the appropriate flag. Since Im not building X, I don't need gtk, and I also disabled perl to prevent packages from pulling in support for various perl libraries that we don't need. 

-X -gtk -perl

Some packages (like cairo) require flags that we just disabled (in this case X). Portage will warn you when this happens. Simply add those packages to /etc/portage/package.use and enable the required flags there. This way you dont have to set them globally - just on the package that needs them.

Enabling php support in geos causes threading conflicts with apache and php since its not thread-safe and forces -threads on these. You will have a much easier time if you simply disable php support in geos.

Add the following to /etc/portage/package.use, substituting the versions that your own system will be installing. 

# Cairo needs X
=x11-libs/cairo-1.10.2-r2 X
# geos causes use flag conflicts with apache and php. The current version also seems locked to php-5.3
=sci-libs/geos-3.3.3 -php

Whenever you're installing a package with emerge, you should examine the list of dependent packages to be installed and set any use flags which will allow them to build support for other packages that have been installed previously. Do a pretend emerge, before you launch the real thing, and look them over: (note the 'p' and 'v' options below) 

emerge -pv <package>

Add any use flags that will be beneficial to your build, and remove any that will install extra packages that you don't need. You can quickly set/unset use flags globally with ufed, or add the individual packages to /etc/portage/package.use.

Keywords

Many of the packages we are about to install are not part of the official portage tree and are therefore not marked as 'stable' for any given architecture. Some of the official packages are not yet stable on some architectures, so portage will prompt you along the way to add them to /etc/portage/package.keywords. Go ahead and add them as required.

Portage Overlay

For this project I used my own portage overlay which you can also use to obtain the ebuilds which are not yet in the official portage tree. Make sure that you don't have the git use flag set. For this overlay we only require subversion support, and enabling git will pull in a lot of packages that we don't need. Just set the subversion use flag, as shown in the example (Of course if you really need git for your own build, you can go ahead and add it - we just don't need it for the virtual server described here).

Build it

Install Postgres Database

Installing postgis will also install postgresql, since it is a dependency, but to ensure that postgresql is added to the world file (and thus updated by future portage updates) we will install them separately.

Note: This will pull in several other packages, including php 

 emerge -av postgresql
 emerge -av postgis

Portage will suggest the following. Ignore it. We do this by hand later 

* To finish installing or updating PostGIS edit:
*     /etc/postgis_dbs
* 
* Then, run:
*     emerge --config =dev-db/postgis-2.0.0

Edit /etc/conf.d/postgres-9.1 and ensure the following is set to a valid locale that is installed on your system. 

PG_INITDB_OPTS="--locale=en_AU.UTF-8"

You can get a list of installed locales with the following command 

locale -a

Once the locale has been set, use portage to configure postgresql 

emerge --config =dev-db/postgresql-server-9.1.5

By default, Gentoo is set up to be secure, however postgres allows full access to anyone with a shell account. Lets change that to make it require passwords (Check below to see how we avoid having to type the password every time)

Edit /etc/postgresql-9.1/pg_hba.conf and change 'trust' to 'password' for the localhost connections only 

# IPv4 local connections:
host    all             all             127.0.0.1/32            password
# IPv6 local connections:
host    all             all             ::1/128                 password

Start the postgresql server 

# /etc/init.d/postgresql-9.1 start

Open a connection to the server and set the password for the postgres super user. 

 # psql -U postgres

Set a password 

 postgres=# \password
 Enter new password:
 Enter it again:
 postgres=# \q

Edit /etc/postgresql-9.1/pg_hba.conf again, and this time, change 'trust' to 'password' for the local socket connection. 

# "local" is for Unix domain socket connections only
local   all             all                                     password

Tell the server to reload it's configuration. 

/etc/init.d/postgresql-9.1 reload

Typing passwords all the time is no fun, so lets fix that. In your home directory (cd ~) create a file called .pgpass and add the following to it. Put the password you just created in the password field. The first line is a comment so you can remember where everything goes. Note that the file name begins with a dot (.)

If you're using nano as your editor, you would enter: 

 nano ~/.pgpass

Add the following: 

# hostname:port:database:username:password
localhost:*:*:postgres:<your password>

Postgres will ignore this file if anyone else can read it, so set the permissions like so: 

chmod 600 ~/.pgpass

Create the user that mapnik and other software will use to access the database. You can make it a super user for now (just say 'yes' when asked). We will refer to this user as <username> but you type in the username that you chose. 

createuser -U postgres <username>

Create the database (most applications assume its called 'gis' by default, so lets go with that) 

createdb -U postgres -E UTF8 -O <username> gis

Set a password for your database user 

 # psql -U postgres

Set the password 

 postgres=# \password <username>
 Enter new password:
 Enter it again:
 postgres=# \q

Add the database user to your .pgpass file 

# hostname:port:database:username:password
localhost:*:*:postgres:<postgres user password>
localhost:*:*:<username>:<database user password>

Set up postgis on the database 

psql -U <username> -f /usr/share/postgresql-9.1/contrib/postgis-2.0/postgis.sql -d gis
psql -U <username> -d gis -c "ALTER TABLE geometry_columns OWNER TO <username>;"
psql -U <username> -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO <username>;"

Install Osm2pgsql

emerge -av osm2pgsql
psql -U <username> -f /usr/share/osm2pgsql/900913.sql -d gis

Install Mapnik

emerge -av mapnik

Mapnik requires at least 3Gb of memory to compile and will take a while. If the build fails with cc1plus (internal error) you probably need to give the virtual machine more memory. I had 2Gb of real memory and 2Gb of swap.

Verify the mapnik installation 

python
>>> import mapnik
>>>

If python replies with the second chevron prompt (>>>) mapnik is installed properly. Type quit() or press CTRL-D to exit.

Install Mod_tile

This will also install Apache 

emerge -av mod_tile

Install Mapnik Utilities

We need these to configure renderd later. 

emerge -av mapnik-utils

We will also need unifont since Mapnik-2.1 uses it by default. 

emerge -av media-fonts/unifont

Configure it

Configure apache

Edit /etc/conf.d/apache2 and add -D MOD_TILE to the options 

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D MOD_TILE"

Configure mod_tile and renderd

Edit the mod_tile configuration file installed as /etc/apache/modules.d/14_mod_tile.conf. I have updated the file installed by my portage overlay and taken care of most of it, but you should check that the server name matches your local machine (localhost should be fine for now) and notice the <Directory> section that I have added towards the end. The Apache default is to not allow access to anything. This section allows access our web site's DOCROOT directory where the slippymap is installed. (/var/www/osm).

Edit /etc/renderd.conf. There are a few changes needed here - its simpler if I just show you mine 

 [renderd]
 socketname=/var/run/renderd/renderd.sock
 num_threads=4
 tile_dir=/var/lib/mod_tile ; DOES NOT WORK YET
 stats_file=/var/run/renderd/renderd.stats

 [mapnik]
 plugins_dir=/usr/lib/mapnik/input
 font_dir=/usr/share/fonts
 font_dir_recurse=1

 [default]
 URI=/osm/
 XML=/usr/lib/mapnik/osm-local.xml
 HOST=localhost
 ;HTCPHOST=proxy.openstreetmap.org

Lines beginning with a semicolon are comments

Make sure the socketname is uncommented. Check your paths for font_dir, plugins_dir, the host name, and the URI and XML settings.

We will create the XML stylesheet in a moment. The URI is the virtual path used by mod_tile (and also found in the slippymap html).

The tile_dir path cannot be changed here, but the example shows where it should be.

Next, create the directories needed by renderd 

mkdir /var/lib/mod_tile
mkdir /var/run/renderd

Load some data

You can download the complete planet.osm file https://planet.openstreetmap.org/

To save space we chose not to install the entire planet file and used an extract from http://download.geofabrik.de/openstreetmap/

You can use bzip2 or pbf files directly with the latest osm2pgsql: 

 wget http://download.geofabrik.de/openstreetmap/australia-oceania/australia.osm.bz2
 osm2pgsql -U <username> -d gis -m -s australia.osm.bz2

Generate the stylesheet

Generate the default stylesheet for renderd 

cd /usr/lib/mapnik
./generate_xml.py osm.xml osm-local.xml --dbname gis --user <username> --accept-none

Edit the slippymap

Edit the slippymap html located at /var/www/osm/index.html

  • Update the start lat/lon and zoom (if you dont know the values, you can do this later)
  • Update URI for local tiles (localhost/osm)

Add startup scripts

rc-update add postgresql-9.1
rc-update add apache2
rc-update add renderd

Have a coffee - we're done

Point your browser at the slippymap on http://localhost/ and play

Next Steps

Now that our tile sever is built, we'll be exploring the following options, and more as we develop applications on top of it.

  • configure remote access to the postgres database if needed (/etc/postgres-9.1/pg_hba.conf)
  • emerge phppgadmin and install the webhost to visualise/manage the postgres database
  • Fine tuning - Eg: add use flags to enhance php features

Other considerations

  • Disk space

At this point, having browsed around the slippy map a bit the system is using 8Gb of disk space. The virtual disk image is using 13Gb which suggests I should use the Virtualbox tools to compact the image.

  • Expiry of tiles and updating the database

There are some scripts in the mod_tile distribution but they are hard coded and out of date. They could be used as examples to create your own expiry/update process. These were not installed by default. Look for them in the SVN sources under /usr/portage/distfiles/svn-src

Reference

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=User:Geekasylum/Gentoo_Linux_Virtual_Tile_Server&oldid=2019042"