Create a contact sheet of images

On Linux Mint (and just about any other Debian based distro) gThumb has an option to create a contact sheet of images in a directory.  However, on my machine running Mint 14 it crashes every time.  Hit duckduckgo to find an alternative and voila! ImageMagick Montage.  The montage program is part of imagemagick. Install imagemagick.

$ sudo aptitude install imagemagick

Continue reading Create a contact sheet of images

Optimising Raster For Web Mapping (on Linux)

OS raster colourThe previous post showed how to process panchromatic and colour TIFF images on Windows to optimise them for web mapping.  Here I’ll show you how to do the same thing using bash scripts on Linux.  This assumes you have GDAL installed (gdal-bin and python-gdal on Ubuntu/Debian/Mint).  Continue reading Optimising Raster For Web Mapping (on Linux)

Optimising Raster For Web Mapping

OS rasterI have been working through an exercise to find alternatives to storing raster data in an expensive proprietary database format. We have a lot of raster data in various formats (ECW, jpg, LZW and Packbits compressed TIFF, uncompressed TIFF, GeoTiff) and of different types (historic black and white mapping, full colour aerial photography, panchromatic (greyscale) aerial photography, full colour base mapping tiles at different resolutions). Continue reading Optimising Raster For Web Mapping

pgRouting on Windows 7

pgrouting-networkDrive time isochrones with pgRouting and PostGIS on Windows 7 Pro.  I am using Postgresql 9.0, 9.1 and 9.2, PostGIS 2.0.3 and pgRouting 1.0.7-dev.  Anita Graser has some excellent pgRouting resources on her blog  and I used them in conjunction with Jo Cook’s pgRouting with ITN guide to load and build my network dataset.

One of the problems I had while following this post was that I had more than the three airports she used in her example and to manually update the table 116 times for each play park in the county was not an option.

So, how to write a looping function in Postgresql to loop through all the play parks and calculate all the drive times for each facility in one go?

Postgresql docs are a good place to go and this is what worked for me in the end.  Once the table is complete finish up using Anita’s how to.

-- 1. create calculate playpark costs function
CREATE OR REPLACE FUNCTION insertPlayParkCosts() RETURNS INTEGER AS $$
-- NOTE: function returns an integer representing the next play park nearest node id
DECLARE
 nn INTEGER;
BEGIN
-- NOTE: loops through every nearest_node, calculating drive times and inserting records into the table
FOR nn IN SELECT nearest_node FROM osplayparks_pts LOOP
 INSERT INTO catchment_pp (
 SELECT
 id,
 the_geom,
 (SELECT sum(cost) FROM (
 SELECT * FROM shortest_path('
 SELECT objectid AS id,
 source::int4 AS source,
 target::int4 AS target,
 traveltime::float8 AS cost
 FROM angus_routes',
 nn, -- this updates each time with a new value for nearest_node
 id,
 false,
 false)) AS foo ) AS cost
 FROM vertices_tmp);
END LOOP;
END;
$$ LANGUAGE 'plpgsql';

-- 2. run calculate playpark cost function
SELECT * FROM insertPlayParkCosts();

Installing pgRouting on Windows 7

pgrouting-isochrones

This quick guide outlines the steps I took to get Postgresql, PostGIS and pgRouting installed on Windows 7 Pro.

1. Install Postgresql

Download the installer from EnterpriseDB.com, double click to start the installer and accept most of the defaults except for the port to run the server on.  I installed 9.0.13, 9.1.9 and 9.2.4 running each server on a different port.  So 9.0.13 is running on 5432, 9.1.9 on 5433 and 9.2.4 on 5434.

The installers install the correct version of pgAdmin III for each version of the database and it is recommended to use the right pgAdmin III with the right Postgresql.  This ensures database integrity and interactive functionality suitable for each version.  For Postgresql 9.2 use pgAdmin 1.16.1, for Postgresql 9.1 use 1.14.3 and for Postgresql 9.0 use 1.12.3.

2. Install PostGIS

Download the installers from OSGEo.

For Postgresql 9.0 install http://download.osgeo.org/postgis/windows/pg90/postgis-pg90-setup-1.5.5-1.exe and select the correct instance of Postgresql on port 5432

For Postgresql 9.1 install http://download.osgeo.org/postgis/windows/pg91/postgis-pg91-setup-1.5.5-1.exe and select the correct instance of Postgresql on port 5433. I could have installed a later version of PostGIS on 9.1 but I needed 1.5.5 to work with AstunTechnology’s PortableGIS Loader! (See archaeogeek’s blog)

For Postgresql 9.2 install http://download.osgeo.org/postgis/windows/pg92/postgis-pg92-setup-2.0.3-2.exe and select the correct instance of Postgresql on port 5434

3. Install pgRouting

For the development versions of pgRouting visit Boston GIS and get 1.0.7dev for Windows 32-bit and PostgreSQL 9.2 (I am running 32-bit Windows).  pgRouting contains the following features:

  • Dijkstra algorithm – Shortest path algorithm, which named in honor of Prof. Dr. Edsger Wybe Dijkstra who has invented it
  • A-star (A*) algorithm – Shortest path algorithm using heuristical function
  • Driving distance – area person can cover in certain time from start point using road network
  • TSP Travelling Salesman Problem solution
  • Shooting star (Shooting*) algorithm – Shortest path algorithm for real road networks with turn restrictions, traffic lights and one way streets.

To install and enable pgRouting (read the README) copy the contents of /lib (.dlls required for problem solving) and /share/extension (.sql files to enable pgrouting in the database) to the C:\Program Files\PostgreSQL\9.2\lib\ and C:\Program Files\PostgreSQL\9.2\share\extension folders respectively.

Then, using the psql prompt or pgAdmin III, do:

CREATE EXTENSION pgrouting;

Now you should have a pgRouting-enabled PostGIS database.

There are older version of pgRouting available to download from Github.  You’ll need to extract the zip archive and run the various SQL files to install the required routing functionality as well as copying the .DLL files to the correct folder as above.  Boston GIS have also release dev versions of pgRouting 2.0.0 which I have not tried.

Varnish cache, munin-node and server-status

varnish-cacheOn my server running Ubuntu 12.04, I have Apache2 listening on port 8008 (20 virtual hosts) and Varnish on port 80. I use munin and monit to keep tabs on the various services running on the machine and they use the mod_status output to keep tabs on Apache2 processes. As mod_status is compiled into Apache2 by default and the module is enabled this means that the detailed information about your secure webserver is exposed for all the world to see. The mod_status configuation file allows you to lock access down to localhost access only which is nice.

munin

However, Varnish connects to the backend Apache2 server as localhost and so exposes the server-status page to the wild wild web. With the Varnish VCL I was using my server-status page was cached for an hour so it was still available but only provided a static status report that changed when the cache refreshed. Implementing a solution from serverfault.com made the live server-status available through Varnish as it passed it to the backend. This did not help. To secure my server-status I had to block access to the server-status URL but still keep it open for munin-node. Continue reading Varnish cache, munin-node and server-status

Replacing a failing HDD in RAID1 in Ubuntu Server

I have a software RAID1 (mirrored) setup with four partitions on two 160GB SATA drives. Munin started emailing me reports that a drive was failing with bad sectors.

This is what I did to get the system up and running again.

From advosys.ca I completed the following steps first in the terminal:

To make Ubuntu Server automatically boot when one drive in a RAID array has failed do the following:

From a running server, do a package update to make sure you have the latest kernel and boot loader

sudo apt-get update && apt-get upgrade

Reboot the server to ensure any new kernel and bootloader packages are in place.
From the command line run

sudo grub-install /dev/md0

to ensure GRUB is installed on all members of the boot RAID device.

When asked “Should mdadm run monthly redundancy checks of the RAID arrays?”, select either Yes or No (read the warning about possible performance impact and decide. “Yes” is the safer choice)

From the command line run

sudo dpkg-reconfigure mdadm

When asked “Do you want to start the md monitoring daemon?” select Yes.

Enter a valid email address to send warning messages to.

When asked “Do you want to boot your system if your RAID becomes degraded?” select Yes. Continue reading Replacing a failing HDD in RAID1 in Ubuntu Server

Batch Exporting to PostGIS from Oracle

The process in the previous post loaded one layer at a time.  This batches the process.

Some preparation of a text file with a list of all Oracle table names to be extracted and loaded into PostGIS is required.

rem This reads a list of table names from txt file and passes them to ogr2ogr which then exports from Oracle into PostGIS
for /F %%f in (C:\path\to\your\textfile\OracleTables.txt) do ogr2ogr -a_srs EPSG:27700 -overwrite -f "PostgreSQL" -nln yourschema.%%f PG:"host=localhost user=youruser password=yourpassword dbname=yourdatabase" OCI:"oracleuser/oraclepassword@(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = oraclehost)(PORT = 1521)))(CONNECT_DATA = (SID =oraclesid))):oracleschema.%%f"
rem
pause

Exporting Data from Oracle to PostGIS

Traditionally exporting from Oracle to PostGIS involved an intermediate step Oracle to Shapefile and then Shapefile to PostGIS. But this means that field names longer than 10 characters in Oracle get truncated in the conversion to Shapefile. And using the QGIS or PgAdminIII shp2pgsql gui has no easy way to set the schema or SRID other than manual intervention. I guess it could be run from the command line though…

Using Safe’s FME you can go directly from Oracle to PostGIS. But there is no way to define the destination schema in the PostGIS database. The PostGIS writer expects there to be a schema with the same name as the Oracle schema in order to complete the extraction and translation. See https://groups.google.com/forum/#!topic/fmetalk/zQsUUJNR5h8

So, open-source here we come. ogr2ogr can read and write to a number of different formats including Oracle (this uses the proprietary OCI.dll). On Windows install OSGeo4W with GDAL/OGR. Run the setup.exe and “Advanced Install” to install the gdal-oracle10g package. See http://stackoverflow.com/questions/9789686/migrate-spatial-data-from-oracle-to-postgresql

The command to issue at the command line is (all on one line):

ogr2ogr -a_srs <srs> -overwrite -f "PostgreSQL" -nln <postgis schema>.<postgis table> PG:"host=<postgis host> user=<postgis user> password=<postgis password> dbname=<postgis database>" OCI:"<oracle user>/<oracle password>@(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = <hostname>)(PORT = <port number>)))(CONNECT_DATA = (SID =<sid name>))):<oracle schema>.<oracle table>"

<srs> = in the EPSG format e.g EPSG:27700

This uses ogr2ogr to connect to an Oracle database, pull a table from a schema and write it to a defined schema in PostGIS database.

Nice!

Sources: