Category Archives: Computers

Varnish cache, munin-node and server-status

Posted on by
Reply

community 150x150 Varnish cache, munin node and server statusOn my server running Ubuntu 12.04, I have Apache2 listening on port 8008 (20 virtual hosts) and Varnish on port 80. I use munin and monit to keep tabs on the various services running on the machine and they use the mod_status output to keep tabs on Apache2 processes. As mod_status is compiled into Apache2 by default and the module is enabled this means that the detailed information about your secure webserver is exposed for all the world to see. The mod_status configuation file allows you to lock access down to localhost access only which is nice.

munin Varnish cache, munin node and server status

However, Varnish connects to the backend Apache2 server as localhost and so exposes the server-status page to the wild wild web. With the Varnish VCL I was using my server-status page was cached for an hour so it was still available but only provided a static status report that changed when the cache refreshed. Implementing a solution from serverfault.com made the live server-status available through Varnish as it passed it to the backend. This did not help. To secure my server-status I had to block access to the server-status URL but still keep it open for munin-node. Continue reading →

Replacing a failing HDD in RAID1 in Ubuntu Server

Posted on by
Reply

I have a software RAID1 (mirrored) setup with four partitions on two 160GB SATA drives. Munin started emailing me reports that a drive was failing with bad sectors.

This is what I did to get the system up and running again.

From advosys.ca I completed the following steps first in the terminal:

To make Ubuntu Server automatically boot when one drive in a RAID array has failed do the following:

From a running server, do a package update to make sure you have the latest kernel and boot loader

sudo apt-get update && apt-get upgrade

Reboot the server to ensure any new kernel and bootloader packages are in place.
From the command line run

sudo grub-install /dev/md0

to ensure GRUB is installed on all members of the boot RAID device.

When asked “Should mdadm run monthly redundancy checks of the RAID arrays?”, select either Yes or No (read the warning about possible performance impact and decide. “Yes” is the safer choice)

From the command line run

sudo dpkg-reconfigure mdadm

When asked “Do you want to start the md monitoring daemon?” select Yes.

Enter a valid email address to send warning messages to.

When asked “Do you want to boot your system if your RAID becomes degraded?” select Yes. Continue reading →

Batch Exporting to PostGIS from Oracle

Posted on by
Reply

The process in the previous post loaded one layer at a time.  This batches the process.

Some preparation of a text file with a list of all Oracle table names to be extracted and loaded into PostGIS is required.

rem This reads a list of table names from txt file and passes them to ogr2ogr which then exports from Oracle into PostGIS
for /F %%f in (C:\path\to\your\textfile\OracleTables.txt) do ogr2ogr -a_srs EPSG:27700 -overwrite -f "PostgreSQL" -nln yourschema.%%f PG:"host=localhost user=youruser password=yourpassword dbname=yourdatabase" OCI:"oracleuser/oraclepassword@(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = oraclehost)(PORT = 1521)))(CONNECT_DATA = (SID =oraclesid))):oracleschema.%%f"
rem
pause

Exporting Data from Oracle to PostGIS

Posted on by
Reply

Traditionally exporting from Oracle to PostGIS involved an intermediate step Oracle to Shapefile and then Shapefile to PostGIS. But this means that field names longer than 10 characters in Oracle get truncated in the conversion to Shapefile. And using the QGIS or PgAdminIII shp2pgsql gui has no easy way to set the schema or SRID other than manual intervention. I guess it could be run from the command line though…

Using Safe’s FME you can go directly from Oracle to PostGIS. But there is no way to define the destination schema in the PostGIS database. The PostGIS writer expects there to be a schema with the same name as the Oracle schema in order to complete the extraction and translation. See https://groups.google.com/forum/#!topic/fmetalk/zQsUUJNR5h8

So, open-source here we come. ogr2ogr can read and write to a number of different formats including Oracle (this uses the proprietary OCI.dll). On Windows install OSGeo4W with GDAL/OGR. Run the setup.exe and “Advanced Install” to install the gdal-oracle10g package. See http://stackoverflow.com/questions/9789686/migrate-spatial-data-from-oracle-to-postgresql

The command to issue at the command line is (all on one line):

ogr2ogr -a_srs <srs> -overwrite -f "PostgreSQL" -nln <postgis schema>.<postgis table> PG:"host=<postgis host> user=<postgis user> password=<postgis password> dbname=<postgis database>" OCI:"<oracle user>/<oracle password>@(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = <hostname>)(PORT = <port number>)))(CONNECT_DATA = (SID =<sid name>))):<oracle schema>.<oracle table>"

<srs> = in the EPSG format e.g EPSG:27700

This uses ogr2ogr to connect to an Oracle database, pull a table from a schema and write it to a defined schema in PostGIS database.

Nice!

Sources:

Open-source GIS cluster

Posted on by
Reply

This is the challenge for this week.

PostgreSQL + PostGIS + MapServer + MapProxy + Quantum GIS on Windows 7 32-bit

The aim is to create a scalable mapping solution based on open-source software that is easy to use and maintain, is fast and responsive and that can provide data in a number of different formats. Windows installers are available for most of the applications I wanted to use which made life a bit easier. Doing the same thing on Linux might be even easier. The corporate proxy and firewall certainly made things more complicated than they should have been.

mapproxy Open source GIS cluster Continue reading →

Amarok Last.fm integration

Posted on by
3

I am running Fedora 16 with KDE 4.8.3 and Amarok 2.5.0.  I have a last.fm account but I couldn’t get Amarok to scrobble my tracks.  It turns out it is an issue with KWallet, the KDE password manager.

The solution was found here in the KDE forums and, while this references version 2.3.x and the problem persists in 2.5.0, the solution is the same.

  • Open ~/.kde/share/config/amarokrc
  • Find the Last.fm plugin section
  • Add the following:
  • username=<yourlastfmusername>
  • password=<yourlastfmpassword>
  • ignoreWallet=true

Restart Amarok and start playing.  Check your last.fm account and see your currently playing tracks.  It works!

Scribble that spam

Posted on by
Reply

akismet stats 1024x615 Scribble that spamSince I linked this blog to Facebook and Twitter at the end of December 2011 the amount of spam it has attracted has increased by two orders of magnitude.  Where does this spam come from?  The posts published to Facebook are restricted in that they can only be seen by my friends so this should reduce the possibility that Facebook is the source.  The same links are published to Twitter and are public.  The chances are there are bots out there scouring Twitter (and other sites) for new links.  A regular tweeter and fresh content is a gift to spammers and unless you have some serious protection on your side your life will quickly become all about deleting junk from your site.

This is where Akismet comes in – Akismet has done a good job of filtering out the 24,000 (and counting) spam comments and should be the number one plugin installed and configured when you start up your WordPress site.

However, some spammers are getting clever and are using sophisticated software (and humans) to generate plausible strings of words – see the 22 missed spam in the image – that get past the filter.  The fact that people can be eomplyed to hand-type spam highlights how lucrative it is for the spammers!  22 out of 24,484 is not a lot (0.09%) but if those comments were to be published then the 22 or more links contained within them would start earning pennies/cents/rubles for the spammer who have sold on the links from my website.

I am sticking with cross-posting to Twitter and Facebook at the moment as Akismet seems to be getting on top of things (see the month on month decline) through its self-learning and the dedication of the team of developers behind the project.

New Weather Station

Posted on by
Reply
wxstn 001 20120121T093154 150x150 New Weather Station

Poor site

For Christmas Kevin gave me a WH1081 wireless USB weather station. It has a wind vane and anemometer, minimum and maximum thermometer, barometer, rain gauge and hygrometer mounted externally. It communicates wirelessly with a base station inside the house. The base station is connected to the server in the loft and there is much chatter between the two. The server runs pywws (Python Wireless Weather Station) software developed by Jim Easterbrook. This processes the data from the weather station at regular intervals and submits the information to a number of different sites. You can check the weather for IPERTHSH12 on Weather Underground (wunderground.com) and on the Met Office WOW site (wow.metoffice.gov.uk). You can also follow the current weather and 12 hourly forecasts on twitter (@newtyleweather). I also wrote up my own pages to display the weather information (cribbing from various other sites on the web) and give further details here: http://wx.mixedbredie.net/ Continue reading →

Upgrading WordPress to 3.3

Posted on by
Reply

blue l Upgrading Wordpress to 3.3Normally upgrading WordPress is as simple as clicking the Updates button.  This is a pleasure when you have six live sites and two test/dev sites.  The latest upgrade to version 3.3 from 3.2.1 was not as straightforward as is normal.  Updates informed me there were several plugins and themes with upgrades available.  OK, quick scan to check what was changing, check all, upgrade.  Voila!  It is done.  Backup datebase.  Export blog XML.  Ready to move from 3.2.1 to 3.3… Continue reading →