Aim:

• Ubuntu Linux 10.10 desktop
• Apache 2.2
• MySQL 5.1
• PHP 5.2

Firstly, install Ubuntu 10.10. Make sure you opt to encrypt your home directory. While this means it’s difficult to recover contents of your home directory in the event of an OS crash, it’s also means you can sleep at night knowing that if you lose your laptop you haven’t lost your intellectual property. The system overhead for doing this is not really noticable. The down side of this is that you cannot access the files unless you’re logged on (see the bit later about the projects directory and Apache).

Now, let’s install Apache, MySQL and PHP. Note that I’m installing some extra packages that I need for my development purposes.

bob@bob-laptop:~$ sudo apt-get install libapache2-mod-php5 mysql-server php5-mysql
bob@bob-laptop:~$ sudo apt-get install php5-curl php-pear  # some extras I use
bob@bob-laptop:~$ sudo apt-get install openssh-server subversion bzr  # some extras I use

During the installation of those packages you’ll be asked for a password for the MySQL root user. I always use “root” for my convenience. You can use whatever you like. This user does not have root access to the system, but will have full access to the MySQL databases. By default MySQL only listens on the localhost interface so an insecure password like “root” is acceptable for my purposes. You’ll see later that we can shift certain databases to reside on the encrypted home directory if that’s important to you.

Set up the projects directory

I like to keep all my web stuff in a subdirectory of my home directory called projects. The full path to that will be /home/bob/projects. In order to be able to browse these projects with Apache I’ll put a symlink in /var/www (Apache’s default web root directory) to it.

Here’s a terminal session where I create the projects directory.

bob@bob-laptop:~$ mkdir ~/projects
bob@bob-laptop:~$ sudo ln -s ~/projects /var/www/projects
bob@bob-laptop:~$ ls -l /var/www
total 4
-rw-r--r-- 1 root root 177 2011-12-11 15:30 index.html
lrwxrwxrwx 1 root root  18 2011-12-12 21:25 projects -> /home/bob/projects

Great. But now if you try to access http://localhost/projects you’ll get a 403 Forbidden page. This is because Apache isn’t able to see your home directory. A small tweak will fix that.

bob@bob-laptop:~$ chmod o+x ~

Note that this command will allow any other user (of which Apache’s www-data is one) on your system to see the files in your home directory. For me this isn’t an issue as I’m the only user on my system.

Now if you revisit http://localhost/projects you’ll get a directory index instead of a 403 Forbidden page. You can now create files in /home/bob/projects (or whatever your user name is) and Apache will be able to serve them up to your web browser.

Note that because we opted to encrypt the home directory, the /home/bob directory is only mounted when bob logs in. This means that if you boot your system and try to access it over the network without being logged in, you’ll likely get a 403 Forbidden or 404 Page Not Found error for any URL in /projects. Other files can be placed in /var/www/whateveryoulike and will be accessible regardless.

Example:

bob@bob-laptop:~/projects$ cat > phpinfo.php
<?php phpinfo(); ?>

(press Ctrl-D after typing the phpinfo() line to save the file)

Now if you browse to http://localhost/projects/phpinfo.php you will get the standard PHP Information page.

Using an encrypted database

We’re not so much using an encrypted database as we are moving it onto an encrypted file system. We use the same trick that we used for Apache but we’re doing this for MySQL. Note that while it is possible to have all databases encrypted, you will need to make sure you log on first before you start MySQL. Here I will just encrypt one database:

-- Create a mysql directory in my home dir.
bob@bob-laptop:~$ mkdir ~/mysql

-- Create a sample mysql database called 'secure'
bob@bob-laptop:~$ mysqladmin -uroot -proot create secure

-- Stop the mysql service
bob@bob-laptop:~$ sudo stop mysql
mysql stop/waiting

-- Move the newly created 'secure' database to my home dir
bob@bob-laptop:~$ sudo mv /var/lib/mysql/secure ~/mysql

-- Create a symbolic link for mysql to use
bob@bob-laptop:~$ sudo ln -s ~/mysql/secure /var/lib/mysql/secure

-- Start mysql again
bob@bob-laptop:~$ sudo start mysql
mysql start/running, process 2778

-- Now access the database in the new place
bob@bob-laptop:~$ mysql -uroot -proot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 34
Server version: 5.1.49-1ubuntu8.1 (Ubuntu)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use secure;
Database changed

No errors – the connection was successful. Again the same thing applies in that you can only access this database while you are logged on.

.htaccess files (optional, but recommended)
Out of the box, Apache won’t read .htaccess files. To enable these permanently for all directories under your /var/www directory (including the project directory), edit the /etc/apache2/sites-enabled/000-default file (as root) and change line 11 to say “AllowOverride All” (instead of None).

mod_rewrite (optional, but recommended)

Also, you may want to enable mod_rewrite as many web applications use this to generate nice looking URL’s.

bob@bob-laptop:~$ sudo a2enmod rewrite
bob@bob-laptop:~$ sudo apache2ctl restart

xdebug plugin for PHP

The xdebug plugin gives you nice stack traces and formatted var_dump()s which makes it easier to develop. It also comes with breakpoints, remote debugging and a range of other things. Check out the whole shebang at http://xdebug.org/.

Installation is simple:

bob@bob-laptop:~$ sudo apt-get install php5-xdebug

Restart Apache for the change to take effect (sudo apache2ctl restart).

Enabling display of errors in your browser

By default PHP ships with display_errors = Off. This is a requirement for a production system but it will drive you nuts on a development box. When off, any errors are logged to Apache’s error_log file in /var/log/apache.

To turn it on, edit the file /etc/php5/apache2/php.ini and look for the lines like this:

display_errors = Off
display_startup_errors = Off
html_errors = Off

And change them all to “On”. Again, restart Apache for the change to take effect (sudo apache2ctl restart).

Happy developing!

I’ve been developing for a long time – the first programs that I first wrote that other people were using was back in 1992. At the time, I was the only one working on the code and there was no reason for others to get involved. In fact back then it seemed that we were driven by one-upmanship and sharing code was almost unheard of, unless you had to get an assignment in the next day and you had some stupid reason for not being able to complete it, then someone might take pity on your ass and bail you out. Back then, consistency wasn’t important – what mattered was whether the code ran or not. Even better if it ran and you got paid.

If there’s one thing I could share with you about my experiences over the last twenty years, I would have to say that it’s this. Be consistent.

Consider the following code:

<?php
  $query = 'select * from customer where id = 123';
  $result = mysql_query($query);
  $row = mysql_fetch_assoc($result);
  $identifier = $row['id'];
  echo $identifier;
?>

We’re fetching a row from a table and echoing out the id field. The code above doesn’t really matter, but what does matter is the inconsistency. Can you spot it?

The inconsistency is that the field in the database is called “id” and we put it into a variable called $identifier. This does not break the code, but it does mean that you have just introduced an inconsistency that will bite you.

That’s a pretty simple example, but imagine those kind of differences across an entire application, a database schema or even your whole organisation. It happens, and all too frequently.

The consistency issue isn’t a coding style issue – let me be very clear on that – it’s more a culture issue. You need to work hard to spot these inconsistencies in your organisation and nip them in the bud really early. And they may not appear in code either. You can find them anywhere from code, to documentation, to common usage in discussions in the workplace.

If you let these inconsistencies propagate you’re going to have to deal with them time and time again. Here’s a real world example: I am dealing with a system at the moment where products are identified by their SKU number (Stock Keeping Unit). This number has been referred to as the following in various places (code, documentation, wiki pages, emails etc):

• ItemCode
• ItemId
• Item_Id
• ItemID (yes, case makes a difference!)
• ItemSKU
• Identifier
• Identifer (yes, misspelling of Identifier)
• ProductId
• ProductSku
• SKU

The boat has sailed on that one – there is no way that this situation could be wrestled under control, you’ll have to do what you can if you come across this. But if you have the luxury of a fresh project then it’s time to gather your troops and spell it out on the board – CONSISTENCY. Drum it into your processes and procedures, your code reviews and commit messages. There’s no excuse for any inconsistency at all – especially when it will cost you in time chasing bugs because the value isn’t turning up like it should or the record isn’t saving.

If you haven’t already I highly recommend putting together some coding standards for your company – or even for yourself. And follow them to the letter. It doesn’t really matter what style of indenting you use (K&R, Allman, BSD KNF, Whitesmiths etc) – what matters is that you’re … you know what I’m going to say don’t you.

Feel free to borrow from the Turboweb Coding Standards (which in turn were based on the Drupal Coding Standards). Make sure that your new starts read your coding standards and understand them. And take time to teach them well.

Addendum: Ironically I managed to misspell the title of this blog post, I’ve now corrected it but as the link is out and about on various social media sites I can’t change that, therefore if you’re viewing this post by itself you’ll see that the URL says “consitency”

To get access to it you will need to install the compiz settings manager, and load Compiz’s “Fusion Plugins Extra”:

sudo apt-get install \
    compizconfig-settings-manager \
    compiz-fusion-plugins-extra

Once you’ve done this simply enable it:

1. System, Preferences, CompizConfig Settings Manager
2. In the Filter box in the top left, type in “grid”
3. Tick the box to enable it and click “Close” (bottom left)

Now try using Ctrl-Alt and numbers on your keypad. CA-9 will snap a window into the top right corner (press it a few more times to get different sizes) and CA-8 will make it snap to the top half of the screen (the number is the position on the screen, you’ll get the gist of it).

Check out this demo video:

Happy Ubuntuing!

“I’m a hundred year old kid – don’t you see,
I’m proud to see the end of our first century.
From 1885 through ’til 1985,
We’ll still be century children at our next centenary.”

Author unknown, can you help?

Today was Walk To Work Day in Dunedin, an initiative spearheaded by Living Streets Aotearoa with the support of Sport Otago and the DCC. I decided to make the trip as the amount of excercise I get thanks to the extreme convenience of my scotter is minimal. I left home at 7:20 this morning and arrived in the Octagon at about 8:30 where a few stands were laid out for registration. The registration process was rewarded with a $7 voucher redeemable at Barakah, MASH, The Ra Bar, Alibi or The Craic. I chose MASH and a nice looking breakfast bap which was delicious, but unfortunately my order was forgotten in the rush of breakfast time cafe operations. After a reminder it turned up with an apology.

They say New Zealand is a small place and indeed it is – I ran into community advocates Frank Buddingh and Nina Arron from Lawrence who were on the registration stands. A familiar face was very much a welcome sight. I first ran into Frank and Nina some time ago when Frank was looking for an alternative provider for the Buddingtree Consultancy website.

For me the nicest thing (other than the fantastic weather!) about walking into town was it was a chance for me to “sit down” and listen to some podcasts. Listening to Leo Laporte, Jono Bacon and Randal Schwartz interview Jamie Cameron about Webmin on the FLOSS weekly podcast was a delight, especially as in my dark days I had a fair use for Webmin, so much so that I bought Jamie’s “Using Webmin” book.

Now that I have my walking method sussed all I need to do is load up my iPod and get an early night and a good sleep beforehand – unfortunately the early night and good sleep was thwarted last night by some jQuery and AJAX goodness and also by Zoe night-babbling, then Linus who woke up crying because “Ada ate my pie in a dream!” and then some more Zoe babbling. Things had quietened down by about 4am anyway.

Check out the local Channel Nine TV coverage at http://www.ch9.co.nz/node/14223

MagentoCommerce is an open-source enterprise level ecommerce system suitable for high end ecommerce websites. We’re in the throes of developing a large solution for a local company, and the developers involved in provisioning the site are installing local copies of Magento on their Apache servers to test with.

Here’s a list of issues that I’ve come across with installing Magento, and how I’ve solved them. For the record I’m using Magento 1.3.2.2 (released 19.07.2009)

1. Can’t get past the second install step

This is the step that asks you for your database password. If the database you’re specifying doesn’t exist then you can’t proceed. I’m sure there’s a bug here because it’s like an error message isn’t displayed. Would love to dig into this but our project is on a tight time frame. MAKE SURE YOU CREATE THE DATABASE YOURSELF FIRST (no tables required).

2. Can’t log in after installation

If after logging in you go to the /admin directory and you can’t log in, it’s probably because of the URL that you used to install it. I tried installing it to http://localhost/, http://127.0.0.1/ and http://192.168.2.10/ (my IP) all with the same result. After googling it seemed to be something to do with Magento refusing to set a cookie for a top-level domain (or somesuch). Workarounds are to use a host name that has a dot in it, e.g. http://localhost.localdomain/ (or in my case, http://bob-desktop.local). Add this to your hosts file, you may have to restart your browser for it to pick it up.

3. Firefox keeps asking you to download a PHTML file.

After doing a completely fresh install of Apache2, PHP5, mysql, Magento etc on a fresh machine (Ubuntu 9.04, Firefox 3.0.11) Firefox kept asking me what I wanted to do with the PHTML file when accessing the freshly untarred Magento directory. It took me ages to find an answer to this, but it was as simple as clearing your cache in Firefox. I dicked around with the Apache configuration, permissions, .htaccess files etc and finally found a comment about the Firefox cache.

4. AllowOverride All

Magento uses a .htaccess file in the root directory. I noticed that the default for /var/www is AllowOverride None which prevents Apache from looking at the content of .htaccess files. While this didn’t cause me trouble, I did set it to “AllowOverride All” while trying to solve #3. The file is /etc/apache2/sites-enabled/000-default, line 11. Note there may be security implications of doing this if you’re the administrator of a shared hosting environment and as such you should read about the AllowOverride directive.

Installation requirements:

From a stock Ubuntu 9.04 machine I had to install the following packages to support Magento:

• libapache2-mod-php5
• php5-curl
• php5-mcrypt
• php5-mysql
• php5-gd
• mysql-server

Also, I have this command on standby to reset the Magento environment back to scratch (deletes the magento directory, drops the database, untars the source file and creates the empty database again)

cd /var/www ; rm -Rf magento ; tar xvfz /home/bob/magento-1.3.2.2.tar.gz ;
   chmod -R a+rw magento ; mysqladmin -uroot -proot -f drop magento ;
   mysqladmin -uroot -proot create magento

That’s all for now.

I’ve been using jQuery a lot recently and it occurred to me that I could use jQuery’s selectors to target the information that I’m interested in a web page, and then using Ajax POST it to my own script that would be ready waiting to then do something useful with the data, e.g. validate it and save it in a database.  For educational purposes I was keen to keep this completely client-side if possible (except for a script to receive the information).  See later on for a server-side solution.

The situation I was up against was a page that had a heap of data in a table (about 90 items), but the table was interspersed with random images to split it up and make it more pleasing to the eye.  Fortunately for me, all of the data that I wanted was neatly wrapped in <div class=”information”></div> tags.  Selecting these div tags with jQuery is really easy by using $(‘div.information’).

My first problem was that in order to use jQuery, the web page you’re looking at has to be using it.  Fortunately there’s a quick bookmarklet called jQuerify that allows you to load jQuery onto any web page.  Once you’ve got that then you can write further bookmarklets of your own to do stuff.

So, my evil evil plan was to combine a jQuery selector, jQuery’s each() construct, and jQuery’s ajax support to post the content of each div to a “scraper” script, like so:

$('div.information').each(function(){
  $.post('http://localhost/scraper.php',{
    data: this.innerHTML
  });
});

I loaded my source page, clicked the jQuerify bookmarklet and then pasted the code above into the Firebug console (what, oh you’ll need that…) and it was flawless … except that the browser security model stepped in and prevented the ajax call because the XHTTPRequest object is not allowed to post information from one domain to another.  I was stuck – I googled around for a while looking for workarounds, and investigated the use of JSONP but the transport method seemed more weighted at retrieving information rather than posting it.

So, I was stuck with a simple question: “How can I get information from one site to another by using the browser?” – the simplest answer to this question is of course to have a form on the source website, that when submitted posts to the target.  Thanks to the power of JavaScript, modifying the DOM of a loaded web page is a doddle.  Therefore it should be simple to create a form on the page after it has loaded (client side, remember), create and populate some form fields with data and then submit the form to my scraper script.

Suddenly my intentions had outgrown a bookmarklet, but I would still need one for jQuerify and one for my “Scraper Utils”.  My new bookmarket simply asked jQuery to load a local JavaScript file in exactly the same was that jQuery was loaded in the first place:

javascript:$.getScript('http://localhost/scraper.js');

Now I had the freedom of writing chunk loads of stuff in my local scraper.js file.

Scraper = {};
Scraper.createForm = function()
{
  var form = document.createElement('form');
  form.setAttribute('method', 'POST');
  form.setAttribute('action', 'http://localhost/scraper.php');
  document.getElementsByTagName('body')[0].appendChild(form);
  return form;
}
 
Scraper.createSubmitButton = function()
{
  var button = document.createElement('input');
  button.setAttribute('type', 'submit');
  return button;
}
 
Scraper.createFormField = function(name)
{
  var field = document.createElement('textarea');
  field.setAttribute('name', name);
  field.setAttribute('rows', 10);
  field.setAttribute('cols', 50);
  return field;
}		
 
var ScraperForm = Scraper.createForm();
$('div.information').each(function(){
  var field = ScraperForm.appendChild(Scraper.createFormField('data[]'));
  field.value = this.innerHTML;
});
// Create a field that we can post with:
ScraperForm.appendChild(Scraper.createSubmitButton());

You can see here that I’ve set up a few functions, createForm(), createFormField(), createSubmitButton() and then at the bottom I wrap them all together with the $(‘div.information’).each(…) construct.  The end result of this is that when I click my bookmarklet that includes the scraper.js script, a form is created at the bottom of the page and a textarea for each div.information is created that holds the innerHTML from that div.

Then, by clicking the Submit button, the browser posts all of that information across to http://localhost/scraper.php where I then collect the information from $_POST['data'] and poke it into a database.

It’s pretty rough and ready but could easily be extended to do other things like allow you to specify the selector and target URL for the post when you click the Scraper bookmarket.

Server Side Solution

On my travels I also came across the “PHP Simple HTML DOM Parser” which claims a similar ability like so:

// Create DOM from URL or file
$html = file_get_html('http://www.google.com/');
 
// Find all images
foreach($html->find('img') as $element)
       echo $element->src . '<br/>';
 
// Find all links
foreach($html->find('a') as $element)
       echo $element->href . '<br/>';

You can get a hold of this from Sourceforge at the PHP Simple HTML DOM Parser website.

What is fail2ban?

It’s a python script (that runs as a daemon) which monitors log files in your /var/log file.  It monitors them for specific entries, for example “Failed password”, and then updates iptables rules to deny network access for the offending IP for a configured amount of time.

A good example of this is that if you try to ssh into my system three times unsuccessfully, you won’t be able to try again for 10 minutes.  This is sufficient to make automated brute force attacks useless.

Do you need it?

If you have a public-facing server with the ability to log into it (including web applications even) then you need this.  If you’re curious to see if you’ve been targeted for attacks, try running these commands as root on your server:

cat /var/log/auth.log | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort | uniq -c
zcat /var/log/auth.log* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort | uniq -c

The first command examines your current auth.log file and the second examines your historical auth.log.[0-9] files. In my recent history (prior to configuring fail2ban properly) I had over 6,000 failed SSH login attempts on a single day just after Christmas.

What next?
The steps are:

bob@server:~$ sudo apt-get install iptables fail2ban
bob@server:~$ sudo /etc/init.d/fail2ban start

Now, you can check to see if it’s working by “pinging” the service:

bob@server:~$ sudo fail2ban-client ping
Server replied: pong

And you can get information on what’s currently been banned by examining the ssh “jail” – the jail is term used to describe the configuration and current black list for access from remote hosts:

bob@server:~$ sudo fail2ban-client status
Status
|- Number of jail:	1
`- Jail list:		ssh
bob@server:~$ sudo fail2ban-client status ssh
Status for the jail: ssh
|- filter
|  |- File list:	/var/log/auth.log
|  |- Currently failed:	0
|  `- Total failed:	52
`- action
   |- Currently banned:	0
   |  `- IP list:
   `- Total banned:	7

To test everything is working, simply try to log into your system incorrectly three times.  When you’ve done this and you look at the results of “fail2ban-client status ssh” you will see your remote IP in the list.  To unblock your IP, simply restart the fail2ban daemon (i.e. sudo /etc/init.d/fail2ban restart)

These pages were very useful when reading about fail2ban:

• The Art of Web ~ System: fail2ban and iptables
• The manual for fail2ban 0.8 (very readable)

#1: A Stein-smashingly good time!

#2: I have days like these …

#3: Just too true

#4: Fandom

#5: White-belly-making-cereal

Rt. Hon Steven Joyce
Minister for Communications and Information Technology
Care of Distribution Services
Parliament Buildings
Wellington 6160

Dear Steven,

I am writing to you in urgency regarding the proposed Section 92 of the Copyright Amendment Act that assumes guilt upon accusation, forcing the termination of internet connections and websites without evidence, a fair trial, or punishment for false acusations of copyright infringement.

I work for a web development company and cannot let this “roll over” into law without voicing my protest. To know that any one of our clients, or indeed ourselves could be falsely accused without proof required and have our internet access terminated would have dire ramifications as our livelihoods and futures rely on providing these services.

I urge you to carefully consider the ramifications of the proposed Section 92 and speak out on behalf of myself and others who have contacted you regarding this.

I am available for further comment if required.

Yours Sincerely

Bob Brown
Web Developer
(my address)
Dunedin