Every programmer knows the only one way to make sure you get some sleep is to write software that never breaks (or rarely!). When a project gets large enough, it becomes impossible for a human being to carry out enough tests to ensure that even the smallest change to the code base doesn't create a bug somewhere in the system. That's why testing suites were created, and there's finally one available for the Magento E-commerce platform!
This article is a technical tutorial
If you're still wondering what the benefits of using a testing framework are, then you're not reading the right article, since we're absolutely sold on this, and have been for a while. If you're looking for a technical guide to setting a server for running tests using the Magento Test automation Framework, then keep reading!
A Magento Testing Framework is born!
We're big fans of testing here at Doghouse Media and have been using PHPUnit and Selenium for a while now with our custom code. Though we had tried writing a few tests for Magento, we never really found the time to develop a proper test suite for the e-commerce beast Magento. Since last week, things have now changed for Magento/X-commerce agencies throughout the world. If you're in Western Australia, drop by our office to see this in action!
Creating the server
So far, each developer has been installing Selenium on their own computer, but as we've been moving towards a unified development and production environment, we decided to set-up a dedicated testing server. Over the next few weeks, we'll be implementing continuous integration using one of the great solutions out there. Skip to the end of the article to get more details. We've been using VMware for our production environment for a while now, and have started using the community version for our local development environment. One of our goals was being able to clone our testing server and use it from our production environment too, as well as clone it to remote offices. Using VMware and creating a new host made perfect sense. NOTE: Running the test suite establishes a lot of connections! Tests will run much faster if both servers are on the same network. If you're planning on testing remote sites, hosting your test server on your local network will be slow. Whatever you're using, whether it be a virtualised environment or a standalone box, start off by installing Ubuntu Server. Install the 64bit version if your hardware supports it. We set our host with 4Gb of RAM, a 30Gb HDD, and 2 virtual CPU cores. Once you're system is up and running, and you've got a command line prompt ready, you're set to go!
Installing the software
This is very straight-forward so I won't bore you with lengthy explanations. Basically, we're installing all the prerequisites for Selenium and PHPUnit. You should be able to install all of the as one command, but for simplicity we've listed each command separately.
$ sudo apt-get install openssh-server $ sudo apt-get install mysql-server $ sudo apt-get install apache2 $ sudo apt-get install php5 php5-cli $ sudo apt-get install php5-curl $ sudo apt-get install php-pear $ sudo apt-get install phpunit $ sudo apt-get install ant ant-optional $ sudo apt-get install git-core
Installing Java on Linux is not as easy as it seems! Thanks to the excellent work by Martin Wimpress, this process is really easy for Ubuntu and Debian using his OAB-Java6 installer.
$ cd ~ $ wget https://raw.github.com/flexiondotorg/oab-java6/master/oab-java6.sh -O oab-java6.sh $ chmod +x oab-java6.sh $ sudo ./oab-java6.sh $ sudo apt-get install sun-java6-jre
PEAR & PHPUnit
$ sudo pear upgrade $ sudo pear channel-discover pear.phpunit.de $ sudo pear channel-discover pear.symfony-project.com $ sudo pear channel-discover components.ez.no $ sudo pear install --alldeps phpunit/PHPUnit $ sudo pear install --alldeps phpunit/PHPUnit_Selenium $ sudo pear install --alldeps phpunit/DbUnit
Selenium will need to launch a browser to run the tests. You either need to install a graphical interface and launch the tests from the GUI (using a physical screen or remote desktop), or if you're planning on using SSH to run the tests, you'll need to allow Selenium to run headless. Choose one of the following solutions, or as we did, use both!
Installing a Graphical User Interface and Remote Desktop
$ sudo apt-get install --no-install-recommends ubuntu-desktop $ sudo apt-get install xrdp
$ sudo apt-get install xvfb (run headless)
$ sudo mkdir -p /opt/selenium $ cd /opt/selenium $ sudo wget http://selenium.googlecode.com/files/selenium-server-standalone-2.19.0.jar $ sudo chmod +x selenium-server-standalone-2.19.0.jar $ java -jar selenium-server-standalone-2.19.0.jar > Ctrl-c
Installing the Magento TAF
Please note, the Magento TAF guide is using the wrong GIT repo url. Use the up-to-date one from Github!
$ cd /opt $ sudo git clone git://github.com/magento/taf.git magento_taf
The latest versions of Chrome are supported by Selenium. Perfect! Magento TAF (and Selenium in general) has some issues capturing screen-shots with Chrome,
but installing Firefox is not as easy as it seems, so let's start with this.
$ cd ~ $ wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb $ sudo dpkg -i google-chrome-stable_current_amd64.deb You'll also need to download ChromeDriver for Selenium. Download the correct ZIP file, unzip it, and place the binary in your path. We chose /usr/local/bin/chromedriver. Read more about ChromeDriver and Selenium
[UPDATE March 27th 2012] Today, we updated packages on our testing machine, and for the fun of it, tried running the tests with Firefox. To our bigest surprise, it worked! I'm not sure whether it always worked or whether one of the updates fixed it, but now we're getting screen-shots!
sudo apt-get update
sudo apt-get upgrade Read on to see how to configure PHPUnit to use Firefox.
Selenium only supports Firefox 3.6. Ubuntu and Debian now come with the latest versions of Firefox, so you need to tell your operating systems to use different sources. So far, we haven't found a perfect solution, so you're on your own for this one. We are still looking and will update this post once we're happy with our implementation. Please leave your comments.
Configure Magento TAF
Further down the line, we'll set-up a way to run the tests on different sites without having to edit the configuration files, but for now, let's hard-code our test website.
$ cd /opt/magento_taf $ sudo cp config/config.yml.dist config/config.yml $ sudo cp phpunit.xml.dist phpunit.xml
Edit the configuration
$ sudo vim config/config.yml
Since we haven't installed Firefox yet, we need to tell our test to use Google Chrome. [UPDATE March 27th 2012] Now that we've got both Firefox and Chrome working, you can tell PHPUnit which one to use! Edit line 50 accordingly. Use *chrome for Firefox:
default: *chrome Or use *googlechrome for Chrome:
Towards the end of the document, you need to change at least the following settings:
applications > magento > adminLogin
applications > magento > adminPassword
applications > magento > areas > admin > url
applications > magento > areas > frontend > url
NOTE: If your store does not have a valid SSL cert, make sure you change your secure URL setting in Magento to use HTTP instead of HTTPS. If you don't, the test suite will not be able to access your secure pages and will report errors.
$ java -jar /opt/selenium/selenium-server-standalone-2.19.0.jar & $ /opt/magento_taf/runtests.sh
If you're using Chrome for testing, Selenium will complain since it can only take screen-shots with Firefox. In ./lib/Mage/Selenium/TestCase.php, change the following.
protected $captureScreenshotOnFailure = true; to:
protected $captureScreenshotOnFailure = false;
We'll address this later by getting Firefox to work!
Running the tests
Start the Selenium server
Running headless? Create a display.
If you're running headless, you'll need to create a display for selenium to run. Ignore this if you're running through remote desktop or from a physical screen.
Xvfb :99 -ac & export DISPLAY=:99
Launch the Selenium server
$ java -jar /opt/selenium/selenium-server-standalone-2.19.0.jar -Dchrome.binary=/opt/google/chrome/chrome &
Run the test
You might want to understand the output will the test is running. Open this page from the PHPUnit manual in a browser window, then run the test.
Running tests from your IDE
The Magento TAF Installation Guide explains how to run tests from NetBeans or PhpStorm. We use ZendStudio and have been looking for a way to integrate the IDE with PHPUnit. Here are the two guides we've found. We haven't found the time to try these guides out yet, but I'm sure you will, so let us know in the comments!
Configure Selenium as a service
Dan Straw wrote a nice example of an init.d script for Selenium. You'll have to adjust for your environment (don't forget the chrome.library parameters if you're using Chrome). We also added a few lines to ours, to ensure xvfb is running. At the top of the file:
Xvfb :99 -ac & export DISPLAY=:99 http://www.danstraw.com/blog/wp-content/uploads/2010/11/selenium.txt
We would like to turn our test server into a continuous integration server. Here are some good links to get this working with PHP projects.
Something else, we would like to investigate is code quality management. Check out Sonar!
Check back soon
We hope to try these technologies out, and integrate them with our Magento TAF set-up, so check back soon!
Here are some of the resources we used to get things working.