How to install and run the Magento Test Automation Framework

Every programmer knows the only one way to make sure you get some sleep is to write software that never breaks (or rarely!). When a project gets large enough, it becomes impossible for a human being to carry out enough tests to ensure that even the smallest change to the code base doesn't create a bug somewhere in the system. That's why testing suites were created, and there's finally one available for the Magento E-commerce platform!

This article is a technical tutorial

If you're still wondering what the benefits of using a testing framework are, then you're not reading the right article, since we're absolutely sold on this, and have been for a while. If you're looking for a technical guide to setting a server for running tests using the Magento Test automation Framework, then keep reading!

A Magento Testing Framework is born!

We're big fans of testing here at Doghouse Media and have been using PHPUnit and Selenium for a while now with our custom code. Though we had tried writing a few tests for Magento, we never really found the time to develop a proper test suite for the e-commerce beast Magento. Since last week, things have now changed for Magento/X-commerce agencies throughout the world. If you're in Western Australia, drop by our office to see this in action!

Creating the server

So far, each developer has been installing Selenium on their own computer, but as we've been moving towards a unified development and production environment, we decided to set-up a dedicated testing server. Over the next few weeks, we'll be implementing continuous integration using one of the great solutions out there. Skip to the end of the article to get more details. We've been using VMware for our production environment for a while now, and have started using the community version for our local development environment. One of our goals was being able to clone our testing server and use it from our production environment too, as well as clone it to remote offices. Using VMware and creating a new host made perfect sense. NOTE: Running the test suite establishes a lot of connections! Tests will run much faster if both servers are on the same network. If you're planning on testing remote sites, hosting your test server on your local network will be slow. Whatever you're using, whether it be a virtualised environment or a standalone box, start off by installing Ubuntu Server. Install the 64bit version if your hardware supports it. We set our host with 4Gb of RAM, a 30Gb HDD, and 2 virtual CPU cores. Once you're system is up and running, and you've got a command line prompt ready, you're set to go!

Installing the software

This is very straight-forward so I won't bore you with lengthy explanations. Basically, we're installing all the prerequisites for Selenium and PHPUnit. You should be able to install all of the as one command, but for simplicity we've listed each command separately.

Prerequisites

Basics

$ sudo apt-get install openssh-server $ sudo apt-get install mysql-server $ sudo apt-get install apache2 $ sudo apt-get install php5 php5-cli $ sudo apt-get install php5-curl $ sudo apt-get install php-pear $ sudo apt-get install phpunit $ sudo apt-get install ant ant-optional $ sudo apt-get install git-core

Java

Installing Java on Linux is not as easy as it seems! Thanks to the excellent work by Martin Wimpress, this process is really easy for Ubuntu and Debian using his OAB-Java6 installer. $ cd ~ $ wget https://raw.github.com/flexiondotorg/oab-java6/master/oab-java6.sh -O oab-java6.sh $ chmod +x oab-java6.sh $ sudo ./oab-java6.sh $ sudo apt-get install sun-java6-jre

PEAR & PHPUnit

$ sudo pear upgrade $ sudo pear channel-discover pear.phpunit.de $ sudo pear channel-discover pear.symfony-project.com $ sudo pear channel-discover components.ez.no $ sudo pear install --alldeps phpunit/PHPUnit $ sudo pear install --alldeps phpunit/PHPUnit_Selenium $ sudo pear install --alldeps phpunit/DbUnit

Graphics

Selenium will need to launch a browser to run the tests. You either need to install a graphical interface and launch the tests from the GUI (using a physical screen or remote desktop), or if you're planning on using SSH to run the tests, you'll need to allow Selenium to run headless. Choose one of the following solutions, or as we did, use both!

Installing a Graphical User Interface and Remote Desktop

$ sudo apt-get install --no-install-recommends ubuntu-desktop $ sudo apt-get install xrdp

Running Headless

$ sudo apt-get install xvfb (run headless)

Installing Selenium

$ sudo mkdir -p /opt/selenium $ cd /opt/selenium $ sudo wget http://selenium.googlecode.com/files/selenium-server-standalone-2.19.0.jar $ sudo chmod +x selenium-server-standalone-2.19.0.jar $ java -jar selenium-server-standalone-2.19.0.jar > Ctrl-c

Installing the Magento TAF

Please note, the Magento TAF guide is using the wrong GIT repo url. Use the up-to-date one from Github! $ cd /opt $ sudo git clone git://github.com/magento/taf.git magento_taf

Installing browsers

Chrome

The latest versions of Chrome are supported by Selenium. Perfect! Magento TAF (and Selenium in general) has some issues capturing screen-shots with Chrome, but installing Firefox is not as easy as it seems, so let's start with this. $ cd ~ $ wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb $ sudo dpkg -i google-chrome-stable_current_amd64.deb You'll also need to download ChromeDriver for Selenium. Download the correct ZIP file, unzip it, and place the binary in your path. We chose /usr/local/bin/chromedriver. Read more about ChromeDriver and Selenium

Firefox

[UPDATE March 27th 2012] Today, we updated packages on our testing machine, and for the fun of it, tried running the tests with Firefox. To our bigest surprise, it worked! I'm not sure whether it always worked or whether one of the updates fixed it, but now we're getting screen-shots! sudo apt-get update sudo apt-get upgrade Read on to see how to configure PHPUnit to use Firefox. Selenium only supports Firefox 3.6. Ubuntu and Debian now come with the latest versions of Firefox, so you need to tell your operating systems to use different sources. So far, we haven't found a perfect solution, so you're on your own for this one. We are still looking and will update this post once we're happy with our implementation. Please leave your comments.

Configure Magento TAF

Further down the line, we'll set-up a way to run the tests on different sites without having to edit the configuration files, but for now, let's hard-code our test website. $ cd /opt/magento_taf $ sudo cp config/config.yml.dist config/config.yml $ sudo cp phpunit.xml.dist phpunit.xml

Edit the configuration

$ sudo vim config/config.yml

Default browser

Since we haven't installed Firefox yet, we need to tell our test to use Google Chrome. [UPDATE March 27th 2012] Now that we've got both Firefox and Chrome working, you can tell PHPUnit which one to use! Edit line 50 accordingly. Use *chrome for Firefox: default: *chrome Or use *googlechrome for Chrome: default: *googlechrome

Site details

Towards the end of the document, you need to change at least the following settings:

  • applications > magento > adminLogin
  • applications > magento > adminPassword
  • applications > magento > areas > admin > url
  • applications > magento > areas > frontend > url

NOTE: If your store does not have a valid SSL cert, make sure you change your secure URL setting in Magento to use HTTP instead of HTTPS. If you don't, the test suite will not be able to access your secure pages and will report errors. $ java -jar /opt/selenium/selenium-server-standalone-2.19.0.jar & $ /opt/magento_taf/runtests.sh

Configuring screen-shots

If you're using Chrome for testing, Selenium will complain since it can only take screen-shots with Firefox. In ./lib/Mage/Selenium/TestCase.php, change the following. protected $captureScreenshotOnFailure = true; to: protected $captureScreenshotOnFailure = false; We'll address this later by getting Firefox to work!

Running the tests

Start the Selenium server

Running headless? Create a display.

If you're running headless, you'll need to create a display for selenium to run. Ignore this if you're running through remote desktop or from a physical screen. Xvfb :99 -ac & export DISPLAY=:99

Launch the Selenium server

$ java -jar /opt/selenium/selenium-server-standalone-2.19.0.jar -Dchrome.binary=/opt/google/chrome/chrome &

Run the test

You might want to understand the output will the test is running. Open this page from the PHPUnit manual in a browser window, then run the test. $ /opt/magento_taf/runtests.sh

Running tests from your IDE

The Magento TAF Installation Guide explains how to run tests from NetBeans or PhpStorm. We use ZendStudio and have been looking for a way to integrate the IDE with PHPUnit. Here are the two guides we've found. We haven't found the time to try these guides out yet, but I'm sure you will, so let us know in the comments!

  • http://blog.bauffman.be/2009/10/14/running-phpunit-tests-in-zend-studio/
  • http://seleniumhq.org/docs/appendix_installing_java_driver_client.html

Further reading

Configure Selenium as a service

Dan Straw wrote a nice example of an init.d script for Selenium. You'll have to adjust for your environment (don't forget the chrome.library parameters if you're using Chrome). We also added a few lines to ours, to ensure xvfb is running. At the top of the file: Xvfb :99 -ac & export DISPLAY=:99 http://www.danstraw.com/blog/wp-content/uploads/2010/11/selenium.txt

Continuous Integration

We would like to turn our test server into a continuous integration server. Here are some good links to get this working with PHP projects.

Code quality

Something else, we would like to investigate is code quality management. Check out Sonar!

Check back soon

We hope to try these technologies out, and integrate them with our Magento TAF set-up, so check back soon!

Resources

Here are some of the resources we used to get things working.

Comments

¬°Hurra for Firefox! :D

Hello! I,ve ran selenium with following comment:
java -jar selenium-server-standalone-2.24.1.jar -Dwebdriver.chrome.driver=/usr/local/bin/chromedriver/chromedriver

And when I'm trying to run my test, I've got:

PHP Fatal error: Uncaught exception 'InvalidArgumentException' with message 'Unable to parse file "/opt/magento_taf/config/browsers.yml": Reference "googlechrome" does not exist (default: *googlechrome).' in /usr/share/php/SymfonyComponents/YAML/sfYaml.php:100
Stack trace:
#0 /opt/magento_taf/lib/Mage/Selenium/Helper/File.php(52): sfYaml::load('/opt/magento_ta...')
#1 /opt/magento_taf/lib/Mage/Selenium/TestConfiguration.php(387): Mage_Selenium_Helper_File->loadYamlFile('/opt/magento_ta...')
#2 /opt/magento_taf/lib/Mage/Selenium/TestConfiguration.php(264): Mage_Selenium_TestConfiguration->_loadConfigData()
#3 /opt/magento_taf/lib/Mage/Selenium/TestConfiguration.php(163): Mage_Selenium_TestConfiguration->_initConfig()
#4 /opt/magento_taf/lib/Mage/Selenium/TestConfiguration.php(147): Mage_Selenium_TestConfiguration->init()
#5 /opt/magento_taf/bootstrap.php(44): Mage_Selenium_TestConfiguration::initInstance()
#6 /usr/share/php/PHPUnit/Util/Fileloader.php(95): include_once('/opt/magento_ta...')
#7 /usr/share/php/PHPUnit/Util/Filel in /usr/share/php/SymfonyComponents/YAML/sfYaml.php on line 100

What am I doing wrong?

Hi Kh4mell, I'm not sure. I don't have access to our install right now, but I can't recall a browsers.yml file (there was a file in config/config.yml though). The file it should look like is this one: https://github.com/magento/taf/blob/master/config/config.yml.dist If that doesn't help, put your browsers.yml in a pastebin and share the link here. I can try to have a look tomorrow morning.

Had a similar problem, doing:

sudo pear install symfony/YAML

solved my problem.

Hi jonathan,
Here is my browsers.yml: http://pastebin.com/NjqcJ1c5 . The reason why I was trying setup mtaf to work with chromium is I can't force mtaf to capturing screenshots.There's no config.yml in my config directory. I set $captureScreenshotOnFailure to true on TestCase.php file. Is there any other place, where should I enable capturing screenshots?

It doesn't look like you're using Magento TAF from Github. https://github.com/magento/taf

Try with that and let me know. Remember, screen-shots only work with Firefox!

After install mtaf from github I need to change #!/bin/sh in runtest.sh to #!/bin/bash, and now I have screenshots! Time to debug ;) Thank you very much Jonathan!

Hi,
Can you give me some points, how to add new test to MTAF. I made new directory in testsuite/Core/Mage and put my CreateTest.php there.
In phpunit.xml:

/opt/magento_taf/testsuite/Core/Mage/LppCustom/CreateTest.php

I got an error:
./runtests.sh
PHP Fatal error: Uncaught exception 'PHPUnit_Framework_Exception' with message 'Neither "LppCustom Tests.php" nor "LppCustom Tests.php" could be opened.' in /usr/share/php/PHPUnit/Util/Skeleton/Test.php:100

Hi Kh4mell, Not sure. I haven't tried that yet. A quick Google search seems to bring up some relevant results. Hope this helps: https://www.google.com/search?q=PHP+Fatal+error%3A+Uncaught+exception+'PHPUnit_Framework_Exception'+with+message+'Neither

I use Chrome and Firefox browsers. Firebox browser is good for test Automation Framework for Magneto.

Hello! Thanks for the tutorial, Jonathan! it saved me tons of time and efforts!
In my turn, I'd like to share a way to quickly test magento for any errors. It is shopping cart diagnostics service. I like the way all the possible tests are bundled together, so you don't need to seek different ones for each store element.

Hi Bruce,
Thanks for the comment!

I removed your link given it looked like a disguised ad! Can you point us to a community review of the product you're mentioning?

Cheers

Hello, Jonathan! No problem, if you remove the link. I just felt like sharing some tip that helps me a lot.

Here's the magento community link to this service: http://www.magentocommerce.com/magento-connect/free-shopping-cart-error-...

What do you think about it?

Not sure Bruce. It looks like a good tool. Hopefully someone will give it an in-depth review soon!

Hi. Could you explain how mtaf connects uimap, data and tests? For examle:
We have the structure magento_taf:
mytests (my custom tests)
-----------core
-----------------Mage
------------------------Customer
--------------------------------------RegisterTest.php
fixture:
--------default (default magento data and uimaps)
...
--------mydata (my data and uimaps)
------------------core
------------------------Mage
--------------------------------Customer
---------------------------------------------data
---------------------------------------------------Customer.yml
---------------------------------------------uimap
-----------------------------------------------------admin
-------------------------------------------------------------Customer.yml
-----------------------------------------------------frontend
---------------------------------------------------------------Customer.yml
---------------------------------------------------------------MyAccount.yml
myphpunit.xml (customized xml)

myphpunit contais:
.....

mytests/Core/Mage/Customer

.....

How can I tell to mtaf connect my files:
mytests/Core/Mage/Customer/RegisterTest.php
fixture/mydata/Core/Mage/Customer/data/Customer.yml
fixture/mydata/Core/Mage/Customer/uimap/admin/Customer.yml
fixture/mydata/Core/Mage/Customer/uimap/frontend/Customer.yml?

Hi Julia,

I'm not sure; I haven't played with custom tests and/or uimap yet.

A quick Google brought up the following document. I'm sure you've read through it already but I'm afraid I can't be a lore more helpful. Maybe try hanging out on IRC and if you're lucky you might find someone knowledgeable enough!

Let me know what you find.

http://www.magentocommerce.com/images/uploads/download/MagentoTestAutoma...

Nice article. Thanks for sharing your experience. I am just getting started on a VERY similar approach to include automated continuous integration tests in our development process. I am wondering how did this approach worked for you guys, if you are still using it and if you ever tried this method while having multiple projects going on. Each with their own sets of fixtures, uimaps and tests

Hi Barbazul,

To be really honest, we haven't found the time to integrate it into our workflow yet (even though the company has practically doubled in size since the post was published!).

We've just recently set-up Jenkins CI and got static code analysis working. Adding the Magento Testing Framework to our CI server is on the roadmap but I don't know when we'll finally get to it.

When we do, I'll be sure to update the post.

Wow MTAF-Jenkins integration would make for an awesome blog post if you ever get to do it!
We are almost exclusively doing Magento work, so we are approaching this in a more "homebrew scripting" kind of way using git to pull code to the testing server and crontab to run the tests. We still have to figure out a couple of issues involving the extensibility of uimaps to play nice with different modules and themes and better support to test multiple projects with global and local testsuites, but we are getting there

Add new comment

Filtered HTML

  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>, <c>, <cpp>, <drupal5>, <drupal6>, <java>, <javascript>, <php>, <python>, <ruby>. The supported tag styles are: <foo>, [foo].
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockcode>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.