Have been busy recently comparing Rails 1.1’s integration testing and WATIR and selenium for end-to-end testing a rails application.
The testing scenario is that the webserver is running on a remote Linux box in US, to which we have ssh access.
And we are testing the application from India. So we cannot run the browser on that machine.
We are writing test cases for various parts of the application using each of these tools and then we would compare notes.
A few things however are clear rightaway.
WATIR will test only for IE on windows. Firefox support for watir called firewatir is being built, and is not robust as of now. Also we might need different sets of test scripts for firefox and IE.
Selenium is the easiest to get started in the BrowseBot FITRunner mode; haven’t tried the “driven” mode yet. FIT mode would not allow us to verify values on the screen against those in the db… but then probably those tests should go into the “controller testing”(functional testing) and not be at this level. When we want to test against db for multiple controllers then rails’ integration testing is the probably the best way.
Running selenium in “driven” mode will take some research, for which i have not been able to spare time.
WATIR requires ruby and watir gems to be installed on the client computer. Hence if we want to share UI tests with our non-technical client, Selenium is better since it requires no installation.
From the aspect of writing test cases, WATIR is better than selenium’s FIT mode. Selenium test cases need to be written in html. And there is no control flow in the selenium core, though there is a user contributed library for this.
For certain kind of testing, such as against db values and checking values in session, Rails functional tests and integration tests are great.
Will share more experiences on this when we are through with the exercise.
How has your experience with these tools been?
***********************************
Received this profound quotation in my email today.
“The greatest good you can do for another is not just to share your riches, but to reveal to him his own.”
Benjamin Disraeli
1804-1881, Former British Prime Minister
***********************************
If you don’t want to write your Selenium tests in HTML you can check my plugin out, Selenium on Rails, which lets you write them in a couple of formats.
http://andthennothing.net/archives/2006/02/19/new-version-of-selenium-on-rails
Thanks Jonas, I already had “selenium on rails” delicioused.
Will definitely look into it.
Cool stuff!
The original motivation (on an ssh box, can’t view local browser) was one thing I didn’t get. If that was your problem, why didn’t you just upon up port 80 and test from your browser?
Of course, just functionally testing with your browser isn’t enough. You need automated testing at every level, and that’s where tools like watir and selenium come in.
Are you having success at doing actual test-driven development? I find it’s a tradition honored more in the breach than in the observance … my tests always get written AFTER my code, no matter how much I intend to write them first!
Jon,
That was not the original motivation … but a constraint… we have webrick running on that linux box and can access the app from our browser.
What I was looking for was a way to validate values on the browser screen against those in the db. To get values directly from the db, I needed to have some scripts running outside the web server, accessing the db directly. These scripts needed to be embedded in the testing tool driving the browser.
But now i realize, that in wanting to do that I was violating the DRY principle. When i already test those values in the controller’s functional test, I rather not do them again in selenium.
As confused as I may sound, I am beginning to see a separate role for these testing tools such as selenium, separate from Rails inbuilt testing framework that is. It is not actually an either-or situation.
About following TDD, we also write tests after the code. What we have found is that writing a model followed by it’s unit tests and writing a controller followed by it’s functional tests is the best way it works for us. Though some hardcore TDD guys advocate writing test cases before the actual code, we have never even given that a try till date.
Venkat, the author of Practices of an Agile Developer , has described this process in a very nice way in this blog post “Walking along the development beach” .