GeeCON Prague 2014*
The moose likes Testing and the fly likes New Automation Testing Tool Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


JavaRanch » Java Forums » Engineering » Testing
Reply locked New topic
Author

New Automation Testing Tool

Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
I just released Jameleon into public domain last week. It is a whole new idea that really saves time and effort in automation.
Please come check it out and tell me what you think.
I am updating and adding new documentation on a daily basis. The build/install is very easy.
I have a sample web application that comes with Jameleon that I use as examples. The current tutorial should get you up and running.
I am really interested to hear what people have to say about it.
Simon Brown
sharp shooter, and author
Ranch Hand

Joined: May 10, 2000
Posts: 1913
    
    6
That is actually very good timing. I'm just writing a section about testing Java based web applications for a new Apress book so I'll be sure to drop in a reference.
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
How does Jameleon compare to FitNesse?
[ July 04, 2003: Message edited by: Ilja Preuss ]

The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
I don't know if you can compare Jameleon to FitNess. Jameleon is an acceptance testing framework. However, you could do integration testing with it, I haven't tried it.
I looked into FIT and FitNess before writing the tool and they just didn't seem to fit into the QA world as well as they do in the development world.
The front page of Jameleon explains the problems its solves.
Most people that haven't actually tried it think that Jameleon is complicated. That's probably because of the way I explained it. Actually, it's quite simple. I think the hardest thing is Jameleon makes you think before you start down the road of automation.
The steps to writing a Jameleon Function point are:
1) Write the function points code in Java.
2) Register your new code in the macro provided.
3) If Jameleon doesn't know about the application yet, register it.
The steps to writing a test-case are:
1) Organize your function points in the macro.
We are testing a few applications with it and it has actually saved us a lot time over just using HttpUnit or JWebUnit.
It's pretty cool to say "I need to write a function point for change email" and not have to worry about writing code for login, verify password, and navigate to change email. When you start writing a function point, you assume you are already at the page you need to be at to test change email.
The other thing that it is does that most other tools don't even take into consideration is it allows for the idea of different test environments and different organizations. To be able to have the same basic application deployed into different environments and to different organizations that probably require different business rules is a requirement for most QA departments. If you plan out the function points well enough, you can run the same exact test case against all environments and even against all organizations. This is a HUGE time saver.

Then there's GUI tool I am trying to get some input on. Imagine having separated your application into functional points and only documenting the functional points. Then using this tool to create your test cases on the fly. No only will you not be bothered with macro at all, but you also won't need to worry about creating documentation for all of those test cases. Since a test case is just a bunch of function points grouped together, the script and the documentation get automatically generated. This means that the automated script and the documentation will always be in sync and you won't have to waist time in writing documentation.
The GUI will allow for both automated function points and manual function points. You will be able to assign test cases to people and releases and those people will then be the ones to update the results of the tests ( well the ones that are manual anyways since Jameleon will be able automatically tell whether the automated ones passed or not ).

As you can tell, I'm not very passionate about the idea. But where automation tools stop Jameleon is starting.

I would love to have some help on this. I just ( in about 2 hours ) wrote another plugin for HttpUnit. The other plugin was for JWebUnit, but I found a few bugs in it that were blockers. I am planning on writing a 3270 mainframe emulator plugin and possibly for Swing/AWT.
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
Has anyone played with Jameleon yet? I am just hungry for feedback.
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by Christian Hargraves:
I don't know if you can compare Jameleon to FitNess. Jameleon is an acceptance testing framework.

Fit and FitNesse are, too!
I looked into FIT and FitNess before writing the tool and they just didn't seem to fit into the QA world as well as they do in the development world.

Why?

The steps to writing a Jameleon Function point are:

Those seem to be quite similar to writing FitNesse tests...

1) Write the function points code in Java.

Write a Fixture in Java.

2) Register your new code in the macro provided.
3) If Jameleon doesn't know about the application yet, register it.

Register the classes on the ClassPath page. For your convenience, also add Fixtures to the FixtureList.

The steps to writing a test-case are:
1) Organize your function points in the macro.

In Fit/FitNesse, those macros are written in HTML-Tables, referencing the Fixtures and feading them data.
Do I miss something???
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
One of the main focuses of FIT and FitNesse is to make your tests data-driven. This is very powerful. I agree that FitNess will do the job. I think FitNess and FIT are very useful frameworks. However, I don't believe they have all of the features needed to be an end-to-end solution of automated and manual testing for QA.
The Jameleon framework is conceptually different enough from other automated test frameworks that it required development from scratch. While most open-source testing tools are for the developer, Jameleon is intended for more QA-specific features to be added to it down the road.

As far as your comparison between Jameleon and FitNesse goes: I assume you didn't visit the Jameleon site and read what Jameleon is all about. What is currently released as Jameleon is the first step towards a very long journey.
While you may call mapping a single class and a method to a bunch of parameters through HTML a macro, FitNesse does not have a macro that ties independent functions together and passes these functions parameters without the requirement to know which parameters to pass to which function. How can you tie seemingly independent features together in FitNesse? Maybe by creating a new test which calls all of the other tests that it depends on? I guess you could create a plugin to FIT that allows mulitple classes and their methods to be registered and to have FIT know when to pass which parameters to which methods.
Now, what if you want to execute all tests that match a certain criteria. You need to go and create a separate test suite and run all of those tests. In QA, these criteria are constantly changing. Maybe one time, you only want to run a regression test to make sure no bugs slipped back into the release. Then, a little later, you only want to run tests against the open dialog feature. Maybe you know the application isn't ready yet and you just want to make sure the basic functionality is there. Now we need to go back in and create a new suite. Now let's say we create a test and it is a regression test, a change email test, and a priority three (only warn me if it fails) test. We need to remember to throw that test in all of the other suites and somehow name the suites well enough that someone will be able to hunt them down later.
What if we weren't able to automate all of the tests or some of the automated tests failed and we needed to execute the test cases manually? We would need to go to the documentation for both the manual and the automated test cases. How are you to to know for sure that when the manual test was automated or the automated test became manual that it was recorded in the test case which is stored separately from the script? How is one to record the results of both the automated and the manual test in a way that is efficient and easy to query on later? Most places have their automated test case documentation, manual test cases, test case results, and test case scripts stored separately and independently from one another. This, of course, adds many more steps to a process that can be much simplified by a central tool.
Most QA departments don't have the skillset and/or the time to look through a bunch of code or javadocs to see if there is already a test case written and if there isn't, document the test case new and store it somewhere, create a new Java file, compile it, throw it in the version control system, think about all of the test suites it needs to be added to, deploy it out to the testing farm and run it. My QA team tests over 100 applications. Let's say each application has about 500 test each, that's now 50,000 tests that they need to categorize, record results on, differeniate between automated and manual tests, query on ...
QA and even the customer need to have the ability to tie separate features (that were scripted by a more skilled QA engineer) together and feed the new test some data without having to get their hands dirty. It's the customer that creates the business rules. Why not have them map them over to test cases too?
Jameleon generates the test case documentation, the test-script (macro), categorizes the test case based on the function points and on a simple test-level that is set, and makes the new case available to the test farm and anyone else based solely on the order of the features grouped together in the test case.
Now, Joe Tester or the customer can come along and run a suite of tests that match a certain criteria or even a single test just like they can under FitNess.
In other words, I think FitNesse does the job it was intended to do and Jameleon does the job it was intended to do. They just do different jobs for different roles. The Jameleon framework and FIT do similar things differently. The difference is Jameleon's framework is the basis of many more planned features exclusive to the QA world. Currently, only the Jameleon framwork or automated testing engine is done and there are two HTTP plugins available. The management UI is next on my list.
I hope this helps.
HS Thomas
Ranch Hand

Joined: May 15, 2002
Posts: 3404
Christian ,
Wwould I be able to extend Jameleon to cover larger areas of business functionality across markets thereby speeding up the normal multi-functional testing where responsibilities for checking lie with many Analysts globally ?
Currently, the markets are in Germany, France, Spain, Britain where the bulk of the processing is done.
Assuming that barriers like adhering to a single Test process, language are overcome.
Sorry if I have sprung this a bit too early in the game.
regards
[ July 20, 2003: Message edited by: HS Thomas ]
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
Thomas,
I don't know if I fully understand your question. Could you give me an example?
It is pretty easy to extend Jameleon. I wrote two plugins in less than an hour each ( mostly because I used other open-source projects ).
What types of applications are you talking about testing? GUI apps? Mainframe apps? Console apps? Web apps? ...
HS Thomas
Ranch Hand

Joined: May 15, 2002
Posts: 3404
Oh , I think by now all applications are "Enterprise" applications with some Web front-ends.There may be some legacy mainframe processing buried underneath probably to hold some fairly static data.
Sorry my question wasn't that clear.But I meant Enterprise and Web applications.
Test data and the tests are created; The tests are run for all markets; some tests are more relevant to certain markets in that they kick off processes which need to be monitored closely.(Those processes may , yet , be in different markets.)
If you had no control of other systems Test proces , traditionally you'd hand them some data
(database or files) and ask politely to run some tests against that.Hopefully, if there is any systems design the system will be able to return adequate feedback on the processes in other markets.Finally, there needs to be a reconciliation that all widgets have been accounted for and have ended in the state(s) expected.
I think you mentioned that Jameleon is pitched towards QA Testing, which other test frameworks fail to address.
The above scenario is what I am used to at a QA level. A lot of the processes are already in place. Can Jameleon still be adopted ?
Remember we don't really want to test the "Test Cases" but the system.
regards
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
For starters, thanks for your interest.
Jameleon is a QA tool designed for testing of all types of applications. Currently, I am using Jameleon to test a very complex financial application that gets it data from a mainframe and a RDBMS and then runs that through a rules engine.
We also use Jameleon to log into one application to make a request, log into another application to process that request, and then log back into the original application to make sure the request was processed.
Since Jameleon is written in Java, it is possible to call any current automated tests, possibly parse the results and then record those results into a central place thus getting the accountability you are looking for.
Jameleon can also test the same application across multiple domains that then have different business rules. Have you visited the Jameleon website? The Overview Section talks about different testing environments and different organizations for the same application.
Our intent is to use Jameleon as that central testing tool that will call legacy tests and current tests and then keep track of the results for us. Because of the idea of testing features of an application and then tying those features together later, there are several possibilities that you can now do with this. You call an external script that tests a data migration from one datasource to another, then call another script that tests email notifications that should have been kicked off from this data-migration, and chain these seemingly indepedent tests together to produce the overall effect.
I hope this answers your question?
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
Sorry about posting two answers to the same question, but here it goes anyways.
This is an answer to the question you asked about running scripts that you have no control over. I can think of two answers to this.
One, you can set up Jameleon for the deparment or group of people ro run their legacy scripts. Two you can write a test in Jameleon that requests this test be done and then tries to make heads or tails out of the results sent back. It probably wouldn't be too hard to set up a simple workflow application that takes requests and posts their responses.
Just grasping at straws ...
Cheers
HS Thomas
Ranch Hand

Joined: May 15, 2002
Posts: 3404
Our intent is to use Jameleon as that central testing tool that will call legacy tests and current tests and then keep track of the results for us.

Thanks Christian, I think I could find this feature very useful.
Thanks for the information on the other features.
Have you any published information on "Case Studies" or projects that use Jameleon ?
regards
[ July 22, 2003: Message edited by: HS Thomas ]
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
Unfortunately, Jameleon has only been out for a month. I haven't gotten any feedback from people using it and I have posted bits about Jameleon at several places, asking for contributors and opinions. Sourceforge ( the place Jameleon is hosted ) has a feature to keep track of downloads, but currently that feature is also broken.
Please be aware that currently Jameleon is only a simple framework with no UI. If you are interested in contibuting ideas, recommendations, or even code, please just let me know.
HS Thomas
Ranch Hand

Joined: May 15, 2002
Posts: 3404
Please be aware that currently Jameleon is only a simple framework with no UI. If you are interested in contibuting ideas, recommendations, or even code, please just let me know.

Thanks, Christian. That is good to know should I find myself working in QA again.
There seems to be a resurgence for Testing and a demand for excellent tools.
regards
[ July 28, 2003: Message edited by: HS Thomas ]
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
I started development on the 2.0 release two weeks ago. I have been adding a lot of features and making it easier to use overall. If anyone is interested, I could really use some feedback and or help.
I have been trying to keep my thoughts at least a little documented in the "feature request" area of the http://sf.net/projects/jameleon project site for those interested in see what I have done so far and what I am planning to do.
Arjun Shastry
Ranch Hand

Joined: Mar 13, 2003
Posts: 1874
Hi,In one of the project,we are using Remedy for web development.Remedy has its AR(Action Request) system where Server and client communicate through RPC in intranet.To work on the web,Remedy creates webviews for each of the component i.e. it creates JSP for each form(Form consists of all GUI components which can be created using drag and drop in AR system).Remedy developer creates a business logic behind these forms using Remedy server side components.So as a Remedy developer,you don't have control over these JSPs or servlets.But overall application performance can be affected by business logic.My question is can your tool be used in this Remedy context to test the performance of Web application?
Remedy


MH
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
Currently, Jameleon does not have a preformance testing plugin and it is not planned in the near future.
Are you not interested in black-box testing with Jameleon?
JL Junk
Greenhorn

Joined: Jul 03, 2003
Posts: 3
Pity you chose the GNU licenses as its terms will actively discourage use of your tool at work.
The business I work for will not allow GNU licensed code in the door because of issues of ownership of products that use GNU, even if developed for inhouse use.
[ August 13, 2003: Message edited by: JL Junk ]
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
I have been thinking about changing it or releasing it under a new license, but I don't know what my legal rights are now I opened up under LGPL.
Ernest Friedman-Hill
author and iconoclast
Marshal

Joined: Jul 08, 2003
Posts: 24187
    
  34

The business I work for will not allow GNU licensed code in the door because of issues of ownership of products that use GNU, even if developed for inhouse use.

Thankfully, such businesses are in the minority. What an oppressive environment it must be in which to work!


[Jess in Action][AskingGoodQuestions]
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Christian Hargraves:
I have been thinking about changing it or releasing it under a new license, but I don't know what my legal rights are now I opened up under LGPL.

Have you got the copyright for all (or most) of the code? If you have then you should be able to publish it with another licence without problems.
There was some licence-related discussion in a thread at theserverside.com that you might consider taking a look at.


Author of Test Driven (2007) and Effective Unit Testing (2013) [Blog] [HowToAskQuestionsOnJavaRanch]
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
Yes, so far I am the only contributor. Do you know if I have to offer a duel license then?
Ashik Uzzaman
Ranch Hand

Joined: Jul 05, 2001
Posts: 2370

I am just downloading Jameleon now. I only saw HttpUnit and JWebUnit a little earlier. I am also involved in a web based testing tool that heaviliy depends on Java and XML. Developing testing tools are really now a joke...


Ashik Uzzaman
Senior Member of Technical Staff, Salesforce.com, San Francisco, CA, USA.
Christian Hargraves
Ranch Hand

Joined: Jul 01, 2003
Posts: 42
I need to make a release on the 2.0-dev branch. It has a ton of added features, better error messages and some bugfixes. However, the only way to know about the added features is through the test case scripts in the tst/xml directory and in the build.xml ( I added two new jameleon Ant Tasks ) since I don't have a site up for the 2.0-dev release.
I am very interested to hear your opinions and suggestions.
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Christian Hargraves:
Yes, so far I am the only contributor. Do you know if I have to offer a duel license then?

If you're the only contributor, you own the copyright for everything. You can decide to distribute Jameleon with GPL only, GPL and LGPL, LGPL and a commercial licence, or whatever combination makes you tick. What you can't do is revoke a previous licence (i.e. claim something like "you guys are not allowed to use the GPL'd code you downloaded yesterday") because once you've granted a licence, it's out there.
This is just generally speaking. You might want to consult someone with more knowledge about licencing issues.
herurmath saraswathi
Greenhorn

Joined: Oct 06, 2006
Posts: 3
Hi all,

I recently started studying this jameleon tool. It is really good. I would like to know is there any recording feature for this......?
If anybody knows about this, please respond immediately..
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Please start a new thread for your question. Thanks!
 
GeeCON Prague 2014
 
subject: New Automation Testing Tool