When the customer gets demanding :)

[Sabine Chemaly]

Hi,
I have a customer who has a web application already running since more than a year with many users working on it and huge amounts of records already created on the database.
They want me to perform load testing, stress testing (for 3000 users!!) and soak testing on the production environment because they don't want to build a duplicate test environment.
Do you think this is possible?
If yes what are the precautions that should be taken and what are the negative effects that we might face?
Thank you in advance

[Brian Burress]

Most things are possible given proper resources, and as you observe, may have side effects.

Are they wanting you to literally test the production site or are they asking you to create an additional deployment of the application on the site and test against it?

What type of system is it? If it is read only, then the negative effects would be impaired application response time to the real users.

If it is update/read (which I am assuming it is) and they literally want you to use a live production URL, then how are you going to separate real data from test when it comes to the purpose of the site? Is there authentication involved? If so, you'll need to have a user base set up to load test against.

[Sabine Chemaly]

The system is a Microsoft Dynamics CRM.
It is a read, write system. They suggested that we make a backup of the database and that the testing will be done during the weekend so no user will be working.
But I wasn't recommended to perform a stress testing on a production environment.
So I'm not sure if it's the right thing to do.
I'd like to know what could be the side effects because it's a complex installation with many third party systems integrated and setting it up again won't be easy.
Thank you

[Brian Burress]

If there will be a data backup and everyone involved is comfortable that the restore will work (don't laugh, if it hasn't been tried you never know!) then I don't know that there are any real side effects to be concerned about. Document the plan for the back up, testing, restore and think through what the contingencys need to be in case you run into unknown situations.

While it would be ideal to have an isolated test environment to ensure no test activity 'bleeds' into production, there is at least some reasoning to using the prod environment in the sense that in no other case would you be testing in a match of prod other than using actual production machines, network, etc.

[Sabine Chemaly]

Hi again,
When I took a small online JMeter Fundamentals course, I asked the trainer this question and his answer was that testing the production server with JMeter is like committing suicide because we never know what might happen, the server might crash and because sometimes the records which are created on the database can never be deleted.
So what I want to know is what could exactly happen?
And how we can avoid it from happening if it's avoidable.

[Brian Burress]

Sabine Chemaly wrote:
I asked the trainer this question and his answer was that testing the production server with JMeter is like committing suicide because we never know what might happen, the server might crash and because sometimes the records which are created on the database can never be deleted.
So what I want to know is what could exactly happen?
And how we can avoid it from happening if it's avoidable.

Well, short answer is I do not know. I am not familiar with a situation like you describe but depending on the operating environment and how the backup/restore functions I could see it happening. As an example in the other extreme, if you are operating in a windows environment and take a disk image backup of the entire server which you restore after the test I do not see how you'd be left with such records. This approach may not be what you'd do, but just pointing out that if you are essentially wiping the system clean and going back to an image I don't know how a crash record would be present.

Suggestions from here:
1) Can you contact the trainer you refer to and get more details? Maybe he can explain in more detail what his concern is so you may address it.
2) Supply additional information here including your operating environment and what the database and application servers are. While I doubt I may be able to help, someone else may pick up the details and have some input. As an additional through, consider if there is another forum here which may be suitable. Not sure if there is, but you are heading more toward wanting information about a specific DB system.
3) Consider posting this type of question on a forum specifically for the DB system you are using, assuming one exists. If you do so, please provide a link to that post here and also provide a link to this site from it so folks have "two way visibility" to any replies.