This week's book giveaway is in the Game Development forum.
We're giving away four copies of Badass: Making Users Awesome and have Kathy Sierra on-line!
See this thread for details.
The moose likes Agile and Other Processes and the fly likes System Tests and the Testing Process Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login

Win a copy of Badass: Making Users Awesome this week in the Game Development forum!
JavaRanch » Java Forums » Engineering » Agile and Other Processes
Bookmark "System Tests and the Testing Process" Watch "System Tests and the Testing Process" New topic

System Tests and the Testing Process

Jim Lang

Joined: Oct 04, 2000
Posts: 27
I was wondering how you would define a system test.
Is it a test that tests discrete parts of system functionality, or an end-to-end test to test every part of a system?
With so many different kinds of test (Unit Test, Regression Test, System Test, Quality Assurance Test, Factory Acceptance Test, Release Integration Test, User Test, Beta Test) it can be confusing to differentiate.
Can anyone give an overview of the different kinds? Or is there anywhere that I can find definitions for all the different kinds of test in a process?
Steve Fahlbusch

Joined: Sep 18, 2000
Posts: 591

In advance, i would like to apologize for the length of the following, as i lack to time to reduce to a smaller explanation.
Below i will attempt to answer some of the items you raised, but while you read it you have to remember that the same term may have different meanings in different contexts. Or said another way, there are various degrees to which we undertake to construction of software. Who we are, which industry we work for and our understanding of activities shapes the way we look at "programming" hence what we mean by "testing".
To start we need to define a few ideas:
1) Why do we test?
2) What do we test?
3) When do we test?
4) What is testing?
*** be prepared for some major oversimplication to set the stage for the overall view ***
1) We test to:
a) provide some level of confidence that tested item meets some pre-determined specification,
b) to reduce costs of reworking errors, and
c) provide feedback that we understand the task to be completed.
2) We test requirements, specifications, designs, interfaces, code, builds, sub-systems and systems (basically everything).
3) We test as soon as possible.
4) Testing is the formal to semi-formal process of comparing the pre-determined, expected results to the actual results at any point in the software development process.
? OK so far ?
Given the above, i now need to provide an overview of a typical project. This project will be defined using a more heavy-weight (read: formal) approach. This assumption for this project is that we a mid to large company, so that we have some user departments and we have a software development organization. (we can easily extend this to vendor supplied/developed, but we do need a separation of departments for this to make sense). This example project will be developed using the waterfall approach (because it is so much easier to discuss), but the concepts can easily be transformed into any approach.
As a very high overview, what we are going to do is define a system, going from the most general to the most specific, then build the system going from the most specific to the most general. A great analogy is that we are going to dig a hole for ourselves one level at a time and then fill it up one level at time, testing to insure we filled the level correctly.

Phase 0 - Actualization that a need exists.
The company realizes that we have a process that currently costs too much. Employees of the company discuss the issue and want to replace process A with a process B that saves 50%.
We have a Business Requirement and a test specification: ie when all is said and done, process B will save 50%

Phase 1 - Each user department defines requirements.
Given the Business Requirement, the effected user departments generates a list of "features" necessary for the system, such as:
User Department - D1 has features { f1, f2, f4, f6}
User Department - D2 has features { f1, f3, f5, f6}
User Department - D3 has features { f4, f6, f7 }
For each of these features (or requirements) each User Department generates a list of test specifications that will determine if the system meets the needs. (here is where we have the user requirements (or better still functional requirements)).
Phase 2 - The software development organization defines what features the system to be developed will have, such as:
System will have features - { f1, f2, f3, f4, f6, f7 } and will start detailing what each feature will do, in addition to creating test cases for each feature. (System Requirements).
Phase 3 - The system definition is formalized to an "engineering specification" of precisely what the system will (not how) do.
System will have critical factors { cf1......cfn } where each of these critical factors map to one or more system features and each system feature is fully defined by one or more critical factors. For each critical factor, there is a test specification (System Specifications).
Phase 4 - Generation of the High-Level Design
The high-level design defines the module public interface, the system architecture and defines the builds. At this point all modules, methods, parameters should be defined. for each build, test specifications are generated (Build and Integration Specifications).
Phase 5 - Generation of the Detailed Design
Each method is defined down to the level of how the method is to perform it's task and a set of test cases to run against the method. (Code Specification).
Phase 6 - Code and unit test.
The code specification (from phase 5) is given to a programmer. That programmer writes the code and executes the code specification. <First real test in code>
Phase 7 - Build and integration test.
The code from phase 6 is gathered together into builds and is tested against the build and integration specifications <from Phase 4>.
Phase 8 - System Test.
The code from Phase 7 is subjected to testing from the system specifications <from Phase 3>. This test should be performed under the guidance of another project manager and is "usually" called QC (quality control) testing.
Phase 9 - Formal System Test or Quality Assurance Test
The code from Phase 8 is tested against the Systems Requirements <from Phase 2>. This should be managed by someone outside the software development organization.
Phase 10 - The User Departments test the system against the User Specifications <Phase 1> in what some call the acceptance test.
Phase 11 - After some time in production, the system is reviewed against the Business Specifications <Phase 0>.
*** Well that's how it's supposed to work in theory ***

shailesh sonavadekar
Ranch Hand

Joined: Oct 12, 2000
Posts: 1874
what a useful & wonderful reply Steve ? It really doesn't matter about the length .
Can you please give me the online references on this topic ?
Steve Fahlbusch

Joined: Sep 18, 2000
Posts: 591

Most of the above comes from software engineering and formal test books i have seen. for some online references i would recommend: the acm (association of computing machinery), the ieee computer society and the sei at cmu (software engineering institute)
I agree. Here's the link:
subject: System Tests and the Testing Process