Program testing

Testing is the process of finding differences between the expected behavior of the desired system and the observed behavior of the existing system.

Testing is insuring that what you expect is what you get.

You would like to find and fix these differences before your customers find them.

Black-box testing is testing to verify that the software does what it was designed to do.

Black box testing is also called:

specification testing
data-driven testing
functional testing
input/output-driven testing

Functional testing treats the module as a function that provides a value given a set of inputs.

However, most modules cannot be easily modeled by just a simple function, especially in the presence of side-effects, so the method can get quite complicated in practice.

The main idea in black box testing is that you cannot directly see the code inside the module. You can only see and use the module interface.

The code is a function that when provided with inputs, computes an output.

White-box testing is testing the internal consistency of the software.

White-box testing is also called:

structure testing
glass-box testing
behavioral testing
logic-driven testing
path-oriented testing

The main idea in white-box testing is that you can see inside the code, ignoring the specifications, in creating the test cases.

Knowledge of internal workings of the code are used to derive test cases.

Manual testing involves manually testing the system.

This is done using input from a keyboard and mouse.

Manual testing can be tedious, time-intensive, and error-prone.

Test automation software helps create and use test cases for software (e.g., unit testing, integration testing, etc.).

Often, a scripting language is provided.

In regression testing, a module has one or more input test data sets. For each input data set, the module produces simple output in text file form.

Then, if a change is made, each module can be automatically tested by rerunning it (automatically) on the test data sets and seeing if the text output is the same. Each output data set that is different from the previous version must be evaluated to determine if that is what is actually desired, or is there an inconsistency that has been introduced into the software. One way to do this is as follows.

Prepare a general-purpose set of test cases.
Run the existing program version with these test cases and save the result files.
Make program modifications.
Run the modified program to create a new set of result files.
Automatically compare the two results files and resolve any discrepancies.

A file comparator program compares two files and indicates the differences between the two files.

Some form of file comparator program is needed to make regression testing possible.

In simple regression testing, every unit of code to be tested is given some input in text form and produces some output in text form. We come up with test cases that given the input will produce the output.

Now, we make a change to the software.

We can now run all of the input test cases to produce new output text, and then compare the new output text with the old output text. Using file comparison techniques, if there is a change, then something
might be wrong. What are the four possibilities.

Is the new code wrong in that the new output text is incorrect but the old was correct?
Was the old code wrong in that the new output text is correct but the old was incorrect?
Did the change to the code cause the new output text to change, but the old was not really wrong?
Or, maybe both the old output text and the new output text are wrong and we do not know it yet.

The advantage of regression testing is that you can automate lots of testing and quickly locate the problem areas.