Testing and Unit Testing

As you have learned in previous laboratory experiences, getting a program to compile is only a minor part of programming. The compiler will tell you if there are any syntactical errors, but it isn't capable of detecting logical problems in your program. It's very important to test the program's functions to validate that they produce correct results.

These tests can be performed by hand; this is, running the program multiple times, providing representative inputs and visually checking that the program outputs correct results. A more convenient way is to implement functions in the program whose sole purpose is to validate that other functions are working correctly. In this laboratory experience, you will be practicing both testing methods.

Objectives:

1. Design tests to validate several programs “by-hand”, then use these tests to determine if the program works as expected.
2. Create unit tests to validate functions, using the assert function.

Pre-Lab:

Before you get to the laboratory you should have:

1. Reviewed the basic concepts related to testing and unit tests.

2. Reviewed how to use the assert function to validate another function.

3. Studied the concepts and instructions for this laboratory session.

4. Taken the Pre-Lab quiz, available in Moodle.

Testing a Function

When we test a function's validity we should test cases that activate the various results the function could return.

Example 1: If you were to validate the function isEven(unsigned int n) that determines if a positive integer n is even, you should test the function using even and odd numbers. A set of tests for that function could be:

Example 2: Suppose that a friend has created a function unsigned int ageRange(unsigned int age) that is supposed to return 0 if the age is between 0 and 5 (inclusive), 1 if the age is between 6 and 18 (inclusive), and 2 if the age is above 18. A common source of errors in functions like this one are the values near to the limits of each range, for example, the number 5 can cause errors if the programmer did not use a correct comparison. One good set of tests for the ageRange function would be:

Theassert Function

The assert(bool expression) function can be used as a rudimentary tool to validate functions. assert has a very powerful, yet simple functionality. If the expression that we place between the assert parenthesis is true, the function allows the program to continue onto the next instruction. Otherwise, if the expression we place between the parenthesis is false, the assert function causes the program to terminate and print out an error message on the terminal, that informs the user about the assert instruction that failed.

For example, the following program will run from start to finish without problems since every expression included in the assert's parentheses evaluates to true.

#include <iostream>
#include <cassert>
using namespace std;

int main() {
   int i = 10, j = 15;
   assert(i == 10);
   assert(j == i + 5);
   assert(j != i);
   assert( (j < i) == false);
   cout << "That's all, folks!" << endl;
   return 0;
}

Figure 1. Example of a program that passes all of the assert tests.

The following program will not run to completion because the second assert (assert(j == i);) contains an expression (j == i) that evaluates to false.

#include <iostream>
#include <cassert>
using namespace std;

int main() {
   int i = 10, j = 15;
   assert(i == 10);
   assert(j == i);
   assert(j != i);
   assert( (j < i) == false);
   cout << "That's all, folks!" << endl;
   return 0;
}

Figure 2. Example of a program that does not pass an assert test.

When the program is ran, instead of getting the phrase "That's all, folks!" in the terminal, we will obtain something like:

Assertion failed: (j == i), function main, file ../program01/main.cpp, line 8.

The program will not execute the remaining instructions after line 8.

How to Use Assert to Validate Functions?

Suppose that you want to automate the validation of the ageRange. One way to do it is by implementing and calling a function that calls the ageRange function with different arguments and verifies that each returned value is equal to the expected result. If the ageRange function returns a value that is not expected, the testing function aborts the program and reports the test that failed. The following illustrates a function to test the ageRange function. Observe that it consists of one assert per each of the tests we had listed earlier.

void test_ageRange() {
   assert(ageRange(5) == 0);
   assert(ageRange(2) == 0);
   assert(ageRange(6) == 1);
   assert(ageRange(18) == 1);
   assert(ageRange(17) == 1);
   assert(ageRange(19) == 2);
   assert(ageRange(25) == 2);
   cout << "ageRange passed all tests!!!" << endl;
}

Figure 3. Example of a test function using assert.

The function isEven(int n) is supposed to return true if n is even. Which assert(s) would be recommended to validate the function bool isEven(unsigned int)? assert( isEven(8) == true); assert( isEven(11) == false); assert( isEven(2) == true); assert( isEven(7) == true); All of the options except for the last one would be recommended. The last option is a wrongly formulated test since it is verifying that the result of isEven(7) is true.

A friend provides a function called int roundIt(double n) that given a number of type double in the range of -1000.0 to +1000.0 returns the rounded number to the nearest integer. Another friend provides the following function to validate the roundIt function:


When the function is called, the program stops on line 28 and reports that the assertion has failed. What can we conclude?
The function roundIt does not have a valid result for certain argument values. The function test_roundIt contains a poorly formulated test. In this case, the only thing we can certainly say is that the function test_roundIt contains one poorly formulated test. Line 28 compares the result of test_roundIt(10.4) with 11 which is not the function's expected result, e.g. roundIt(10.4) should result in 10, not 11.

Laboratory Session:

Exercise 1 - Designing Tests by Hand

In this exercise you will practice how to design tests to validate functions, using only the function's description and the graphical user interface that is used to interact with the function.

The exercise DOES NOT require programming, it only requires that you understand the function’s description, and your ability to design tests. This exercise and Exercise 2 are an adaptation of the activity described in [1].

Example 3: Suppose that a friend provides you with a program. She makes sure the program solves the following problem:

"given three integers, it displays the max value".

Suppose that the program has an interface like the following:

Figure 4. Interface for a program that finds the max value out of three integers.

You could determine if the program provides valid results without analyzing the source code. For example, you could try the following cases:

• a = 4, b = 2, c = 1; expected result: 4
• a = 3, b = 6, c = 2; expected result: 6
• a = 1, b = 10, c = 100; expected result: 100

If one of these three cases does not have the expected result, your friend's program does not work. On the other hand, if the three cases work, then the program has a high probability of being correct.

Functions to Validate

In this exercise you will be designing tests to validate various versions of the functions that are described below. Each one of the functions has four versions, "Alpha", "Beta", "Gamma" and "Delta".

• 3 Sorts: A function that receives three strings and orders them in lexicographic (alphabetical) order. For example, given giraffe, fox, and coqui, it would order them as: coqui, fox, and giraffe. To simplify the exercise, we will be using strings with lowercase letters. Figure 5 shows the function's interface. Notice there is a menu to select the implemented version.

  ---

  

  Figure 5. Interface for the 3 Sorts function.

  ---

• Dice: When the user presses the Roll them! button, the program generates two random integers between 1 and 6. The program informs the sum of the two random integers.

  ---

  

  Figure 6. Interface for the Dice function.

  ---

• Rock, Paper, Scissors: Each one of the players enters their play and the program informs who the winner is. Figure 7 shows the options where one object beats the other. The game's interface is shown in Figure 8.

  ---

  

  Figure 7. Ways to win in the "Rock, paper, scissors" game.

  ---

  

  Figure 8. Interface for the Rock, Paper, Scissors function.

  ---

• Zulu time: Given a time in Zulu format (time at the Greenwich Meridian) and the military zone in which the user wants to know the time, the program shows the time in that zone. The format for the entry data is in the 24 hour format ####, for example 2212 would be 10:12pm. The list of valid military zones can be found in http://en.wikipedia.org/wiki/List_of_military_time_zones. The following are examples of some valid results:

  • Given Zulu time 1230 and zone A (UTC+1), the result should be 1330.
  • Given Zulu time 1230 and zone N (UTC-1), the result should be 1130.

  • Puerto Rico is in military zone Q (UTC-4), therefore, when its 1800 in Zulu time, it's 1400 in Puerto Rico.

    ---

    

    Figure 9. Interface for the Zulu time function.

    ---

Instructions

1. For each of the functions described above, write in your notebook the tests that you will do to determine the validity of each implementation (Alpha, Beta, Gamma and Delta). For each function, think of the logical errors that the programmer could have made and write tests that determine if these errors were made. For each test, write the values that you will use and the expected result.

   For example, you could organize your answers in a table like the following:

Figure 10. Table to organize the test results.

You can see examples of how to organize your results here and here.

Exercise 2 - Doing Tests “by Hand”

The testing project implements several versions of each of the four functions that were described in Exercise 1. Some or all of the implementations could be incorrect. Your task is, using the tests you designed in Exercise 1, to test the versions for each function to determine which of them, if any, are implemented correctly.

This exercise DOES NOT require programming, you should make the tests without looking at the code.

Instructions

1. Load the project Testing into QtCreator. There are two ways to do this:

   • Using the virtual machine: Double click the file Testing.pro located in the folder /home/eip/labs/testing-testing of your virtual machine.

   • Downloading the project’s folder from Bitbucket: Use a terminal and write the command git clone http:/bitbucket.org/eip-uprrp/testing-testing to download the folder testing-testing from Bitbucket. Double click the file Testing.pro located in the folder that you downloaded to your computer.

1. Configure the project and run the program. You will see a window similar to the following:

   ---

   

   Figure 11. Window to select the function that will be tested.

   ---

2. Select the button for 3 Sorts and you will obtain the interface in Figure 5.

3. The "Alpha Version" in the box indicates that you're running the first version of the 3 Sorts algorithm. Use the tests that you wrote in Exercise 1 to validate the "Alpha Version". Afterwards, do the same with the Beta, Gamma and Delta versions. Write down which are the correct versions of the function (if any), and why. Remember that, for each function, some or all of the implementations could be incorrect. Additionally, specify which tests allowed you to determine the incorrect versions.

Exercise 3 - Using assert to Make Unit Tests

Doing tests by hand each time you run a program is a tiresome task. In the previous exercises you did it for a few simple functions. Imagine doing the same for a complex program like a search engine or a word processor!

Unit tests help programmers validate code and simplify the process of debugging while avoiding having to do these tests by hand in each execution.

Instructions

1. In the QtCreator menu, go to Build and select Clean Project "Testing". Then go to File and select Close Project "Testing".

2. Load the project UnitTests into QtCreator by double clicking the UnitTests.pro file. This file is also included in the testing-testing folder.

3. The project only contains the source code file main.cpp. This file contains four functions: fact, isALetter, isValidTime, and gcd, whose results are only partially correct.

   Study the description of each function that appears as a comment before the function's code to understand the task that the function is supposed to carry out.

   Your task is to write unit tests for each of the functions to identify the erroneous results. You do not need to rewrite the functions to correct them.

   For the fact function, a test_fact() is provided as a unit test function. If you invoke the function from main, compile, and run the program, you should obtain a message like the following:

   Assertion failed: (fact(2) == 2), function test_fact, file ../UnitTests/ main.cpp, line 69.

   This is evidence enough to establish that the fact function is NOT correctly implemented.

4. Notice that, by failing the previous test, the program did not continue its execution. To test the code you will write, comment the test_fact() called in main.

5. Write a unit test called test_isALetter for the isALetter function. Write various asserts in the unit test to try some data and its expected values (use the test_fact function for inspiration). Invoke test_isALetter from main and execute your program. If the isALetter function passes the test you wrote, continue writing asserts and executing the program until one of them fails.

6. Comment the test_isALetter call in main so you can continue with the other functions.

7. Repeat the steps in 5 and 6 for the other two functions, isValidTime and gcd. Remember that you should call each of the unit test functions from main for them to run.

Deliverables

1. Use "Deliverable 1" in Moodle to turn in the table with the tests you designed in Exercise 1 and that you completed in Exercise 2, with the results from the function tests.

2. Use "Deliverable 2" in Moodle to turn in the main.cpp file that contains the test_isALetter, test_isValidTime, test_gcd functions and their calls. Remember to use good programming techniques, include the name of the programmers involved, and document your program.

References

[1] http://nifty.stanford.edu/2005/TestMe/