Exhaustive Testing

The perfect impossible dream is to prove your program is 100% correct.  With a little reflection it’s easy to see testing is good at finding errors, but that it won’t do very much to show that there are no errors, or even to show there aren’t numerous errors.  Formal proofs seem like a hope, but they too can have bugs, and even with automated verification tools the tool itself might have a bug.

So an intriguing idea is to test every single possible combination of inputs, and verify each and every output result is correct. We might hope exhaustive testing like this could prove a program is correct, but it still shares many of the limits of any other testing.   Despite correct results under exhaustive testing, a significant remaining bug could be that your program is invoking undefined behavior.  One possible result of undefined behavior is that everything proceeds exactly as you planned.  Yet change the compiler, or change the hardware used for testing, or change the time of day, or change anything at all really, and your executable could begin to fail.  Even without undefined behavior, a bug in your compiler or a hardware fault could conceivably mask a bug in your program.  As unlikely as it might seem, I’ve read about one program created via machine learning that found the most optimal solution was to use a short circuit that existed due to a defect in the substrate of the development board.  It worked perfectly on that one board, but nowhere else.

Yet if you can do exhaustive testing, and it doesn’t take too long, why not?  It provides the best test coverage you could hope for, and it’s typically easy to write the test.  It’s usually possible to exhaustively test any function that has just a single 32 bit input parameter, or any function that has two 16 bit input parameters.  So long as the combined bit depth of all the input parameters isn’t much beyond 32 bits, it’s usually possible to exhaustively test any function with today’s hardware.  It might take fifteen minutes or more, but it’s something that could be tested when you want to run long tests.  If you have a function with total input parameter bit depth around 16 bits (or less), you might reasonably expect exhaustive testing of that function to take only 10 milliseconds or so.  That’s something you could run every time you test.

Posted in Uncategorized | Leave a comment

Google Test Projects in Visual Studio

This is a continuation of the blog entry on a quick start guide to setting up Google Test, this time specific to Visual Studio.

First Things First

If you used the quick start guide to set up Google Test, you may need to be aware of a bug in Visual Studio that occurs when using tests in static libraries or DLLs with Google Test. I’d recommend not using libraries for your tests, but if you do, Google provides some simple information on what you’ll need to know and how to work around the bug. If you don’t place your tests in libraries, then you won’t need to worry about any of this. I personally started out by putting my tests in libraries, since it would let me compose larger tests out of small test groups. However, I ultimately found it unnecessary since I could do the same thing with multiple test executables and CTest. Just keep in mind if you plan to create any tests in a static library (or DLL), the extra steps documented by google will probably be required in order for your tests to run.

Adding a Project

Here’s how to add a Google Test project to the Visual Studio solution that you created when you used CMake in the quick start guide.

In Visual Studio, open the solution and add a new project of type Visual C++ > Windows Desktop > Windows Console Application or Static Library. In the new project’s properties, change the following settings:

  1. Configuration Properties > C/C++ > General > “Additional Include Directories”:  Add
    $(GTEST_DIR)\googletest\include;$(GTEST_DIR)\googletest
  2. Configuration Properties > C/C++ > General > “Warning Level”:  Choose
    Level4 (/W4)
  3. Configuration Properties > C/C++ > General > “Treat Warnings as Errors”:  Choose
    Yes
  4. Configuration Properties > C/C++ > Code Generation > “Runtime Library”:  Depending on release/debug configuration, choose either
    Multi-threaded (/MT)   or  Multi-threaded Debug (/MTd)
  5. Configuration Properties > C/C++ > Precompiled Headers > “Precompiled Headers”:  Choose
    Not Using Precompiled Headers
  6. Configuration Properties > C/C++ > Precompiled Headers > “Precompiled Header File”: Remove any filename, so that this setting is blank.

 

If the project you added is a Windows console application (executable):

Expand the project name in the Solution Explorer window, and right click on References, and choose “Add Reference…” and select gtest, gtest_main, and any other static library projects from the solution that this project will depend upon. [At the time of this writing, the Google Test CMake creates a “gtest” static library/project that contains all the essential google test library functions, and a “gtest_main” static library/project that contains the main() function.]

In this project’s properties, set the following:

  1. Configuration Properties > Linker > General > “Link Library Dependencies”:  Choose
    Yes
  2. Configuration Properties > Linker > Optimization > “References”:  Choose
    No (/OPT:NOREF)
Posted in Uncategorized | Leave a comment

Quick Start Guide to Setting Up Google Test

There are a few different ways that you can get Google Test and then set it up to begin using it. The Google Test documentation skims a bit over the basic starting steps, then presents perhaps a choice overload for a lot of later steps. I have no doubt it’s the best overall reference available, but it’s not necessarily the best guide for someone who just wants to get started. You might find Google Test integrated into some of your tools (for example Visual Studio 17) – so that may be easier to use than this guide, depending on your needs. The method in this guide gives you flexibility and control over how you use Google Test.
Since my own interest was in setting up Google Test on Windows, this guide adds Windows specific details in parentheses. This may help clarify some steps regardless of OS. But aside from the parenthetical information, the steps should (hopefully) work for any OS.

Quick Steps:

  1. Get the git repository for googletest.
    1. Open a command prompt and navigate to the directory where you want the googletest repository to exist (On Windows, I created a new directory for repositories, called C:\Users\Jeff\Documents\repositories, which I then cd’d into)
    2. On the command line, type
      git clone https://github.com/google/googletest.git
  2. Set an environment variable called GTEST_DIR to the path of the new “googletest” directory that was just created by git clone.
    (On Windows, to reflect my own googletest path, I set GTEST_DIR to C:\Users\Jeff\Documents\repositories\googletest. You should set GTEST_DIR as a permanent environment variable, which on Windows you can do via Control Panel > System > Advanced system settings > Environment Variables… then “New…” under System variables. If you still have the command window open, you’ll need to relaunch it in order to use the GTEST_DIR variable.)
  3. If you don’t have CMake, download and install it.
  4. For an expanded version of this final step, see Building Google Test as a Standalone CMake Project.
    On the command line, navigate to a directory where you would like to place a new build of Google Test. On the command line, type
    mkdir mybuild
    cd mybuild
    Then if you are on Unix or OSX, type
    cmake -Dgtest_build_samples=ON $GTEST_DIR
    If you are on Windows, type
    cmake -Dgtest_build_samples=ON %GTEST_DIR%If you don’t want any of the google test sample projects, you can omit -Dgtest_build_samples=ON from the cmake command above.

You should now have Google Test built on your system. To go from here, if you are using Visual Studio, I’d suggest continuing on to read Google Test projects in Visual Studio. Otherwise -and in general- I’d suggest reading the Google Test documentation.

Posted in Uncategorized | Leave a comment