There are too many ways to build and test a project with CMake. On the other hand, there is too little knowledge out there about those ways. As a consequence, people wrap the CMake invocation in custom scripts written in Bash, Python, Typescript etc.

Just look at the Jenkins configuration in your company. Look at different

implementations of GitHub actions/workflows for CMake. I bet what you will find

is a complete framework with custom abstractions of core utilities, version

control systems, the CMake command line, the actual build system, and more.

Looking at the commands that actually perform the steps for configuring,

building, and testing, it is very likely that you see those:

```sh

mkdir -p $build_dir

cd $build_dir

cmake -G Ninja $source_dir

ninja

ninja test

```

I assume you know that CMake can create the build directory and provides an

abstraction for invoking the actual build system. You should also know that the

test target essentially just runs ctest. So you could simplify and

generalize the above commands to this:

```sh

cmake -G Ninja -S $sourcedir -B $builddir

cmake --build $build_dir

ctest --test-dir $build_dir

```

But did you know that CTest already provides a command line abstraction to

execute the three steps?

```sh

ctest --build-and-test $sourcedir $builddir \

--build-generator Ninja --test-command ctest

```

Don't ask me why the above command stops after the build step when

--test-command ctest is omitted. After all, this mode is called

"build and test", so just executing ctest would be a sane default when no

test command is explicitly set by the user.

Anyway, there is more. CTest also provides abstractions for the version control

system, coverage analysis, and memory ckecking. But here be dragons.

There are, believe it or not, four different ways to configure CTest as a

dashboard client:

1. [CTest Command-Line](https://cmake.org/cmake/help/v3.30/manual/ctest.1.html#dashboard-client-via-ctest-command-line)

2. Declarative CTest Script (undocumented)

3. [CTest Module](https://cmake.org/cmake/help/v3.30/module/CTest.html)

4. [CTest Script](https://cmake.org/cmake/help/v3.30/manual/ctest.1.html#dashboard-client-via-ctest-script)

In the first approach, the command-line flag -D or a combination of

-M and -T is used to control which steps to execute.

The actual logic that is executed for those steps is controlled through a

configuration file called DartConfiguration.tcl which is read from the current

working directory.

Note that the documentation claims that this approach works in an

already-generated build tree. This is not true in all cases.

What is definitely needed, is that the source repository is already checked out.

While the source directory can be updated, it cannot be initialized with this

approach. We will get back to those details later.

For now, copy the following content into a file called DartConfiguration.tcl:

```tcl

SourceDirectory: Example

BuildDirectory: Example-build

UpdateCommand: git

ConfigureCommand: cmake -G Ninja -DCMAKECFLAGS_INIT=--coverage ..

CoverageCommand:gcov

MemoryCheckCommand: valgrind

```

Make sure that Example is a directory next to the DartConfiguration.tcl

file and contains a local clone of a git repository. Then execute the following:

```sh

ctest -M Experimental \

-T Start \

-T Update \

-T Configure \

-T Build \

-T Test \

-T Coverage \

-T MemCheck

```

Observe in the output that ctest updates the repository to the latest revision,

configures the project, builds it, runs the tests, analyzes the coverage, and

finds some memory leaks.

In the second approach, the DartConfiguration.tcl file is replaced with a file

written in the CMake syntax:

```cmake

set(CTESTSOURCEDIRECTORY "/home/dpfeifer/Example")

set(CTESTBINARYDIRECTORY "/home/dpfeifer/Example-build")

set(CTEST_COMMAND "ctest")

set(CTESTCMAKECOMMAND "cmake")

set(CTESTCVSCHECKOUT "gh repo clone Example")

```

The name of the file does not really matter. I use the name CTestScript.cmake

and invoke ctest like this:

```sh

ctest --script CTestScript.cmake --verbose

```

Remember that, with the previous approach, it was impossible to initialize the

source directory? With this approach, it is possible via the

CTESTCVSCHECKOUT variable. Despite the name, this variable can be used to

checkout a repository with any version control system, as shown in the example.

However, updating probably only works with CVS.

What is worse, is that this approach basically just handles the update,

configure, and test steps. Yes, the project is not even built before

running the tests. I wonder if anyone finds this useful.

Why am I even mentioning this approach when it is so barely useful? Because it

can get in the way when you don't expect it. I will get back to that.

The third approach also allows setting variables in the CMake syntax. Not in

a separate file, but in the top level project's CMakeLists.txt file, right

before include(CTest). This module internally calls configure_file to place

DartConfiguration.tcl into the build tree.

Now, it becomes clear why the documentation claims that ctest may be invoked

with command-line flags -D, -M, and -T in an already-generated build tree:

Because the CTest module places DartConfiguration.tcl there.

It also becomes clear under which circumstance it does not work as advertized:

When the project does not include(CTest)!

But when a project does include(CTest), it will get several custom targets

like ExperimentalCoverage that will execute ctest -D ExperimentalCoverage.

The last approach uses the same file and command-line as the second one.

The difference is that the build-and-test logic is scripted with CTest commands:

```cmake

cmakeminimumrequired(VERSION 3.14)

set(CTESTSOURCEDIRECTORY "/home/dpfeifer/Example")

set(CTESTBINARYDIRECTORY "/home/dpfeifer/Example-build")

set(CTESTCMAKEGENERATOR "Ninja")

findprogram(CTESTGIT_COMMAND "git")

findprogram(CTESTCOVERAGE_COMMAND "gcov")

findprogram(CTESTMEMORYCHECK_COMMAND "valgrind")

cmakehostsysteminformation(RESULT NPROC QUERY NUMBEROFLOGICALCORES)

if(NOT EXISTS ${CTESTSOURCEDIRECTORY})

set(CTESTCHECKOUTCOMMAND "gh repo clone Example")

endif()

ctest_start("Experimental")

ctest_update()

ctestconfigure(OPTIONS -DCMAKECFLAGSINIT=--coverage)

ctestbuild(PARALLELLEVEL ${NPROC})

ctesttest(PARALLELLEVEL ${NPROC})

ctest_coverage()

ctestmemcheck(PARALLELLEVEL ${NPROC})

```

This is the only approach that can both initialize and update the source

directory. It is also the only approach that allows you to execute the same

step more than once. Imagine you want to use a multi-config generator and then

run ctest_build for each configuration.

It gives full control over the logic what steps to run under what conditions.

Imagine you want to run the expensive memory checking only when the build

finishes without warnings, as the warnings may already indicate memory issues.

The possibilities are endless.

How does ctest --script distinguish between "CTest Script" mode and the

dreaded "Declarative CTest Script" mode?

At the beginning of the script, CTest implicitly sets the variable

[CTESTRUNCURRENTSCRIPT](https://cmake.org/cmake/help/v3.30/variable/CTESTRUNCURRENTSCRIPT.html) to 1.

Each of the ctest_* functions sets the variable to 0. When this variable is

still 1 at the end of the script, CTest assumes that none of the ctest_*

functions have been called. However, when the ctest_* functions are called

from inside a scoped block, there may be cases when the variable is unchanged.

In such cases, it is necessary to explicitly set(CTESTRUNCURRENT_SCRIPT 0).

My recommendation to everyone who wants to setup a CI system for CMake projects

is to use a CTest Script. For an example GitHub action built using a CTest

script have a look at

[purpleKarrot/cmake-action](https://github.com/purpleKarrot/cmake-action).

The fact that there are so many different approaches to the same use case is an

issue in my optionion. Also, the user experience of the CTest scripts needs to

be improved. I have some ideas how those issues can be addressed. I will write

about them in a follow up.