2019 May 16

Write C to Test C

It disappoints me to see messy, procedural, assertion-based testing in C codebases. Here’s an example from the wild:

const char *key = "404";
const int values[3] = { 0, 1, 2 };

plan_no_plan();

Map *map = map_new();

ok(map && map_empty(map), "Creation");
ok(map_first(map, &key) == NULL && strcmp(key, "404") == 0, "First on empty map");
ok(map_empty(map_prefix(map, "404")), "Empty prefix map");

In assertion-based testing, it’s common to test an entire data structure in a single function, instead of testing one of its operations. This groups together tests for unrelated behaviors. Assertions tend to contain difficult boolean expressions that mutate state silently, coupling test cases together. They use magic to hide the test framework implementation, and reimplement features that your programming language already has.

There’s often a better way.

The C programming language relies heavily on convention to get things done. One convention I find productive is table-driven testing. C has several features that facilitate this convention: anonymous structs, variadic functions, function pointers, and arrays. Macros can be useful too, in moderation.

Create one test file per translation unit

To test buf.c, create buf_test.c in the same directory. I’ll admit that I’m perplexed by the practice of putting tests in a different directory, or even a different repository.

Include the translation unit

#include "buf.c"

The test file includes a source file, not a header. This keeps static declarations in scope, making it possible to test the entire translation unit, not just whatever the header file (if any) declares.

It’s required that the translation unit doesn’t define main. If it does, I’d guess in most cases it can be factored out into a separate file.

Write tester functions

int
test_add()
{
	...
}

A tester function

returns an int and takes no arguments
tests one function, the testee
has a name that begins with test_ and includes the testee’s name

Gather test cases in arrays

Here we combine two of the nice, orthogonal C features I mentioned at the beginning: arrays and anonymous structs.

A tester function iterates over a statically initialized array of test cases. The test case type can be an anonymous struct; if it isn’t used elsewhere, there’s no need for a named struct declaration or typedef. The struct holds given input values and wanted output values.

struct {
	int a, b;
	int want;
} t, test[] = {
	{
		.a = 0,
		.b = 0,
		.want = 0,
	},
	{
		.a = 3,
		.b = 4,
		.want = 7,
	},
	{
		.a = 1,
		.b = -1,
		.want = 0,
	},
};

For brevity, you can put each case on a single line

	{ .a = 1, .b = -1, .want = 0 },

or omit the designators, but be careful of ordering.

	/* a   b  want */
	{  1, -1,    0  },

Fail big

On failure, print out a message specifying

what tester function we’re in (don’t type it twice; use __func__)
what case failed (inputs and wanted outputs)
what output was gotten

When a test fails, don’t quit. Run the rest of the tests to observe the whole picture. And count and return the number of failed cases.

int fails = 0;
const char *format = "%s: {a=%d, b=%d, want=%d}: got %d\n";
for (unsigned i = 0; i < len(test); i++) {
	t = test[i];
	int got = add(t.a, t.b);
	if (got != t.want) {
		fails++;
		printf(format, __func__, t.a, t.b, t.want, got);
	}
}
return fails;

When your code breaks, you’ll get a pithy message:

test_add: {a=0, b=1, want=1}: got 42

Write multiple testers, if necessary

Don’t try to force all of your test cases into one tester function. It’s OK to have test_buf, test_buf_empty, and test_buf_full if each of these require significantly different logic in the loop body. Shared code can often be factored out into helper functions.

Use comparable values in tester arrays

C’s equality operator doesn’t work for more dynamic structures like trees and lists. Even if your application never has to test for equality between these data types, you can still write a comparison function to use in tests. Doing so will simplify observing your code’s behavior!

Construct data with expressions

Many compound data structures, like stacks, are constructed imperatively, through a sequence of statements. This makes them difficult to use in tables. Avoid constructing these values imperatively before the table declaration. Instead, write a constructor that makes it possible to set up elaborate test cases in a single expression, like initializing a stack with a certain sequence of members.

{
	.s = newStack(42),
	.push = 68,
	.want = newStack(42, 68),
},

This keeps your code looking declarative, even if it’s highly imperative under the covers.

In assertion-based testing, data construction is interleaved with operation testing. For imperative data structures, data construction itself is an operation, and thus presents an opportunity for testing. But this mixes concerns; it’s more intelligible to write a separate tester for newStack(), for instance, than to test it while it’s being used for another purpose.

Iterate over your testers

The main function follows the same pattern: it iterates over an array of test cases and tallies up the number of failed cases. But here, the cases are tester functions.

As you write new testers, don’t forget to add them to the array.

#define len(x) (sizeof(x)/sizeof(x[0]))

int
main()
{
	int (*test[])() = {
		test_newbuf,
		test_append,
		test_free,
		test_read,
	};

	int fails = 0;
	for (unsigned i = 0; i < len(test); i++)
		fails += test[i]();

	return fails;
}

Run test programs with Valgrind

Be vigilant about memory errors.

valgrind --error-exitcode=1 --leak-check=full -q ./buffer_test

Conclusion

While there’s some amount of boilerplate in this approach, the use of data to specify test cases makes the code easier to read by factoring out repetition and state mutation. C can be a decent language for writing declarative code, if you adopt effective conventions to that end. And you can use C without integrating your project with another third-party dependency.

Feedback

Discuss this page by emailing my public inbox. Please note the etiquette guidelines.