Running selected tests with FCTMF API

[Diedel]

Hello,
I have a set of test suites each in a .c file (test_suite_module_a.c, test_suite_module_b.c test_suite_module_c.c and so on), and I'm thinking how to run a custom selection of suites/tests and modify this selection quickly. An idea can be using an external file defining a boolean macro for every test suite (#define RUN_TEST_SUITE_MODULE_X 1) and use FCT_TEST_BGN_IF(RUN_TEST_SUITE_MODULE_X,name)/FCT_TEST_END_IF() with the corresponding X. The problem of this approach is the need of rebuild the main test program every time I change the selection.
Another idea could be using the 'filter prefix' when invoking the main.exe program but it can be annoying to modify the command line with many tests (and I'm not sure if the filter prefix works with test suites names).

Any ideas or suggestions?

Thanks in advance.

Diego.

[Ian Blumel]

> I have a set of test suites each in a .c file (test_suite_module_a.c,
> test_suite_module_b.c test_suite_module_c.c and so on), and I'm thinking how
> to run a custom selection of suites/tests and modify this selection quickly.
> An idea can be using an external file defining a boolean macro for every
> test suite (#define RUN_TEST_SUITE_MODULE_X 1) and use
> FCT_TEST_BGN_IF(RUN_TEST_SUITE_MODULE_X,name)/FCT_TEST_END_IF() with the
> corresponding X. The problem of this approach is the need of rebuild the
> main test program every time I change the selection.
> Another idea could be using the 'filter prefix' when invoking the main.exe
> program but it can be annoying to modify the command line with many tests
> (and I'm not sure if the filter prefix works with test suites names).
>
> Any ideas or suggestions?

For projects were I had a similar need, I either ended up using a
strict naming convention so the filter would apply consistently, like,

  test_suite_a__this_is_a_test
  test_suite_a__this_is_another_test
  test_suite_b__..
...

and so on, so I could run with a filter prefix like "test_suite_a" and
it would run all the tests in suite a, but if I wanted to run a
particular test in test suite a, I would use
"test_suite_a__this_a_test". However, I found that as soon as your
project group gets larger, the convention sometimes is gets lost.

So another idea is to customize your command line [1]. Roughly
speaking (this is untested code below), you could do something like,
following,

    /* Add our command line options. */
    static fctcl_init_t my_cl_options[] = {
        {"--use-suite",        /* long_opt */
            NULL,              /* short_opt (optional) */
            FCTCL_STORE_VALUE, /* action */
            "selects a test suite  to execute"  /* help */
        },
        FCTCL_INIT_NULL  /* sentinel */
    };

    /* If the option is blank (i.e. the default) or it matches the "test",
    then use it. assumes option & test != null.  */
    int use_suite(char const *option, char const *test)  {
        return (option[0] == '\0' /* blank */) \
                || (strcmp(option, test) == 0);
    }

    FCT_BGN() {
        char const  *suite =NULL;

        /* Install the command line options defined above. */
        fctcl_install(my_cl_options);

        /* Load the option, and default to a blank string if no option given.
        you should be able to entire main.exe --use-suite=suite_a, and
        the value "suite_a" will be returned if it was set.
        */
        suite = fctcl_val2("--use-suite", "");

        /* Now call the test suites in the other files if the conditions
        are met. */
        if ( use_suite(suite, "suite_a" ) ) {
            FCTMF_CALL_SUITE(suite_a);
        }
        if ( use_suite(suite, "suite_b") ) {
            FCTMF_CALL_SUITE(suite_b);
        }
        /* and so on... */
    }
    FCT_END();

Now the user  can type,

   main.exe => will launch all tests (since "suite" will be blank).
   main.exe --use-suite=suite_a (and it will launch suite_a).

and so on. Off course you could also handle the problem with multiple
command line options and setting flags for each of those options (i.e.
you could have "--use-suite-a" and "--use-suite-b" and so on).

Let me know if you need any more help following that rough example through.

Cheers.

_____
[1] Customizing the Command Line (since v1.2),
http://fctx.wildbearsoftware.com/static/fctx/doc/1.4.1/clapi_fct.html#customizing-the-command-line

--
Ian Blumel
www.wildbearsoftware.com

[diedel]

First thanks and sorry by the late reply.

It sounds good the method of the command line options. Using this method it could be possible to write an script like this:

suite_a = --use-suite-a
#suite_b = --use-suite-b
suite_c = --use-suite-c
...
suite_z = --use-suite-z
main.exe $(suite_a) $(suite_b) $(suite_c) ... $(suite_z)

Then to execute a selection of tests is as easy as editing this script and simply activating/deactivating the corresponding script variables. And only is needed to update the script every time a new test suite is added to the project.
What do you think?

Regards.

Diego.

[Ian Blumel]

> It sounds good the method of the command line options. Using this method it
> could be possible to write an script like this:
>
> suite_a = --use-suite-a
> #suite_b = --use-suite-b
> suite_c = --use-suite-c
> ...
> suite_z = --use-suite-z
> main.exe $(suite_a) $(suite_b) $(suite_c) ... $(suite_z)
>
> Then to execute a selection of tests is as easy as editing this script and
> simply activating/deactivating the corresponding script variables. And only
> is needed to update the script every time a new test suite is added to the
> project.
>
> What do you think?

Yes, that's a reasonable approach.

Over the last few years I find that what works is I create a single
program for each test suite, and then just invoke the program I am
interested in. If I want to run all the programs I write the script
that runs all the programs. In actual fact I find a combination of
cmake and ctest work great for this as well (see the fctx test suites
themselves).

So just another way of looking at it, and it all depends on works well for you.

Cheers.

--
Ian Blumel
www.wildbearsoftware.com