1--- 2c: Copyright (C) Daniel Stenberg, <daniel.se>, et al. 3SPDX-License-Identifier: curl 4Title: runtests.pl 5Section: 1 6Source: runtests 7See-also: 8 - runtests.pl 9Added-in: 7.5 10--- 11 12# NAME 13 14runtests.pl - run one or more test cases 15 16# SYNOPSIS 17 18**runtests.pl [options] [tests]** 19 20# DESCRIPTION 21 22*runtests.pl* runs one, several or all the existing test cases in curl's 23test suite. It is often called from the root Makefile of the curl package with 24'make test'. 25 26# TESTS 27 28Specify which test(s) to run by specifying test numbers or keywords. 29 30If no test number or keyword is given, all existing tests that the script can 31find are considered for running. You can specify single test cases to run by 32specifying test numbers space-separated, like `1 3 5 7 11`, and you can 33specify a range of tests like `45 to 67`. 34 35Specify tests to not run with a leading exclamation point, like `!66`, which 36runs all available tests except number 66. 37 38Prefix a test number with a tilde (~) to still run it, but ignore the results. 39 40It is also possible to specify tests based on a keyword describing the test(s) 41to run, like `FTPS`. The keywords are strings used in the individual tests. 42 43You can also specify keywords with a leading exclamation point and the keyword 44or phrase, like "!HTTP NTLM auth" to run all tests **except** those using this 45keyword. Remember that the exclamation marks and spaces need to be quoted 46somehow when entered at many command shells. 47 48Prefix a keyword with a tilde (~) to still run it, but ignore the results. 49 50# OUTPUT 51 52When running without `-s` (short output), for instance when running 53runtests.pl directly rather than via make, each test emits a pair of lines 54like this: 55 56 Test 0045...[simple HTTP Location: without protocol in initial URL] 57 --pd---e-v- OK (45 out of 1427, remaining: 16:08, took 6.188s, duration: 00:31) 58 59the first line contains the test number and a description. On the second line, 60the characters at the beginning are flags indicating which aspects of curl's 61behavior were checked by the test: 62 63 s stdout 64 r stderr 65 p protocol 66 d data 67 u upload 68 P proxy 69 o output 70 e exit code 71 m memory 72 v valgrind 73 E the test was run event-based 74 75The remainder of the second line contains the test result, current test sequence, 76total number of tests to be run and an estimated amount of time to complete the 77test run. 78 79# OPTIONS 80 81## `-a` 82 83Continue running the rest of the test cases even if one test fails. By 84default, the test script stops as soon as an error is detected. 85 86## `-ac \<curl\>` 87 88Provide a path to a curl binary to talk to APIs (currently only CI test APIs). 89 90## `-am` 91 92Display test results in automake style output (`PASS/FAIL: [number] [name]`). 93 94## `-c\<curl\>` 95 96Provide a path to a custom curl binary to run the tests with. Default is the 97curl executable in the build tree. 98 99## `-d` 100 101Enable protocol debug: have the servers display protocol output. If used in 102conjunction with parallel testing, it is difficult to associate the logs with 103the specific test being run. 104 105## `-E \<exclude_file\>` 106 107Load the **exclude_file** with additional reasons why certain tests should be 108skipped. Useful when testing with external HTTP proxies in which case some of 109the tests are not appropriate. 110 111The file contains colon-delimited lines. The first field contains the type of 112exclusion, the second field contains a pattern and the final field contains 113the reason why matching tests should be skipped. The exclusion types are 114*keyword*, *test*, and *tool*. 115 116## `-e` 117 118Run the test event-based (if possible). This makes runtests invoke curl with 119--test-event option. This option only works if both curl and libcurl were 120built debug-enabled. 121 122## `-f` 123 124Force the test to run even if mentioned in DISABLED. 125 126## `-g` 127 128Run the given test(s) with gdb. This is best used on a single test case and 129curl built --disable-shared. This then fires up gdb with command line set to 130run the specified test case. Simply (set a break-point and) type 'run' to 131start. 132 133## `-gl` 134 135Run the given test(s) with lldb. This is best used on a single test case and 136curl built --disable-shared. This then fires up lldb with command line set to 137run the specified test case. Simply (set a break-point and) type 'run' to 138start. 139 140## `-gw` 141 142Run the given test(s) with gdb as a windowed application. 143 144## `-h, --help` 145 146Displays a help text about this program's command line options. 147 148## `-j[num]` 149 150Spawn the given number of processes to run tests in. This defaults to 0 to run 151tests serially within a single process. Using a number greater than one allows 152multiple tests to run in parallel, speeding up a test run. The optimum number 153is dependent on the system and set of tests to run, but 7 times number of CPU 154cores is a good figure to start with, or 1.3 times number of CPU cores if 155Valgrind is in use. Enabling parallel tests is not recommended in conjunction 156with the -g option. 157 158## `-k` 159 160Keep output and log files in log/ after a test run, even if no error was 161detected. Useful for debugging. 162 163## `-L \<file\>` 164 165Load and execute the specified file which should contain perl code. This 166option allows one to change *runtests.pl* behavior by overwriting functions 167and variables and is useful when testing external proxies using curl's 168regression test suite. 169 170## `-l` 171 172Lists all test case names. 173 174## `-n` 175 176Disable the check for and use of valgrind. 177 178## `--no-debuginfod` 179 180Delete the `DEBUGINFOD_URLS` variable if that is defined. Makes valgrind, gdb 181etc not able to use this functionality. 182 183## `-o \<variablename=value\>` 184 185Overwrite the specified internal **variable** with **value**. Useful to change 186variables that did not get a dedicated flag to change them. Check the source to 187see which variables are available. 188 189## `-P \<proxy\>` 190 191Use the specified HTTP proxy when executing tests, even if the tests 192themselves do not specify a proxy. This option allows one to test external 193proxies using curl's regression test suite. 194 195## `-p` 196 197Prints out all files in the log directory to stdout when a test case fails. 198Practical when used in the automated and distributed tests since then the 199people checking the failures and the reasons for them might not have physical 200access to the machine and logs. 201 202## `-R` 203 204Run the tests in a scrambled, or randomized, order instead of sequentially. 205 206The random seed initially set for this is fixed per month and can be set with 207*--seed*. 208 209## `-r` 210 211Display run time statistics. (Requires the `Perl Time::HiRes` module) 212 213## `-rf` 214 215Display full run time statistics. (Requires the `Perl Time::HiRes` module) 216 217## `-rm` 218 219Force removal of files by killing locking processes. (Windows only, requires 220the **Sysinternals** `handle[64].exe` to be on PATH) 221 222## `--repeat=[num]` 223 224This repeats the given set of test numbers this many times. If no test numbers 225are given, it repeats ALL tests this many times. It adds the new repeated 226sequence at the end of the initially given one. 227 228If **-R** option is also used, the scrambling is done after the repeats have 229extended the test sequence. 230 231## `-s` 232 233Shorter output. Speaks less than default. 234 235## `--seed=[num]` 236 237When using *--shallow* or *-R* that randomize certain aspects of the behavior, 238this option can set the initial seed. If not set, the random seed is set based 239on the currently set local year and month and the first line of the "curl -V" 240output. 241 242## `--shallow=[num]` 243 244Used together with **-t**. This limits the number of tests to fail in torture 245mode to no more than **num** per test case. If this reduces the amount, the 246script randomly discards entries to fail until the amount is **num**. 247 248The random seed initially set for this is fixed per month and can be set with 249*--seed*. 250 251## `-t[num]` 252 253Selects a **torture** test for the given tests. This makes runtests.pl first 254run the tests once and count the number of memory allocations made. It then 255reruns the test that number of times, each time forcing one of the allocations 256to fail until all allocations have been tested. By setting *num* you can force 257the allocation with that number to be set to fail at once instead of looping 258through everyone, which is handy when debugging and then often in combination 259with *-g*. 260 261## `-u` 262 263Error instead of warning on server unexpectedly alive. 264 265## `-v` 266 267Enable verbose output. Speaks more than by default. If used in conjunction 268with parallel testing, it is difficult to associate the logs with the specific 269test being run. 270 271## `-vc \<curl\>` 272 273Provide a path to a custom curl binary to run when verifying that the servers 274running are indeed our test servers. Default is the curl executable in the 275build tree. 276 277# RUNNING TESTS 278 279Many tests have conditions that must be met before the test case can run fine. 280They could depend on built-in features in libcurl or features present in the 281operating system or even in third-party libraries that curl may or may not 282use. 283 284The test script checks most of these by itself to determine when it is safe to 285attempt to run each test. Those which cannot be run due to failed requirements 286are simply skipped and listed at the completion of all test cases. In some 287unusual configurations, the test script cannot make the correct determination 288for all tests. In these cases, the problematic tests can be skipped using the 289"!keyword" skip feature documented earlier. 290 291# WRITING TESTS 292 293The simplest way to write test cases is to start with a similar existing test, 294save it with a new number and then adjust it to fit. There is an attempt to 295document the test case file format in **tests/FILEFORMAT.md**. 296