1<!-- 2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al. 3 4SPDX-License-Identifier: curl 5--> 6 7# The curl Test Suite 8 9# Running 10 11 See the "Requires to run" section for prerequisites. 12 13 In the root of the curl repository: 14 15 ./configure && make && make test 16 17 To run a specific set of tests (e.g. 303 and 410): 18 19 make test TFLAGS="303 410" 20 21 To run the tests faster, pass the -j (parallelism) flag: 22 23 make test TFLAGS="-j10" 24 25 "make test" builds the test suite support code and invokes the 'runtests.pl' 26 perl script to run all the tests. The value of `TFLAGS` is passed 27 directly to 'runtests.pl'. 28 29 When you run tests via make, the flags `-a` and `-s` are passed, meaning 30 to continue running tests even after one fails, and to emit short output. 31 32 If you'd like to not use those flags, you can run 'runtests.pl' directly. 33 You must `chdir` into the tests directory, then you can run it like so: 34 35 ./runtests.pl 303 410 36 37 You must have run `make test` at least once first to build the support code. 38 39 To see what flags are available for runtests.pl, and what output it emits, run: 40 41 man ./tests/runtests.1 42 43 After a test fails, examine the tests/log directory for stdout, stderr, and 44 output from the servers used in the test. 45 46## Requires to run 47 48 - perl (and a unix-style shell) 49 - python (and a unix-style shell, for SMB and TELNET tests) 50 - python-impacket (for SMB tests) 51 - diff (when a test fails, a diff is shown) 52 - stunnel (for HTTPS and FTPS tests) 53 - OpenSSH or SunSSH (for SCP and SFTP tests) 54 - nghttpx (for HTTP/2 and HTTP/3 tests) 55 - An available `en_US.UTF-8` locale 56 57### Installation of python-impacket 58 59 The Python-based test servers support both recent Python 2 and 3. 60 You can figure out your default Python interpreter with python -V 61 62 Please install python-impacket in the correct Python environment. 63 You can use pip or your OS' package manager to install 'impacket'. 64 65 On Debian/Ubuntu the package names are: 66 67 - Python 2: 'python-impacket' 68 - Python 3: 'python3-impacket' 69 70 On FreeBSD the package names are: 71 72 - Python 2: 'py27-impacket' 73 - Python 3: 'py37-impacket' 74 75 On any system where pip is available: 76 77 - Python 2: 'pip2 install impacket' 78 - Python 3: 'pip3 install impacket' 79 80 You may also need to manually install the Python package 'six' 81 as that may be a missing requirement for impacket on Python 3. 82 83### Port numbers used by test servers 84 85 All test servers run on "random" port numbers. All tests should be written 86 to use suitable variables instead of fixed port numbers so that test cases 87 continue to work independent on what port numbers the test servers actually 88 use. 89 90 See [`FILEFORMAT`](FILEFORMAT.md) for the port number variables. 91 92### Test servers 93 94 The test suite runs stand-alone servers on random ports to which it makes 95 requests. For SSL tests, it runs stunnel to handle encryption to the regular 96 servers. For SSH, it runs a standard OpenSSH server. 97 98 The listen port numbers for the test servers are picked randomly to allow 99 users to run multiple test cases concurrently and to not collide with other 100 existing services that might listen to ports on the machine. 101 102 The HTTP server supports listening on a Unix domain socket, the default 103 location is 'http.sock'. 104 105 For HTTP/2 and HTTP/3 testing an installed `nghttpx` is used. HTTP/3 106 tests check if nghttpx supports the protocol. To override the nghttpx 107 used, set the environment variable `NGHTTPX`. The default can also be 108 changed by specifying `--with-test-nghttpx=<path>` as argument to `configure`. 109 110### Shell startup scripts 111 112 Tests which use the ssh test server, SCP/SFTP tests, might be badly 113 influenced by the output of system wide or user specific shell startup 114 scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which 115 output text messages or escape sequences on user login. When these shell 116 startup messages or escape sequences are output they might corrupt the 117 expected stream of data which flows to the sftp-server or from the ssh 118 client which can result in bad test behavior or even prevent the test server 119 from running. 120 121 If the test suite ssh or sftp server fails to start up and logs the message 122 'Received message too long' then you are certainly suffering the unwanted 123 output of a shell startup script. Locate, cleanup or adjust the shell 124 script. 125 126### Memory test 127 128 The test script will check that all allocated memory is freed properly IF 129 curl has been built with the `CURLDEBUG` define set. The script will 130 automatically detect if that is the case, and it will use the 131 `memanalyze.pl` script to analyze the memory debugging output. 132 133 Also, if you run tests on a machine where valgrind is found, the script will 134 use valgrind to run the test with (unless you use `-n`) to further verify 135 correctness. 136 137 The `runtests.pl` `-t` option enables torture testing mode. It runs each 138 test many times and makes each different memory allocation fail on each 139 successive run. This tests the out of memory error handling code to ensure 140 that memory leaks do not occur even in those situations. It can help to 141 compile curl with `CPPFLAGS=-DMEMDEBUG_LOG_SYNC` when using this option, to 142 ensure that the memory log file is properly written even if curl crashes. 143 144### Debug 145 146 If a test case fails, you can conveniently get the script to invoke the 147 debugger (gdb) for you with the server running and the same command line 148 parameters that failed. Just invoke `runtests.pl <test number> -g` and then 149 just type 'run' in the debugger to perform the command through the debugger. 150 151### Logs 152 153 All logs are generated in the log/ subdirectory (it is emptied first in the 154 runtests.pl script). They remain in there after a test run. 155 156### Log Verbosity 157 158 A curl build with `--enable-debug` offers more verbose output in the logs. 159 This applies not only for test cases, but also when running it standalone 160 with `curl -v`. While a curl debug built is 161 ***not suitable for production***, it is often helpful in tracking down 162 problems. 163 164 Sometimes, one needs detailed logging of operations, but does not want 165 to drown in output. The newly introduced *connection filters* allows one to 166 dynamically increase log verbosity for a particular *filter type*. Example: 167 168 CURL_DEBUG=ssl curl -v https://curl.se 169 170 will make the `ssl` connection filter log more details. One may do that for 171 every filter type and also use a combination of names, separated by `,` or 172 space. 173 174 CURL_DEBUG=ssl,http/2 curl -v https://curl.se 175 176 The order of filter type names is not relevant. Names used here are 177 case insensitive. Note that these names are implementation internals and 178 subject to change. 179 180 Some, likely stable names are `tcp`, `ssl`, `http/2`. For a current list, 181 one may search the sources for `struct Curl_cftype` definitions and find 182 the names there. Also, some filters are only available with certain build 183 options, of course. 184 185### Test input files 186 187 All test cases are put in the `data/` subdirectory. Each test is stored in 188 the file named according to the test number. 189 190 See [`FILEFORMAT`](FILEFORMAT.md) for a description of the test case file 191 format. 192 193### Code coverage 194 195 gcc provides a tool that can determine the code coverage figures for the 196 test suite. To use it, configure curl with `CFLAGS='-fprofile-arcs 197 -ftest-coverage -g -O0'`. Make sure you run the normal and torture tests to 198 get more full coverage, i.e. do: 199 200 make test 201 make test-torture 202 203 The graphical tool `ggcov` can be used to browse the source and create 204 coverage reports on \*nix hosts: 205 206 ggcov -r lib src 207 208 The text mode tool `gcov` may also be used, but it doesn't handle object 209 files in more than one directory correctly. 210 211### Remote testing 212 213 The runtests.pl script provides some hooks to allow curl to be tested on a 214 machine where perl can not be run. The test framework in this case runs on 215 a workstation where perl is available, while curl itself is run on a remote 216 system using ssh or some other remote execution method. See the comments at 217 the beginning of runtests.pl for details. 218 219## Test case numbering 220 221 Test cases used to be numbered by category ranges, but the ranges filled 222 up. Subsets of tests can now be selected by passing keywords to the 223 runtests.pl script via the make `TFLAGS` variable. 224 225 New tests are added by finding a free number in `tests/data/Makefile.inc`. 226 227## Write tests 228 229 Here's a quick description on writing test cases. We basically have three 230 kinds of tests: the ones that test the curl tool, the ones that build small 231 applications and test libcurl directly and the unit tests that test 232 individual (possibly internal) functions. 233 234### test data 235 236 Each test has a master file that controls all the test data. What to read, 237 what the protocol exchange should look like, what exit code to expect and 238 what command line arguments to use etc. 239 240 These files are `tests/data/test[num]` where `[num]` is just a unique 241 identifier described above, and the XML-like file format of them is 242 described in the separate [`FILEFORMAT`](FILEFORMAT.md) document. 243 244### curl tests 245 246 A test case that runs the curl tool and verifies that it gets the correct 247 data, it sends the correct data, it uses the correct protocol primitives 248 etc. 249 250### libcurl tests 251 252 The libcurl tests are identical to the curl ones, except that they use a 253 specific and dedicated custom-built program to run instead of "curl". This 254 tool is built from source code placed in `tests/libtest` and if you want to 255 make a new libcurl test that is where you add your code. 256 257### unit tests 258 259 Unit tests are placed in `tests/unit`. There's a tests/unit/README 260 describing the specific set of checks and macros that may be used when 261 writing tests that verify behaviors of specific individual functions. 262 263 The unit tests depend on curl being built with debug enabled. 264