1[IMPORTANT NOTICE] 2------------------ 3 Failed tests usually indicate a problem with your local system setup 4and not within PHP itself (at least for official PHP release versions). 5You may decide to automatically submit a test summary to our QA workflow 6at the end of a test run. 7 Please do *not* submit a failed test as a bug or ask for help on why 8it failed on your system without providing substantial backup information 9on *why* the test failed on your special setup. Thank you :-) 10 11 12[Testing Basics] 13---------------- 14 The easiest way to test your PHP build is to run "make test" from the 15command line after successfully compiling. This will run the complete 16tests for all enabled functionalities and extensions using the PHP 17CLI binary. 18 To execute test scripts, you must build PHP with some SAPI, then you 19type "make test" to execute all or some test scripts saved under 20"tests" directory under source root directory. 21 22Usage: 23make test 24 25 "make test" basically executes "run-tests.php" script 26under the source root (parallel builds will not work). Therefore you 27can execute the script as follows: 28 29TEST_PHP_EXECUTABLE=sapi/cli/php \ 30sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB] 31 32 33[Which "php" executable "make test" look for] 34--------------------------------------------- 35If you are running the run-tests.php script from the command line (as above) 36you must set the TEST_PHP_EXECUTABLE environment variable to explicitly 37select the PHP executable that is to be tested, that is, used to run the test scripts. 38 39If you run the tests using make test, the PHP CLI and CGI executables are 40automatically set for you. "make test" executes "run-tests.php" script with the CLI binary. Some 41test scripts such as session must be executed by CGI SAPI. Therefore, 42you must build PHP with CGI SAPI to perform all tests. 43 44NOTE: PHP binary executing "run-tests.php" and php binary used for 45executing test scripts may differ. If you use different PHP binary for 46executing "run-tests.php" script, you may get errors. 47 48 49[Which php.ini is used] 50----------------------- 51 "make test" uses the same php.ini file as it would once installed. 52The tests have been written to be independent of that php.ini file, 53so if you find a test that is affected by a setting, please report 54this, so we can address the issue. 55 56 57[Which test scripts are executed] 58--------------------------------- 59 "run-tests.php" ("make test"), without any arguments executes all 60test scripts by extracting all directories named "tests" 61from the source root and any subdirectories below. If there are files, 62which have a "phpt" extension, "run-tests.php" looks at the sections 63in these files, determines whether it should run it, by evaluating 64the 'SKIPIF' section. If the test is eligible for execution, the 'FILE' 65section is extracted into a ".php" file (with the same name besides 66the extension) and gets executed. 67When an argument is given or TESTS environment variable is set, the 68GLOB is expanded by the shell and any file with extension "*.phpt" is 69regarded as a test file. 70 71 Tester can easily execute tests selectively with as follows. 72 73Examples: 74./sapi/cli/php run-tests.php ext/mbstring/* 75./sapi/cli/php run-tests.php ext/mbstring/020.phpt 76 77 78[Test results] 79-------------- 80 Test results are printed to standard output. If there is a failed test, 81the "run-tests.php" script saves the result, the expected result and the 82code executed to the test script directory. For example, if 83ext/myext/tests/myext.phpt fails to pass, the following files are created: 84 85ext/myext/tests/myext.php - actual test file executed 86ext/myext/tests/myext.log - log of test execution (L) 87ext/myext/tests/myext.exp - expected output (E) 88ext/myext/tests/myext.out - output from test script (O) 89ext/myext/tests/myext.diff - diff of .out and .exp (D) 90 91 Failed tests are always bugs. Either the test is bugged or not considering 92factors applying to the tester's environment, or there is a bug in PHP. 93If this is a known bug, we strive to provide bug numbers, in either the 94test name or the file name. You can check the status of such a bug, by 95going to: http://bugs.php.net/12345 where 12345 is the bug number. 96For clarity and automated processing, bug numbers are prefixed by a hash 97sign '#' in test names and/or test cases are named bug12345.phpt. 98 99NOTE: The files generated by tests can be selected by setting the 100environment variable TEST_PHP_LOG_FORMAT. For each file you want to be 101generated use the character in brackets as shown above (default is LEOD). 102The php file will be generated always. 103 104NOTE: You can set environment variable TEST_PHP_DETAILED to enable 105detailed test information. 106 107[Automated Testing] 108-------------- 109 If you like to keep up to speed, with latest developments and quality 110assurance, setting the environment variable NO_INTERACTION to 1, will not 111prompt the tester for any user input. 112 113Normally, the exit status of "make test" is zero, regardless of the results 114of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1, 115and "make test" will set the exit status ("$?") to non-zero, when an 116individual test has failed. 117 118Example script to be run by cron(1): 119========== qa-test.sh ============= 120#!/bin/sh 121 122CO_DIR=$HOME/cvs/php7 123MYMAIL=qa-test@domain.com 124TMPDIR=/var/tmp 125TODAY=`date +"%Y%m%d"` 126 127# Make sure compilation environment is correct 128CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre' 129export MAKE=gmake 130export CC=gcc 131 132# Set test environment 133export NO_INTERACTION=1 134export REPORT_EXIT_STATUS=1 135 136cd $CO_DIR 137cvs update . >>$TMPDIR/phpqatest.$TODAY 138./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE 139$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1 140if test $? -gt 0 141then 142 cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL 143fi 144========== end of qa-test.sh ============= 145 146NOTE: The exit status of run-tests.php will be 1 when 147REPORT_EXIT_STATUS is set. The result of "make test" may be higher 148than that. At present, gmake 3.79.1 returns 2, so it is 149advised to test for non-zero, rather then a specific value. 150 151 152[Creating new test files] 153------------------------- 154 Writing test file is very easy if you are used to PHP. 155See the HOWTO at http://qa.php.net/write-test.php 156 157 158[How to help us] 159---------------- 160 If you find bug in PHP, you can submit bug report AND test script 161for us. You don't have to write complete script, just give us test 162script with following format. Please test the script and make sure 163you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you 164submit. 165 166<?php 167/* 168Bug #12345 169substr() bug. Do not return expected string. 170 171ACTUAL OUTPUT 172XYXA 173 174EXPECTED OUTPUT 175ABCD 176*/ 177 178$str = "XYZABCD"; 179echo substr($str,3,7); 180 181?> 182 183[IMPORTANT NOTICE] 184------------------ 185This is an addendum to README.TESTING with additional information 186specific to server-tests.php. 187 188server-tests.php is backward compatible with tests developed for 189the original run-tests.php script. server-tests is *not* used by 190'make test'. server-tests was developed to provide support for 191testing PHP under it's primary environment, HTTP, and can run the 192PHP tests under any of the SAPI modules that are direct executables, 193or are accessible via HTTP. 194 195[New features] 196---------------- 197* Command line interface: 198 You can run 'php server-tests.php -h' to get all the possible options. 199* Configuration file: 200 the -c argument will allow you to use a configuration file. This is 201 handy if you are testing multiple environments and need various options 202 depending on the environment. 203 see server-tests-config.php for details. 204* CGI Emulation: 205 Will emulate a CGI environment when testing with the cgi sapi executable. 206* HTTP testing: 207 can be configured to run test scripts through an HTTP server running 208 on localhost. localhost is required since either the web server must 209 alias a directory to the php source directory, or the test scripts 210 must be copied to a directory under the web server 211 (see config options TEST_WEB_BASE_URL, TEST_BASE_PATH, and TEST_WEB_EXT) 212* New sections supported for test files (see below) 213 214When running tests over http, tests that require ini settings different that what 215the web server runs under will be skipped. Since the test harness defines a number 216of ini settings by default, the web server may require special configuration to 217make testing work. 218 219[Example Usage] 220---------------- 221Some (but not all!) examples of usage: 222 2231. run tests from the php source directory 224 php server-tests.php -p /path/to/php-cli 225 2262. run tests using cgi emulation 227 php server-tests.php -p /path/to/php-cgi 228 2293. run tests over http, copying test files into document root 230 php server-tests.php -w -u http://localhost/test -m /path/to/htdocs/test 231 2324. run tests over http, php sources have been aliased in web server 233 php server-tests.php -w -u http://localhost/test 234 2355. run tests using configuration file 236 php server-tests.php -c /path/to/server-tests-config.php 237 2386. run tests using configuration file, but overriding some settings: 239 (config file must be first) 240 php server-tests.php -c /path/to/server-tests-config.php -w -t 3 -d /path/to/testdir 241 242NOTE: configuration as described in README.TESTING still works. 243 244[New Test Sections] 245---------------- 246In addition to the traditional test sections 247(see http://qa.php.net/write-test.php), several new sections are available 248under server-tests. 249 250--POST-- 251This is not a new section, but not multipart posts are supported for testing 252file uploads, or other types of POST data. 253 254--CGI-- 255This section takes no value. It merely provides a simple marker for tests 256that MUST be run as CGI, even if there is no --POST-- or --GET-- sections 257in the test file. 258 259--DESCRIPTION-- 260Not used for anything, just a section for documenting the test 261 262--ENV-- 263This section get's eval()'d to help build an environment for the 264execution of the test. This can be used to change environment 265vars that are used for CGI emulation, or simply to set env vars 266for cli testing. A full example looks like: 267 268 --ENV-- 269 return <<<END 270 PATH_TRANSLATED=$filename 271 PATH_INFO=$scriptname 272 SCRIPT_NAME=$scriptname 273 END; 274 275Some variables are made easily available for use in this section, they 276include: 277 $filename full native path to file, will become PATH_TRANSLATED 278 $filepath =dirname($filename) 279 $scriptname this is what will become SCRIPT_NAME unless you override it 280 $docroot the equivalent of DOCUMENT_ROOT under Apache 281 $cwd the directory that the test is being initiated from 282 $this->conf all server-tests configuration vars 283 $this->env all environment variables that will get passed to the test 284 285 286--REQUEST-- 287This section is also eval'd, and is similar in nature to --ENV--. However, 288this section is used to build the url used in an HTTP request. Valid values 289to set in this section would include: 290 SCRIPT_NAME The initial part of the request url 291 PATH_INFO The pathinfo part of a request url 292 FRAGMENT The fragment section of a url (after #) 293 QUERY_STRING The query part of a url (after ?) 294 295 --REQUEST-- 296 return <<<END 297 PATH_INFO=/path/info 298 END; 299 300--HEADERS-- 301This section is also eval'd. It is used to provide additional headers sent 302in an HTTP request, such as content type for multipart posts, cookies, etc. 303 304 --HEADERS-- 305 return <<<END 306 Content-Type=multipart/form-data; boundary=---------------------------240723202011929 307 Content-Length=100 308 END; 309 310--EXPECTHEADERS-- 311This section can be used to define what headers are required to be 312received back from a request, and is checked in addition to the 313regular expect sections. For example: 314 315 --EXPECTHEADERS-- 316 Status: 404 317
