xref: /PHP-7.1/README.TESTING (revision 7f6387b5)
1[IMPORTANT NOTICE]
2------------------
3 Failed tests usually indicate a problem with your local system setup
4and not within PHP itself (at least for official PHP release versions).
5You may decide to automatically submit a test summary to our QA workflow
6at the end of a test run.
7 Please do *not* submit a failed test as a bug or ask for help on why
8it failed on your system without providing substantial backup information
9on *why* the test failed on your special setup. Thank you :-)
10
11
12[Testing Basics]
13----------------
14 The easiest way to test your PHP build is to run "make test" from the
15command line after successfully compiling. This will run the complete
16tests for all enabled functionalities and extensions using the PHP
17CLI binary.
18 To execute test scripts, you must build PHP with some SAPI, then you
19type "make test" to execute all or some test scripts saved under
20"tests" directory under source root directory.
21
22Usage:
23make test
24
25 "make test" basically executes "run-tests.php" script
26under the source root (parallel builds will not work). Therefore you
27can execute the script as follows:
28
29TEST_PHP_EXECUTABLE=sapi/cli/php \
30sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
31
32
33[Which "php" executable "make test" look for]
34---------------------------------------------
35If you are running the run-tests.php script from the command line (as above)
36you must set the TEST_PHP_EXECUTABLE environment variable to explicitly
37select the PHP executable that is to be tested, that is, used to run the test scripts.
38
39If you run the tests using make test, the PHP CLI and CGI executables are
40automatically set for you. "make test" executes "run-tests.php" script with the CLI binary.  Some
41test scripts such as session must be executed by CGI SAPI. Therefore,
42you must build PHP with CGI SAPI to perform all tests.
43
44NOTE: PHP binary executing "run-tests.php" and php binary used for
45executing test scripts may differ. If you use different PHP binary for
46executing "run-tests.php" script, you may get errors.
47
48
49[Which php.ini is used]
50-----------------------
51 "make test" uses the same php.ini file as it would once installed.
52The tests have been written to be independent of that php.ini file,
53so if you find a test that is affected by a setting, please report
54this, so we can address the issue.
55
56
57[Which test scripts are executed]
58---------------------------------
59 "run-tests.php" ("make test"), without any arguments executes all
60test scripts by extracting all directories named "tests"
61from the source root and any subdirectories below. If there are files,
62which have a "phpt" extension, "run-tests.php" looks at the sections
63in these files, determines whether it should run it, by evaluating
64the 'SKIP' section. If the test is eligible for execution, the 'FILE'
65section is extracted into a ".php" file (with the same name besides
66the extension) and gets executed.
67When an argument is given or TESTS environment variable is set, the
68GLOB is expanded by the shell and any file with extension "*.phpt" is
69regarded as a test file.
70
71 Tester can easily execute tests selectively with as follows.
72
73Examples:
74./sapi/cli/php run-tests.php ext/mbstring/*
75./sapi/cli/php run-tests.php ext/mbstring/020.phpt
76
77
78[Test results]
79--------------
80 Test results are printed to standard output. If there is a failed test,
81the "run-tests.php" script saves the result, the expected result and the
82code executed to the test script directory. For example, if
83ext/myext/tests/myext.phpt fails to pass, the following files are created:
84
85ext/myext/tests/myext.php   - actual test file executed
86ext/myext/tests/myext.log   - log of test execution (L)
87ext/myext/tests/myext.exp   - expected output (E)
88ext/myext/tests/myext.out   - output from test script (O)
89ext/myext/tests/myext.diff  - diff of .out and .exp (D)
90
91 Failed tests are always bugs. Either the test is bugged or not considering
92factors applying to the tester's environment, or there is a bug in PHP.
93If this is a known bug, we strive to provide bug numbers, in either the
94test name or the file name. You can check the status of such a bug, by
95going to: http://bugs.php.net/12345 where 12345 is the bug number.
96For clarity and automated processing, bug numbers are prefixed by a hash
97sign '#' in test names and/or test cases are named bug12345.phpt.
98
99NOTE: The files generated by tests can be selected by setting the
100environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
101generated use the character in brackets as shown above (default is LEOD).
102The php file will be generated always.
103
104NOTE: You can set environment variable TEST_PHP_DETAILED to enable
105detailed test information.
106
107[Automated Testing]
108--------------
109 If you like to keep up to speed, with latest developments and quality
110assurance, setting the environment variable NO_INTERACTION to 1, will not
111prompt the tester for any user input.
112
113Normally, the exit status of "make test" is zero, regardless of the results
114of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
115and "make test" will set the exit status ("$?") to non-zero, when an
116individual test has failed.
117
118Example script to be run by cron(1):
119========== qa-test.sh =============
120#!/bin/sh
121
122CO_DIR=$HOME/cvs/php7
123MYMAIL=qa-test@domain.com
124TMPDIR=/var/tmp
125TODAY=`date +"%Y%m%d"`
126
127# Make sure compilation environment is correct
128CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
129export MAKE=gmake
130export CC=gcc
131
132# Set test environment
133export NO_INTERACTION=1
134export REPORT_EXIT_STATUS=1
135
136cd $CO_DIR
137cvs update . >>$TMPDIR/phpqatest.$TODAY
138./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
139$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
140if test $? -gt 0
141then
142        cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
143fi
144========== end of qa-test.sh =============
145
146NOTE: The exit status of run-tests.php will be 1 when
147REPORT_EXIT_STATUS is set. The result of "make test" may be higher
148than that. At present, gmake 3.79.1 returns 2, so it is
149advised to test for non-zero, rather then a specific value.
150
151
152[Creating new test files]
153-------------------------
154 Writing test file is very easy if you are used to PHP.
155See the HOWTO at http://qa.php.net/write-test.php
156
157
158[How to help us]
159----------------
160 If you find bug in PHP, you can submit bug report AND test script
161for us. You don't have to write complete script, just give us test
162script with following format. Please test the script and make sure
163you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
164submit.
165
166<?php
167/*
168Bug #12345
169substr() bug. Do not return expected string.
170
171ACTUAL OUTPUT
172XYXA
173
174EXPECTED OUTPUT
175ABCD
176*/
177
178$str = "XYZABCD";
179echo substr($str,3,7);
180
181?>
182