#
2cfd9df1 |
| 10-Apr-2024 |
David Carlier |
Fix GH-13932: Attempt to fix mbstring on windows build (msvc). Build failure due to lack of VLA support in older compiler versions.
|
#
3394efc6 |
| 18-Mar-2024 |
Alex Dowad |
Fix infinite loop in mb_encode_mimeheader
|
#
67051eb8 |
| 11-Jan-2024 |
Alex Dowad |
Fix segfault caused by use of 'pass' encoding when mbstring converts multipart form POST data When mbstring.encoding_translation=1, and PHP receives an (RFC1867) form-based file upload,
Fix segfault caused by use of 'pass' encoding when mbstring converts multipart form POST data When mbstring.encoding_translation=1, and PHP receives an (RFC1867) form-based file upload, and the Content-Disposition HTTP header contains a filename for the uploaded file, PHP will internally invoke mbstring code to 1) try to auto-detect the text encoding of the filename, and if that succeeds, 2) convert the filename to internal text encoding. In such cases, the candidate text encodings which are considered during "auto-detection" are those listed in the INI parameter mbstring.http_input. Further, mbstring.http_input is one of the few contexts where mbstring allows the magic string "pass" to appear in place of an actual text encoding name. Before mbstring's encoding auto-detection function was reimplemented, the old implementation would never return "pass", even if "pass" was the only candidate it was given to choose from. It is not clear if this was intended by the original developers or not. This behavior was the result of some rather subtle details of the implementation. After mbstring's auto-detection function was reimplemented, if the new implementation was given only one candidate to choose, and it was not running in 'strict' mode, it would always return that candidate, even if the candidate was the non-encoding "pass". The upshot of all of this: Previously, if mbstring.encoding_translation=1 and mbstring.http_input=pass, encoding conversion of RFC1867 filenames would never be attempted. But after the reimplementation, encoding 'conversion' would occur (uselessly). Further, in December 2022, I reimplemented the relevant bit of encoding conversion code. When doing this, I never bothered to implement encoding/decoding routines for the non-encoding "pass", because I thought that they would never be used. Well, in the one case described above, those routines *would* have been used, had they actually existed. Because they didn't exist, we get a nice NULL pointer dereference and ensuing segfault instead. Instead of 'fixing' this by adding encoding/decoding routines for the non-encoding "pass", I have modified the function which the RFC1867 form-handling code invokes to auto-detect input encoding. This function will never return "pass" now, just like the previous implementation. Thanks to the GitHub user 'tstangner' for reporting this bug.
show more ...
|
#
1e92d47f |
| 14-Jan-2024 |
Alex Dowad |
Do not allow zend.script_encoding to be set to 'pass' When investigating another bug reported by GitHub user 'tstangner', I discovered that PHP segfaults when the INI parameter zend.
Do not allow zend.script_encoding to be set to 'pass' When investigating another bug reported by GitHub user 'tstangner', I discovered that PHP segfaults when the INI parameter zend.script_encoding is set to "pass". This bug dates back to December 2022 (caused by yours truly in 953864661a). If any PHP users in the wild were actually setting zend.script_encoding to "pass" (which would be an utterly useless thing to do), I expect that someone would have filed a bug report by now. The absence of such bug reports is evidence that nobody is doing this. Hence, it seems that the best fix is simply to disallow "pass" as a choice for zend.script_encoding. The internal function 'php_mb_zend_encoding_list_parser' which I am modifying to accomplish this has no other in-tree callers, aside from the 'exif' extension. Further, exif only calls the function with a few hard-coded values, and none of them are the string "pass", so this change will not have any impact on exif.
show more ...
|
#
e8141973 |
| 20-Dec-2023 |
Alex Dowad |
Fix bug in mb_get_substr_slow (sometimes outputs wrong number of characters) Thanks to Maurício Fauth for finding and reporting this bug. The bug was introduced in October 2022. It
Fix bug in mb_get_substr_slow (sometimes outputs wrong number of characters) Thanks to Maurício Fauth for finding and reporting this bug. The bug was introduced in October 2022. It originally only affected text encodings which do not have a fixed byte width per characters and for which mbstring does not have an mblen_table. However, I recently made another change to mbstring, such that mb_substr no longer relies on the mblen_table even if one is available. Because of this change, the bug earlier introduced in October 2022 now affected a greater number of text encodings, including UTF-8.
show more ...
|
#
ec348a12 |
| 06-Dec-2023 |
Alex Dowad |
Character indices used by mb_strpos and mb_substr have same meaning, even on invalid strings Starting many years ago, libmbfl included a 'mblen_table' for selected text encodings. This t
Character indices used by mb_strpos and mb_substr have same meaning, even on invalid strings Starting many years ago, libmbfl included a 'mblen_table' for selected text encodings. This table allows looking up the byte length of a (possibly multi-byte) character from the value of the first byte. libmbfl uses these tables to optimize certain operations; if a text-processing operation can be performed using an mblen_table, it may not be necessary to decode the text to codepoints. Since libmbfl's decoding filters are generally slow, this improves performance. Since mbstring is (or was) based on libmbfl, it has always used these mblen_tables to implement some functions. This design has a significant downside. Let me explain: While some mbstring functions are implemented by converting input text to codepoints and operating on the codepoints, others operate directly on the original input bytes (using an mblen_table to identify character boundaries). Both of these implementation styles, if correctly coded, yield equivalent results on valid strings. However, on strings which contain encoding errors, the results are often different. When decoding byte strings to codepoints using some text encoding, mbstring uses the non-existent codepoint 0xFFFFFFFF to represent a byte sequence which cannot be decoded. Then, when mbstring indexes into the resulting sequence of codepoints, the index of any particular character depends on the number of such 'error markers' which were produced during the decoding process. In contrast, when an mblen_table is used to split a byte sequence into characters, there is no question of counting encoding errors; rather, table lookups into the mblen_table are used to repeatedly 'bite off' some number of bytes (which are treated as one 'character'). In the presence of encoding errors, these two methods of mapping between byte indices and character indices are inherently different and will rarely agree. (For completeness, it must be said that some internal mbstring code which operates only on UTF-8 text uses a third method for mapping between byte indices and character indices, that is: counting non-continuation UTF-8 bytes, which are all bytes whose binary representation is NOT like 0b10xxxxxx. This method happens to agree with the method which involves decoding the input text to codepoints and then counting the codepoints.) I have been aware of this issue for years, but only recently became aware that in the case of mb_strstr, mb_strpos, and mb_substr, this issue can cause seriously unintuitive behavior (and even security vulnerabilities). This was reported by Stefan Schiller. Stefan Schiller shared the following example for mb_strstr: var_dump(mb_strstr("\xf0start", "start", false, "UTF-8")); // string(2) "rt" Similarly, when mb_strpos and mb_substr are used to identify and extract a substring from a string with encoding errors, Stefan Schiller pointed out that the extracted portion may be completely different than desired. This is because (for UTF-8 strings) mb_strpos works by counting non-continuation bytes, but mb_substr uses an mblen_table. Since some mbstring functions *cannot* be implemented using an mblen_table, as long as mblen_tables are used, similar inconsistencies cannot be totally avoided. But the mblen_tables are critical to mbstring's performance. Or are they? Benchmarking mb_substr on various UTF-8, SJIS, and EUC-JP strings revealed something interesting. On all SJIS and EUC-JP test cases, mb_substr was slightly faster when the mblen_table based code was deleted. For some UTF-8 test cases, the mblen_table-based code was a tiny bit faster, while for others the fallback code was a touch faster; in no case was the difference significant. Therefore, the simple fix is to delete the mblen_table-based implementation of mb_substr. Aside from making the function behave consistently with other mbstring functions on invalid strings, there is ONE case where behavior is now different on valid strings: that is, on SJIS-Mac (MacJapanese) strings which contain any of the following code units: 0x85AB-0x85AD, 0x85BF, 0x85C0, 0x85C1, 0x8645, 0x864B, 0x865D, 0x869E, 0x86CE, 0x86D3-0x86D5, 0x86D6, 0x8971, 0x8792, 0x879D, 0x87FB, 0x87FC, 0xEB41, 0xEB42, 0xEB50, 0xEB5B, 0xEB5D, 0xEB60-0xEB6E, and all from 0xEB81 and above. All of these SJIS-Mac code units share the (very unusual) property that they do not correspond to any one Unicode codepoint. When converting from SJIS-Mac to Unicode, these must be converted to 2, 3, 4, or 5 codepoints each. The previous, mblen_table-based implementation of mb_substr would treat all of these SJIS-Mac byte sequences as 'one character'. Now, they are treated as multiple characters (one for each of the Unicode codepoints which they decode to). The new behavior is more consistent with other mbstring functions. I don't know if SJIS-Mac users will like this change or not (probably most will never notice), but the BC break is justified by the very real security impact of the previous, inconsistent behavior. Finally, I should comment on whether similar changes are needed elsewhere. The remaining functions which use an mblen_table are: mb_str_split, mb_strcut, and various search functions (such as mb_strpos). The search functions are only affected now when they receive a positive 'offset' parameter specifying where to start searching from. The search functions should definitely be fixed so they do not use an mblen_table to implement the 'offset' parameter. I am not convinced that there is any good reason to change mb_str_split and mb_strcut.
show more ...
|
#
6176538d |
| 19-Nov-2023 |
Niels Dossche <7771979+nielsdos@users.noreply.github.com> |
Fix GH-11992: utf_encodings.phpt fails on Windows 32-bit Similar bug as before in #10776, but now in other code. Closes GH-12726.
|
#
81e236cd |
| 25-Oct-2023 |
Alex Dowad |
Fix infinite loop when mb_detect_encoding is used on UTF-8 BOM This bug was introduced in cb840799b4. Thanks to Ignace Nyamagana Butera for discovering this bug and to Sebastian
Fix infinite loop when mb_detect_encoding is used on UTF-8 BOM This bug was introduced in cb840799b4. Thanks to Ignace Nyamagana Butera for discovering this bug and to Sebastian Bergmann for doing an initial investigation and opening a bug ticket.
show more ...
|
#
08c3b332 |
| 03-Oct-2023 |
Levi Morrison |
fix mbstring.c -Wsingle-bit-bitfield-constant-conversion (#12327) These were both local variables, so there isn't much value in using bitfields in the first place.
|
#
7364b7bc |
| 29-Jul-2023 |
Ilija Tovilo |
Fix uaf of MBSTRG(all_encodings_list) We need to remove the value from the GC buffer before freeing it. Otherwise shutdown will uaf when running the gc. Do that by switching from zen
Fix uaf of MBSTRG(all_encodings_list) We need to remove the value from the GC buffer before freeing it. Otherwise shutdown will uaf when running the gc. Do that by switching from zend_hash_destroy to zend_array_destroy, which should also be faster for freeing members due to inlining of i_zval_ptr_dtor. Closes GH-11822
show more ...
|
#
af3c220a |
| 07-Jul-2023 |
George Peter Banyard |
Deprecate passing a negative width to mb_strimwidth()
|
#
78d98e50 |
| 30-Jun-2023 |
Niels Dossche <7771979+nielsdos@users.noreply.github.com> |
Fix GH-11567: mb_str_pad causes access violation When not providing a pad string, *and* not having other defaulted arguments, the function would crash on a NULL pad zend_string*. Des
Fix GH-11567: mb_str_pad causes access violation When not providing a pad string, *and* not having other defaulted arguments, the function would crash on a NULL pad zend_string*. Despite testing with an empty pad string, the issue wasn't found because when using named arguments the pad string *is* filled in.
show more ...
|
#
14a868b7 |
| 23-Jun-2023 |
nielsdos <7771979+nielsdos@users.noreply.github.com> |
Fix GH-11514: PHP 8.3 build fails with --enable-mbstring enabled I tweaked the #if check such that the workaround only applies on GCC versions older than 8.0. I tested this with GCC
Fix GH-11514: PHP 8.3 build fails with --enable-mbstring enabled I tweaked the #if check such that the workaround only applies on GCC versions older than 8.0. I tested this with GCC 7.5, 8.4, 9.4, GCC 13.1.1, and Clang 10.0. Closes GH-11516.
show more ...
|
#
6c015ae8 |
| 21-Jun-2023 |
Ilija Tovilo |
mbstring count_demerits in reverse order (#11493) This way we can avoid moving candidates that would be eliminated after.
|
#
68591632 |
| 20-Jun-2023 |
Niels Dossche <7771979+nielsdos@users.noreply.github.com> |
[RFC] Implement mb_str_pad() (#11284) Closes GH-10203.
|
#
443927e3 |
| 19-Jun-2023 |
Alex Dowad |
Fix GH-11476: crash with count_demerits negative-size-param Co-authored-by: Ilija Tovilo <ilija.tovilo@me.com> Co-authored-by: Niels Dossche <7771979+nielsdos@users.noreply.github.com>
|
#
03b163b2 |
| 21-May-2023 |
Niels Dossche <7771979+nielsdos@users.noreply.github.com> |
Remove unused variable err in mb_send_mail() (#11285)
|
#
7914b8ce |
| 11-May-2023 |
Alex Dowad |
Use pakutoma's encoding check functions for mb_detect_encoding even in non-strict mode In 6fc8d014df, pakutoma added specialized validity checking functions for some legacy text encoding
Use pakutoma's encoding check functions for mb_detect_encoding even in non-strict mode In 6fc8d014df, pakutoma added specialized validity checking functions for some legacy text encodings like ISO-2022-JP and UTF-7. These check functions perform a more strict validity check than the encoding conversion functions for the same text encodings. For example, the check function for ISO-2022-JP verifies that the string ends in the correct state required by the specification for ISO-2022-JP. These check functions are already being used to make detection of text encoding more accurate when 'strict' detection mode is enabled. However, since the default is 'non-strict' detection (a bad API design but we're stuck with it now), most users will not benefit from pakutoma's work. I was previously reluctant to enable this new logic for non-strict detection mode. My intention was to reduce the scope of behavior changes, since almost *any* behavior change may affect *some* user in a way we don't expect. However, we definitely have users whose (production) code was broken by the changes I made in 28b346bc06, and enabling pakutoma's check functions for non-strict detection mode would un-break it. (See GH-10192 as an example.) The added checks do also make sense. In non-strict detection mode, we will not immediately reject candidate encodings whose validity check function returns false; but they will be much less likely to be selected. However, failure of the validity check function is weighted less heavily than an encoding error detected by the encoding conversion function.
show more ...
|
#
3ab10da7 |
| 30-Apr-2023 |
Alex Dowad |
Take order of candidate encodings into account when guessing text encoding The documentation for mb_detect_encoding says that this function "Detects the most likely character encoding fo
Take order of candidate encodings into account when guessing text encoding The documentation for mb_detect_encoding says that this function "Detects the most likely character encoding for string `string` from an ordered list of candidates". Prior to 28b346bc06, mb_detect_encoding did not really attempt to determine the "most likely" text encoding for the input string. It would just return the first candidate encoding for which the string was valid. In 28b346bc06, I amended this function so that it uses heuristics to try to guess which candidate encoding is "most likely". However, the caller did not have any way to indicate which candidate text encoding(s) they consider to be more likely, in case the heuristics applied are inconclusive. In the language of Bayesian probability, there was no way for the caller to indicate their 'prior' assignment of probabilities. Further, the documentation for mb_detect_encoding also says that the second parameter `encodings` is "a list of character encodings to try, in order". The documentation clearly implies that the order of the `encodings` argument should be significant. Therefore, amend mb_detect_encoding so that while it still uses heuristics to guess the most likely text encoding for the input string, it favors those which are earlier in the list of candidate encodings. One complication is that many callers of mb_detect_encoding use it in this way: mb_detect_encoding($string, mb_list_encodings()); In a majority of cases, this is bad code; mb_detect_encoding will both be much slower and the results will be less reliable than if a smaller list of candidates is used. However, since such code already exists and people are using it in production, we should not unnecessarily break it. The order of candidate encodings obviously does not express any prior belief of which candidates are more likely in this case, and treating it as if it did will degrade the accuracy of the result. Since mb_list_encodings now returns a single, immutable array on each call, we can avoid that problem by turning off the new behavior when we receive the array of encodings returned by mb_list_encodings. This implementation means that if the user does this: $a = mb_list_encodings(); mb_detect_encoding($string, $a); ...then the order of candidate encodings will not be considered. However, if the user explicitly initializes their own array of all supported legacy text encodings, then the order *will* be considered. The other functions which also follow this new behavior are: • mb_convert_variables • mb_convert_encoding (when multiple candidate input encodings are listed) Other places where "detection" (or really "guessing") of text encoding may be performed include: • mb_send_mail • Zend engine, when determining the encoding of a PHP script • mbstring processing of HTTP request contents, when http_input INI parameter is set to a list In these cases, the new logic based on order of candidate encodings is *not* enabled. It *might* be logical to consider the order of candidate encodings in some or all of these cases, but I'm not sure if that is true, so it seems wiser to avoid more behavior changes than is necessary. Further, ever since the new encoding detection heuristics were implemented in 28b346bc06, we have not received any complaints of user code being broken in these areas. So I am reluctant to "fix what isn't broken". Well, some might say that applying the new detection heuristics to mb_send_mail, etc. in 28b346bc06 was "fixing what wasn't broken", but (cough cough) I don't have any comment on that...
show more ...
|
#
97e29bed |
| 22-Apr-2023 |
Alex Dowad |
Use shared, immutable array for return value of mb_list_encodings This will allow us to easily check in other mbstring functions if the list of all supported encodings, returned by mb_li
Use shared, immutable array for return value of mb_list_encodings This will allow us to easily check in other mbstring functions if the list of all supported encodings, returned by mb_list_encodings, is passed in as input to another function. Co-authored-by: Ilija Tovilo <ilija.tovilo@me.com>
show more ...
|
#
732d92c0 |
| 28-Apr-2023 |
Javier Eguiluz |
[skip ci] Fix various typos and grammar issues (#11143)
|
#
6df7557e |
| 02-Apr-2023 |
Alex Dowad |
mb_parse_str, mb_http_input, and mb_convert_variables use fast text conversion code for automatic encoding detection For mb_parse_str, when mbstring.http_input (INI parameter) is a list of
mb_parse_str, mb_http_input, and mb_convert_variables use fast text conversion code for automatic encoding detection For mb_parse_str, when mbstring.http_input (INI parameter) is a list of multiple possible text encodings (which is not the case by default), this new implementation is about 25% faster. When mbstring.http_input is a single value, then nothing is changed. (No automatic encoding detection is done in that case.)
show more ...
|
#
b721d0f7 |
| 10-Mar-2023 |
pakutoma |
Fix phpGH-10648: add check function pointer into mbfl_encoding Previously, mbstring used the same logic for encoding validation as for encoding conversion. However, there are ca
Fix phpGH-10648: add check function pointer into mbfl_encoding Previously, mbstring used the same logic for encoding validation as for encoding conversion. However, there are cases where we want to use different logic for validation and conversion. For example, if a string ends up with missing input required by the encoding, or if a character is input that is invalid as an encoding but can be converted, the conversion should succeed and the validation should fail. To achieve this, a function pointer mb_check_fn has been added to struct mbfl_encoding to implement the logic used for validation. Also, added implementation of validation logic for UTF-7, UTF7-IMAP, ISO-2022-JP and JIS. (The same change has already been made to PHP 8.2 and 8.3; see 6fc8d014df. This commit is backporting the change to PHP 8.1.)
show more ...
|
#
57e194e0 |
| 25-Mar-2023 |
Alex Dowad |
Fix compile error in Windows CI job caused by 0779950768 In 6fc8d014df, pakutoma added some additional validation logic to mb_detect_encoding. Since the implementation of mb_detect_encod
Fix compile error in Windows CI job caused by 0779950768 In 6fc8d014df, pakutoma added some additional validation logic to mb_detect_encoding. Since the implementation of mb_detect_encoding has changed significantly between PHP 8.2 and 8.3, when merging this change down from PHP-8.2 into master, I had to port his code over to the new implementation in master. However, I did this in a wrong way. In merge commit 0779950768, the ported code modifies a function argument (to mb_guess_encoding) which is marked 'const'. In the Windows CI job, MS VC++ rightly flags this as a compile error. Adjust the code to accomplish the same thing, but without destructively modifying 'const' arguments.
show more ...
|
#
6fc8d014 |
| 21-Mar-2023 |
pakutoma |
Fix phpGH-10648: add check function pointer into mbfl_encoding Previously, mbstring used the same logic for encoding validation as for encoding conversion. However, there are ca
Fix phpGH-10648: add check function pointer into mbfl_encoding Previously, mbstring used the same logic for encoding validation as for encoding conversion. However, there are cases where we want to use different logic for validation and conversion. For example, if a string ends up with missing input required by the encoding, or if a character is input that is invalid as an encoding but can be converted, the conversion should succeed and the validation should fail. To achieve this, a function pointer mb_check_fn has been added to struct mbfl_encoding to implement the logic used for validation. Also, added implementation of validation logic for UTF-7, UTF7-IMAP, ISO-2022-JP and JIS.
show more ...
|