#
aa553af9 |
| 10-May-2023 |
Ilija Tovilo |
Fix segfault in mb_strrpos/mb_strripos with ASCII encoding and negative offset We're setting the encoding from PHP_FUNCTION(mb_strpos), but mbfl_strpos would discard it, setting it to mb
Fix segfault in mb_strrpos/mb_strripos with ASCII encoding and negative offset We're setting the encoding from PHP_FUNCTION(mb_strpos), but mbfl_strpos would discard it, setting it to mbfl_encoding_pass, making zend_memnrstr fail due to a null-pointer exception. Fixes GH-11217 Closes GH-11220
show more ...
|
#
6df7557e |
| 02-Apr-2023 |
Alex Dowad |
mb_parse_str, mb_http_input, and mb_convert_variables use fast text conversion code for automatic encoding detection For mb_parse_str, when mbstring.http_input (INI parameter) is a list of
mb_parse_str, mb_http_input, and mb_convert_variables use fast text conversion code for automatic encoding detection For mb_parse_str, when mbstring.http_input (INI parameter) is a list of multiple possible text encodings (which is not the case by default), this new implementation is about 25% faster. When mbstring.http_input is a single value, then nothing is changed. (No automatic encoding detection is done in that case.)
show more ...
|
#
b721d0f7 |
| 10-Mar-2023 |
pakutoma |
Fix phpGH-10648: add check function pointer into mbfl_encoding Previously, mbstring used the same logic for encoding validation as for encoding conversion. However, there are ca
Fix phpGH-10648: add check function pointer into mbfl_encoding Previously, mbstring used the same logic for encoding validation as for encoding conversion. However, there are cases where we want to use different logic for validation and conversion. For example, if a string ends up with missing input required by the encoding, or if a character is input that is invalid as an encoding but can be converted, the conversion should succeed and the validation should fail. To achieve this, a function pointer mb_check_fn has been added to struct mbfl_encoding to implement the logic used for validation. Also, added implementation of validation logic for UTF-7, UTF7-IMAP, ISO-2022-JP and JIS. (The same change has already been made to PHP 8.2 and 8.3; see 6fc8d014df. This commit is backporting the change to PHP 8.1.)
show more ...
|
#
6fc8d014 |
| 21-Mar-2023 |
pakutoma |
Fix phpGH-10648: add check function pointer into mbfl_encoding Previously, mbstring used the same logic for encoding validation as for encoding conversion. However, there are ca
Fix phpGH-10648: add check function pointer into mbfl_encoding Previously, mbstring used the same logic for encoding validation as for encoding conversion. However, there are cases where we want to use different logic for validation and conversion. For example, if a string ends up with missing input required by the encoding, or if a character is input that is invalid as an encoding but can be converted, the conversion should succeed and the validation should fail. To achieve this, a function pointer mb_check_fn has been added to struct mbfl_encoding to implement the logic used for validation. Also, added implementation of validation logic for UTF-7, UTF7-IMAP, ISO-2022-JP and JIS.
show more ...
|
#
0ce755be |
| 17-Feb-2023 |
Alex Dowad |
Implement mb_encode_mimeheader using fast text conversion filters The behavior of the new mb_encode_mimeheader implementation closely follows the old implementation, except for three poi
Implement mb_encode_mimeheader using fast text conversion filters The behavior of the new mb_encode_mimeheader implementation closely follows the old implementation, except for three points: • The old implementation was missing a call to the mbfl_convert_filter flush function. So it would sometimes truncate the input string just before its end. • The old implementation would drop zero bytes when QPrint-encoding. So for example, if you tried to QPrint-encode the UTF-32BE string "\x00\x00\x12\x34", its QPrint-encoding would be "=12=34", which does not decode to a valid UTF-32BE string. This is now fixed. • In some rare corner cases, the new implementation will choose to Base64-encode or QPrint-encode the input string, where the old implementation would have just added newlines to it. Specifically, this can happen when there is a non-space ASCII character, followed by a large number of ASCII spaces, followed by a non-ASCII character. The new implementation is around 2.5-8x faster than the old one, depending on the text encoding and transfer encoding used. Performance gains are greater with Base64 transfer encoding than with QPrint transfer encoding; this is not because QPrint-encoding bytes is slow, but because QPrint-encoded output is much bigger than Base64-encoded output and takes more lines, so we have to go through the process of finding the right place to break a line many more times.
show more ...
|
#
157ca654 |
| 06-Feb-2023 |
Alex Dowad |
Implement mb_decode_mimeheader using fast text conversion filters The new implementation is 2.5x-3x faster. If an invalid charset name was used, the old implementation would get
Implement mb_decode_mimeheader using fast text conversion filters The new implementation is 2.5x-3x faster. If an invalid charset name was used, the old implementation would get 'stuck' trying to parse the charset name and would not interpret any other MIME encoded words up to the end of the input string. The new implementation fixes this bug. If an (invalid) encoded word ends abruptly and a new (valid) encoded word starts, the old implementation would not decode the valid encoded word. The new implementation also fixes this. Otherwise, the behavior of the new implementation has been designed to closely match that of the old implementation.
show more ...
|
#
8a73a681 |
| 16-Jan-2023 |
Alex Dowad |
Use fast encoding conversion filters in mb_send_mail
|
Revision tags: php-8.2.0RC1, php-8.1.10, php-8.0.23, php-8.0.23RC1, php-8.1.10RC1, php-8.2.0beta3, php-8.2.0beta2, php-8.1.9, php-8.0.22 |
|
#
0e7160b8 |
| 20-Jul-2022 |
Alex Dowad |
Implement mb_detect_encoding using fast text conversion filters Regarding the optional 3rd `strict` argument to mb_detect_encoding, the documentation states: Controls the beha
Implement mb_detect_encoding using fast text conversion filters Regarding the optional 3rd `strict` argument to mb_detect_encoding, the documentation states: Controls the behaviour when string is not valid in any of the listed encodings. If strict is set to false, the closest matching encoding will be returned; if strict is set to true, false will be returned. (Ref: https://www.php.net/manual/en/function.mb-detect-encoding.php) Because of bugs in the implementation, mb_detect_encoding did not always behave according to this description when `strict` was false. For example: <?php echo var_export(mb_detect_encoding("\xc0\x00", "UTF-8", false)); // Before this commit, prints: false // After this commit, prints: 'UTF-8' Because `strict` is false in the above example, mb_detect_encoding should return the 'closest matching encoding', which is UTF-8, since that is the only candidate encoding. (Incidentally, this example shows that using mb_detect_encoding with a single candidate encoding in non-strict mode is useless.) The new implementation fixes this bug. It also fixes another problem with the old implementation as regards non-strict detection mode: The old implementation would stop processing of the input string using a particular candidate encoding as soon as it saw an error in that encoding, even in non-strict mode. This means that it could not really detect the 'closest matching encoding'; rather, what it would return in non-strict mode was 'the encoding in which the first decoding error is furthest from the beginning of the input string'. In non-strict mode, the new implementation continues trying to process the input string to its end even after seeing an error. This makes it possible to determine in which candidate encoding the string has the smallest number of errors, i.e. the 'closest matching encoding'. Rejecting candidate encodings as soon as it saw an error gave the old implementation a marked performance advantage in non-strict mode; however, the new implementation still beats it in most cases. Here are a few sample microbenchmark results: UTF-8, ~100 codepoints, strict mode Old: 0.080s (100,000 calls) New: 0.026s (" " ) UTF-8, ~100 codepoints, non-strict mode Old: 0.079s (100,000 calls) New: 0.033s (" " ) UTF-8, ~10000 codepoints, strict mode Old: 6.708s (60,000 calls) New: 1.383s (" " ) UTF-8, ~10000 codepoints, non-strict mode Old: 6.705s (60,000 calls) New: 3.044s (" " ) Notice that the old implementation had almost identical performance between strict and non-strict mode, while the new suffers a significant performance penalty for non-strict detection. This is the cost of implementing the behavior specified in the documentation. A couple more sample results: SJIS, ~10000 codepoints, strict mode Old: 4.563s New: 1.084s SJIS, ~10000 codepoints, non-strict mode Old: 4.569s New: 2.863s This is the only case I found where the new implementation loses: UTF-16LE, ~10000 codepoints, non-strict mode Old: 1.514s New: 2.813s The reason is because the test strings happened to be invalid right from the first few bytes for all the candidate encodings except for UTF-16LE; so the old implementation would immediately reject all those encodings and only process the entire string in UTF-16LE. I believe mb_detect_encoding could be made much faster if we identified good criteria for when to reject candidate encodings before reaching the end of the input string.
show more ...
|
#
95386466 |
| 15-Dec-2022 |
Alex Dowad |
Implement php_mb_zend_encoding_converter using fast text conversion filters
|
#
b9cd1cdb |
| 04-Dec-2022 |
Alex Dowad |
Implement mb_substr_count using fast text conversion filters The performance gain from this change depends on the text encoding and input string size. For very small strings, other overh
Implement mb_substr_count using fast text conversion filters The performance gain from this change depends on the text encoding and input string size. For very small strings, other overheads tend to swamp the performance gains to some extent, such that the speedup is less than 2x. For medium-length strings (~100 bytes or so), the speedup is typically around 2.5x. The greatest performance gains are for UTF-8 strings which have already been marked as valid (using the GC flags on the zend_string object); for those, the speedup is more than 10x in many cases. The previous implementation first converted the haystack and needle to wchars, then searched for matches between the two sequences of wchars. Because we use -1 as an error marker when converting to wchars, error markers from invalid byte sequences in the haystack would match error markers from invalid byte sequences in the needle, even if the specific invalid byte sequence was different. I am not sure whether this behavior is really desirable or not, but anyways, this new implementation follows the same behavior so as not to cause BC breaks.
show more ...
|
#
0c0774f5 |
| 04-Oct-2022 |
Alex Dowad |
Use fast text conversion filters for mb_strpos, mb_stripos, mb_substr, etc This boosts the performance of mb_strpos, mb_stripos, mb_strrpos, mb_strripos, mb_strstr, mb_stristr, mb_strrch
Use fast text conversion filters for mb_strpos, mb_stripos, mb_substr, etc This boosts the performance of mb_strpos, mb_stripos, mb_strrpos, mb_strripos, mb_strstr, mb_stristr, mb_strrchr, and mb_strrichr when used on non-UTF-8 strings. mb_substr is also faster. With UTF-8 input, there is no appreciable difference in performance for mb_strpos, mb_stripos, mb_strrpos, etc. This is expected, since the only real difference here (aside from shorter and simpler code) is that the new text conversion code is used when converting non-UTF-8 input strings to UTF-8. (This is done because internally, mb_strpos, etc. work only on UTF-8 text.) For ASCII, speed is boosted by 30-65%. For other legacy text encodings, the degree of performance improvement will depend on how slow the legacy conversion code was. One other minor, but notable difference is that strings encoded using UTF-8 variants from Japanese mobile vendors (SoftBank, KDDI, Docomo) will not undergo encoding conversion but will be processed "as is". It is expected that this will result in a large performance boost for such input strings; but realistically, the number of users who work with such strings is probably minute. I was not originally planning to include mb_substr in this commit, but fuzzing of the reimplemented mb_strstr revealed that mb_substr needed to be reimplemented, too; using the old mbfl_substr, which was based on the old text conversion filters, in combination with functions which use the new text conversion filters caused bugs. The performance boost for mb_substr varies from 10%-500%, depending on the encoding and input string used.
show more ...
|
#
fa0401b0 |
| 16-Sep-2022 |
NathanFreeman <1056159381@qq.com> |
Fix GH-9535 (unintended behavior change for mb_strcut in PHP 8.1) The existing implementation of mb_strcut extracts part of a multi-byte encoded string by pulling out raw bytes and then
Fix GH-9535 (unintended behavior change for mb_strcut in PHP 8.1) The existing implementation of mb_strcut extracts part of a multi-byte encoded string by pulling out raw bytes and then running them through a conversion filter to ensure that the output is valid in the requested encoding. If the conversion filter emits error markers when doing the final 'flush' operation which ends the conversion of the extracted bytes, these error markers may (in some cases) be included in the output. The conversion operation does not respect the value of mb_substitute_character; rather, it always uses '?' as an error marker. So this issue manifests itself as unwanted '?' characters being inserted into the output. This issue has existed for a long time, but became noticeable in PHP 8.1 because for at least some of the supported text encodings, mbstring is now more strict about emitting error markers when strings end in an illegal state. The simplest fix is to suppress error markers during the final flush operation. While working on a fix for this problem, another problem with mb_strcut was discovered; since it decides when to stop consuming bytes from the input by looking at the byte length of its OUTPUT, anything which causes extra bytes to be emitted to the output may cause mb_strcut to not consume all the bytes in the requested range. The one case where we DO emit extra output bytes is for encodings which have a selectable mode, like ISO-2022-JP; if a string in such an encoding ends in a mode which is not the default, we emit an ending escape sequence which changes back to the default mode. This is done so that concatenating strings in such encodings is safe. However, as mentioned, this can cause the output of mb_strcut to be shorter than it logically should be. This bug has existed for a long time, and fixing it now will be a BC break, so we may not fix it right away. Therefore, tests for THIS fix which don't pass because of that OTHER bug have been split out into a separate test file (gh9535b.phpt), and that file has been marked XFAIL.
show more ...
|
#
3ce888a8 |
| 04-Oct-2022 |
Alex Dowad |
Use uint32_t for 'illegal_substchar' codepoint in mbstring This value is a wchar, so the best type for it is uint32_t.
|
Revision tags: php-8.1.9RC1, php-8.2.0beta1, php-8.0.22RC1 |
|
#
72990960 |
| 19-Jul-2022 |
Alex Dowad |
New implementation of mb_strimwidth This new implementation of mb_strimwidth uses the new text encoding conversion filters. Changes from the previous implementation: • mb_st
New implementation of mb_strimwidth This new implementation of mb_strimwidth uses the new text encoding conversion filters. Changes from the previous implementation: • mb_strimwidth allows a negative 'from' argument, which should count backwards from the end of the string. However, the implementation of this feature was buggy (starting right from when it was first implemented). It used the following code: if ((from < 0) || (width < 0)) { swidth = mbfl_strwidth(&string); } if (from < 0) { from += swidth; } Do you see the bug? 'from' is a count of CODEPOINTS, but 'swidth' is a count of TERMINAL COLUMNS. Adding those two together does not make sense. If there were no fullwidth characters in the input string, then the two counts coincide and the feature would work correctly. However, each fullwidth character would throw the result off by one, causing more characters to be skipped than was requested. • mb_strimwidth also allows a negative 'width' argument, which again counts backwards from the end of the string; in this case, it is not determining the START of the portion which we want to extract, but rather, the END of that portion. Perhaps unsurprisingly, this feature was also buggy. Code: if (width < 0) { width = swidth + width - from; } 'swidth + width' is fine here; the problem is '- from'. Again, that is subtracting a count of CODEPOINTS from a count of TERMINAL COLUMNS. In this case, we really need to count the terminal width of the string prefix skipped over by 'from', and subtract that rather than the number of codepoints which are being skipped. As a result, if a 'from' count was passed along with a negative 'width', for every fullwidth character in the skipped prefix, the result of mb_strimwidth was one terminal column wider than requested. Since these situations were covered by unit tests, you might wonder why the bugs were not caught. Well, as far as I can see, it looks like the author of the 'tests' just captured the actual output of mb_strimwidth and defined it as 'correct'. The tests were written in such a way that it was difficult to examine them and see whether they made sense or not; but a careful examination of the inputs and outputs clearly shows that the legacy tests did not conform to the documented contract of mb_strimwidth. • The old implementation would always pass the input string through decoding/encoding filters before returning it to the caller, even if it fit within the specified width. This means that invalid byte sequences would be converted to error markers. For performance, the new implementation returns the very same string which was passed in if it does not exceed the specified width. This means that erroneous byte sequences are not converted to error markers unless it is necessary to trim the string. • The same applies to the 'trim marker' string. • The old implementation was buggy in the (unusual) case that the trim marker is wider than the requested maximum width of the result. It did an unsigned subtraction of the requested width and the width of the trim marker. If the width of the trim marker was greater, that subtraction would underflow and yield a huge number. As a result, mb_strimwidth would then pass the input string through, even if it was far wider than the requested maximum width. In that case, since the input string is wider than the requested width, and NONE of it will fit together with the trim marker, the new implementation returns just the trim marker. This is the one case where the output can be wider than the requested width: when BOTH the input string and also the trim marker are too wide. • Since it passed the input string and trim marker through decoding/encoding filters, when using "Quoted-Printable" as the encoding, newlines could be inserted into the trim marker to maintain the maximum line length for QP. This is an extremely bizarre use case and I don't think there is any point in worrying about it. QP will be removed from mbstring in time, anyways. PERFORMANCE: • From micro-benchmarking with various input string lengths and text encodings, it appears that the new implementation is 2-3x faster for UTF-8 and UTF-16. For legacy Japanese text encodings like ISO-2022-JP or SJIS, the new implementation is perhaps 25% faster. • Note that correctly implementing negative 'from' and 'width' arguments imposes a small performance burden in such cases; one which the old implementation did not pay. This slightly skews benchmarking results in favor of the old implementation. However, even so, the new implementation is faster in all cases which I tested.
show more ...
|
#
9ac49c0d |
| 12-Jul-2022 |
Alex Dowad |
New implementation of mb_convert_kana mb_convert_kana now uses the new text encoding conversion filters. Microbenchmarking shows speed gains of 50%-150% across various text encodings
New implementation of mb_convert_kana mb_convert_kana now uses the new text encoding conversion filters. Microbenchmarking shows speed gains of 50%-150% across various text encodings and input string lengths. The behavior is the same as the old mb_convert_kana except for one fix: if the 'zero codepoint' U+0000 appeared in the input, the old implementation would sometimes drop it, not passing it through to the output. This is now fixed.
show more ...
|
#
30bfeef4 |
| 12-Jul-2022 |
Alex Dowad |
mbfl_strwidth does not need to use legacy conversion filters now ...Because we have the new (faster) conversion filters now for ALL text encodings supported by mbstring.
|
Revision tags: php-8.0.21, php-8.1.8, php-8.2.0alpha3, php-8.1.8RC1, php-8.2.0alpha2, php-8.0.21RC1, php-8.0.20, php-8.1.7, php-8.2.0alpha1, php-7.4.30, php-8.1.7RC1, php-8.0.20RC1 |
|
#
91969e90 |
| 14-May-2022 |
Alex Dowad |
New implementation of mb_{de,en}code_numericentity This new implementation uses the new encoding conversion filters. Aside from fewer LOC and (hopefully) improved readability, the di
New implementation of mb_{de,en}code_numericentity This new implementation uses the new encoding conversion filters. Aside from fewer LOC and (hopefully) improved readability, the differences are as follows: BEHAVIOR CHANGES: - The old implementation used signed arithmetic when operating on the 'convmap'. This meant that results could be surprising when using convmap entries with 1 in the MSB. Further, types like 'int' were used rather than those with a specific bit width, such as 'int32_t'. This meant that results could also depend on the platform width of an 'int'. Now unsigned arithmetic is used, with explicit bit widths. - Similarly, while converting decimal numeric entities, the legacy implementation would ensure that the value never overflowed INT_MAX, and if it did, the entity would be treated as invalid and passed through unconverted. However, that again means that results depend on the platform size of an 'int'. So now, we use a value with explicit bit width (32 bits) to hold the value of a deconverted decimal entity, and ensure that the entity value does not overflow that. Further, because we are using an UNSIGNED 32-bit value rather than a signed one, the ceiling for how large a decimal entity can be is higher now. All of this will probably not affect anyone, since Unicode codepoints above U+10FFFF are invalid anyways. To see the difference, you need to be using a text encoding like UCS-4, which allows huge 'codepoints'. - If it saw something which looked like a hex entity, but turned out not to be a valid numeric entity, the old implementation would sometimes convert the hexadecimal digits a-f to A-F (uppercase). The new implementation passes invalid numeric entities through without performing case conversion. - The old implementation of mb_encode_numericentity was limited in how many decimal/hex digits it could emit. If a text encoding like UCS-4 was in use, where 'codepoints' can have huge values (larger than the valid range stipulated by the Unicode standard), it would not error out on a 'codepoint' whose value was too large for it, but would rather mangle the value and emit a numeric entity which decoded to some other random codepoint. The new implementation is able to emit enough digits to express any value which fits in 32 bits. PERFORMANCE: Based on micro-benchmarks run on my development machine: Decoding numeric HTML entities is about 4 times faster, for both decimal and hexadecimal entities, across a variety of input string lengths. Encoding is about 3 times faster.
show more ...
|
#
9468fa7f |
| 13-May-2022 |
Alex Dowad |
mbfl_strlen does not need to use old conversion filters any more
|
Revision tags: php-8.1.6, php-8.0.19, php-8.1.6RC1, php-8.0.19RC1, php-8.0.18, php-8.1.5, php-7.4.29, php-8.1.5RC1, php-8.0.18RC1 |
|
#
d0417ebc |
| 17-Mar-2022 |
Christoph M. Becker |
Fix GH-8208: mb_encode_mimeheader: $indent functionality broken We also need to factor in the indent, when getting the encoder result. Closes GH-8213.
|
Revision tags: php-8.1.4, php-8.0.17, php-8.1.4RC1, php-8.0.17RC1, php-8.1.3, php-8.0.16, php-7.4.28, php-8.1.3RC1, php-8.0.16RC1, php-8.1.2, php-8.0.15, php-8.1.2RC1, php-8.0.15RC1, php-8.0.14, php-8.1.1, php-7.4.27, php-8.1.1RC1, php-8.0.14RC1, php-7.4.27RC1, php-8.1.0, php-8.0.13, php-7.4.26, php-7.3.33, php-8.1.0RC6, php-7.4.26RC1, php-8.0.13RC1, php-8.1.0RC5, php-7.3.32, php-7.4.25, php-8.0.12, php-8.1.0RC4, php-8.0.12RC1, php-7.4.25RC1, php-8.1.0RC3, php-8.0.11, php-7.4.24, php-7.3.31, php-8.1.0RC2, php-7.4.24RC1, php-8.0.11RC1, php-8.1.0RC1, php-7.4.23, php-8.0.10, php-7.3.30, php-8.1.0beta3, php-8.0.10RC1, php-7.4.23RC1, php-8.1.0beta2, php-8.0.9, php-7.4.22 |
|
#
3c732251 |
| 21-Jul-2021 |
Alex Dowad |
New internal interface for fast text conversion in mbstring When converting text to/from wchars, mbstring makes one function call for each and every byte or wchar to be converted. Typica
New internal interface for fast text conversion in mbstring When converting text to/from wchars, mbstring makes one function call for each and every byte or wchar to be converted. Typically, each of these conversion functions contains a state machine, and its state has to be restored and then saved for every single one of these calls. It doesn't take much to see that this is grossly inefficient. Instead of converting one byte or wchar on each call, the new conversion functions will either fill up or drain a whole buffer of wchars on each call. In benchmarks, this is about 3-10× faster. Adding the new, faster conversion functions for all supported legacy text encodings still needs some work. Also, all the code which uses the old-style conversion functions needs to be converted to use the new ones. After that, the old code can be dropped. (The mailparse extension will also have to be fixed up so it will still compile.)
show more ...
|
#
28b346bc |
| 09-Oct-2021 |
Alex Dowad |
Improve detection accuracy of mb_detect_encoding Originally, `mb_detect_encoding` essentially just checked all candidate encodings to see which ones the input string was valid in. Howeve
Improve detection accuracy of mb_detect_encoding Originally, `mb_detect_encoding` essentially just checked all candidate encodings to see which ones the input string was valid in. However, it was only able to do this for a limited few of all the text encodings which are officially supported by mbstring. In 3e7acf901d, I modified it so it could 'detect' any text encoding supported by mbstring. While this is arguably an improvement, if the only text encodings one is interested in are those which `mb_detect_encoding` could originally handle, the old `mb_detect_encoding` may have been preferable. Because the new one has more possible encodings which it can guess, it also has more chances to get the answer wrong. This commit adjusts the detection heuristics to provide accurate detection in a wider variety of scenarios. While the previous detection code would frequently confuse UTF-32BE with UTF-32LE or UTF-16BE with UTF-16LE, the adjusted code is extremely accurate in those cases. Detection for Chinese text in Chinese encodings like GB18030 or BIG5 and for Japanese text in Japanese encodings like EUC-JP or SJIS is greatly improved. Detection of UTF-7 is also greatly improved. An 8KB table, with one bit for each codepoint from U+0000 up to U+FFFF, is used to achieve this. One significant constraint is that the heuristics are completely based on looking at each codepoint in a string in isolation, treating some codepoints as 'likely' and others as 'unlikely'. It might still be possible to achieve great gains in detection accuracy by looking at sequences of codepoints rather than individual codepoints. However, this might require huge tables. Further, we might need a huge corpus of text in various languages to derive those tables. Accuracy is still dismal when trying to distinguish single-byte encodings like ISO-8859-1, ISO-8859-2, KOI8-R, and so on. This is because the valid bytes in these encodings are basically all the same, and all valid bytes decode to 'likely' codepoints, so our method of detection (which is based on rating codepoints as likely or unlikely) cannot tell any difference between the candidates at all. It just selects the first encoding in the provided list of candidates. Speaking of which, if one wants to get good results from `mb_detect_encoding`, it is important to order the list of candidate encodings according to your prior belief of which are more likely to be correct. When the function cannot tell any difference between two candidates, it returns whichever appeared earlier in the array.
show more ...
|
#
0b32a15e |
| 22-Sep-2021 |
Alex Dowad |
Optimize mb_str{,im}width for performance Rather than doing a linear search of a table of fullwidth codepoint ranges for every input character, 1) Short-cut the search if the co
Optimize mb_str{,im}width for performance Rather than doing a linear search of a table of fullwidth codepoint ranges for every input character, 1) Short-cut the search if the codepoint is below the first such range 2) Otherwise, do a binary (rather than linear) search
show more ...
|
#
3bf43196 |
| 18-Sep-2021 |
Alex Dowad |
Don't check for impossible error condition in mb_substr_count
|
Revision tags: php-8.1.0beta1, php-7.4.22RC1, php-8.0.9RC1, php-8.1.0alpha3, php-7.4.21, php-7.3.29, php-8.0.8, php-8.1.0alpha2, php-7.4.21RC1, php-8.0.8RC1, php-8.1.0alpha1, php-8.0.7, php-7.4.20, php-8.0.7RC1, php-7.4.20RC1, php-8.0.6, php-7.4.19, php-7.4.18, php-7.3.28, php-8.0.5, php-8.0.5RC1, php-7.4.18RC1, php-8.0.4RC1, php-7.4.17RC1, php-8.0.3, php-7.4.16, php-8.0.3RC1, php-7.4.16RC1, php-8.0.2, php-7.4.15, php-7.3.27, php-8.0.2RC1, php-7.4.15RC2, php-7.4.15RC1, php-8.0.1, php-7.4.14, php-7.3.26, php-7.4.14RC1, php-8.0.1RC1, php-7.3.26RC1, php-8.0.0, php-7.3.25, php-7.4.13, php-8.0.0RC5, php-7.4.13RC1, php-8.0.0RC4, php-7.3.25RC1, php-7.4.12, php-8.0.0RC3, php-7.3.24 |
|
#
8c32deb6 |
| 18-Oct-2020 |
Alex Dowad |
Don't check for impossible error condition in mb_strwidth
|
#
bf78070c |
| 18-Oct-2020 |
Alex Dowad |
Don't check for impossible error condition in mb_strlen
|