Fix bug in mb_get_substr_slow (sometimes outputs wrong number of characters)

Thanks to Maurício Fauth for finding and reporting this bug.

The bug was introduced in October 2022. It

Fix bug in mb_get_substr_slow (sometimes outputs wrong number of characters)

Thanks to Maurício Fauth for finding and reporting this bug.

The bug was introduced in October 2022. It originally only affected
text encodings which do not have a fixed byte width per characters
and for which mbstring does not have an mblen_table. However, I recently
made another change to mbstring, such that mb_substr no longer relies
on the mblen_table even if one is available. Because of this change,
the bug earlier introduced in October 2022 now affected a greater
number of text encodings, including UTF-8.

show more ...

Character indices used by mb_strpos and mb_substr have same meaning, even on invalid strings

Starting many years ago, libmbfl included a 'mblen_table' for selected
text encodings. This t

Character indices used by mb_strpos and mb_substr have same meaning, even on invalid strings

Starting many years ago, libmbfl included a 'mblen_table' for selected
text encodings. This table allows looking up the byte length of a
(possibly multi-byte) character from the value of the first byte.
libmbfl uses these tables to optimize certain operations; if a
text-processing operation can be performed using an mblen_table,
it may not be necessary to decode the text to codepoints. Since
libmbfl's decoding filters are generally slow, this improves
performance.

Since mbstring is (or was) based on libmbfl, it has always used
these mblen_tables to implement some functions. This design has a
significant downside. Let me explain:

While some mbstring functions are implemented by converting input text
to codepoints and operating on the codepoints, others operate directly
on the original input bytes (using an mblen_table to identify character
boundaries). Both of these implementation styles, if correctly coded,
yield equivalent results on valid strings. However, on strings which
contain encoding errors, the results are often different.

When decoding byte strings to codepoints using some text encoding,
mbstring uses the non-existent codepoint 0xFFFFFFFF to represent a
byte sequence which cannot be decoded. Then, when mbstring indexes into
the resulting sequence of codepoints, the index of any particular
character depends on the number of such 'error markers' which were
produced during the decoding process. In contrast, when an mblen_table
is used to split a byte sequence into characters, there is no question
of counting encoding errors; rather, table lookups into the mblen_table
are used to repeatedly 'bite off' some number of bytes (which are
treated as one 'character'). In the presence of encoding errors, these
two methods of mapping between byte indices and character indices are
inherently different and will rarely agree.

(For completeness, it must be said that some internal mbstring code
which operates only on UTF-8 text uses a third method for mapping
between byte indices and character indices, that is: counting
non-continuation UTF-8 bytes, which are all bytes whose binary
representation is NOT like 0b10xxxxxx. This method happens to agree with
the method which involves decoding the input text to codepoints and then
counting the codepoints.)

I have been aware of this issue for years, but only recently became
aware that in the case of mb_strstr, mb_strpos, and mb_substr,
this issue can cause seriously unintuitive behavior (and even security
vulnerabilities). This was reported by Stefan Schiller.

Stefan Schiller shared the following example for mb_strstr:

var_dump(mb_strstr("\xf0start", "start", false, "UTF-8"));
// string(2) "rt"

Similarly, when mb_strpos and mb_substr are used to identify and
extract a substring from a string with encoding errors, Stefan Schiller
pointed out that the extracted portion may be completely different than
desired. This is because (for UTF-8 strings) mb_strpos works by counting
non-continuation bytes, but mb_substr uses an mblen_table.

Since some mbstring functions *cannot* be implemented using an
mblen_table, as long as mblen_tables are used, similar inconsistencies
cannot be totally avoided. But the mblen_tables are critical to
mbstring's performance. Or are they? Benchmarking mb_substr on various
UTF-8, SJIS, and EUC-JP strings revealed something interesting.
On all SJIS and EUC-JP test cases, mb_substr was slightly faster when
the mblen_table based code was deleted. For some UTF-8 test cases, the
mblen_table-based code was a tiny bit faster, while for others the
fallback code was a touch faster; in no case was the difference
significant.

Therefore, the simple fix is to delete the mblen_table-based
implementation of mb_substr.

Aside from making the function behave consistently with other mbstring
functions on invalid strings, there is ONE case where behavior is now
different on valid strings: that is, on SJIS-Mac (MacJapanese) strings
which contain any of the following code units:

0x85AB-0x85AD, 0x85BF, 0x85C0, 0x85C1, 0x8645, 0x864B, 0x865D, 0x869E,
0x86CE, 0x86D3-0x86D5, 0x86D6, 0x8971, 0x8792, 0x879D, 0x87FB, 0x87FC,
0xEB41, 0xEB42, 0xEB50, 0xEB5B, 0xEB5D, 0xEB60-0xEB6E, and all from
0xEB81 and above.

All of these SJIS-Mac code units share the (very unusual) property that
they do not correspond to any one Unicode codepoint. When converting
from SJIS-Mac to Unicode, these must be converted to 2, 3, 4, or 5
codepoints each.

The previous, mblen_table-based implementation of mb_substr would treat
all of these SJIS-Mac byte sequences as 'one character'. Now, they are
treated as multiple characters (one for each of the Unicode codepoints
which they decode to). The new behavior is more consistent with other
mbstring functions.

I don't know if SJIS-Mac users will like this change or not (probably
most will never notice), but the BC break is justified by the very
real security impact of the previous, inconsistent behavior.

Finally, I should comment on whether similar changes are needed
elsewhere. The remaining functions which use an mblen_table are:
mb_str_split, mb_strcut, and various search functions (such as
mb_strpos). The search functions are only affected now when they
receive a positive 'offset' parameter specifying where to start
searching from.

The search functions should definitely be fixed so they do not use
an mblen_table to implement the 'offset' parameter. I am not convinced
that there is any good reason to change mb_str_split and mb_strcut.

show more ...

Add unit tests for mb_str_split/mb_substr on MacJapanese encoding

MacJapanese has a somewhat unusual feature that when mapped to
Unicode, many characters map to sequences of several code

Add unit tests for mb_str_split/mb_substr on MacJapanese encoding

MacJapanese has a somewhat unusual feature that when mapped to
Unicode, many characters map to sequences of several codepoints.
Add test cases demonstrating how mb_str_split and mb_substr behave in
this situation.

When adding these tests, I found the behavior of mb_substr was wrong
due to an inconsistency between the string "length" as measured by
mb_strlen and the number of native MacJapanese characters which
mb_substr would count when iterating over the string using the
mblen_table. This has been fixed.

I believe that mb_strstr will also return wrong results in some cases
for MacJapanese. I still need to come up with unit tests which
demonstrate the problem and figure out how to fix it.

show more ...

Correct entry for 0x80,0xFD-FF in SJIS multi-byte character length table

As a performance optimization, mbstring implements some functions using
tables which give the (byte) length of a

Correct entry for 0x80,0xFD-FF in SJIS multi-byte character length table

As a performance optimization, mbstring implements some functions using
tables which give the (byte) length of a multi-byte character using a
lookup based on the value of the first byte. These tables are called
`mblen_table`.

For many years, the mblen_table for SJIS has had '2' in position 0x80.
That is wrong; it should have been '1'. Reasons:

For SJIS, SJIS-2004, and mobile variants of SJIS, 0x80 has never been
treated as the first byte of a 2-byte character. It has always been
treated as a single erroneous byte. On the other hand, 0x80 is a valid
character in MacJapanese... but a 1-byte character, not a 2-byte one.

The same applies to bytes 0xFD-FF; these are 1-byte characters in
MacJapanese, and in other SJIS variants, they are not valid (as the
first byte of a character).

Thanks to the GitHub user 'youkidearitai' for finding this problem.

show more ...

Use fast text conversion filters for mb_strpos, mb_stripos, mb_substr, etc

This boosts the performance of mb_strpos, mb_stripos, mb_strrpos,
mb_strripos, mb_strstr, mb_stristr, mb_strrch

Use fast text conversion filters for mb_strpos, mb_stripos, mb_substr, etc

This boosts the performance of mb_strpos, mb_stripos, mb_strrpos,
mb_strripos, mb_strstr, mb_stristr, mb_strrchr, and mb_strrichr when
used on non-UTF-8 strings. mb_substr is also faster.

With UTF-8 input, there is no appreciable difference in performance for
mb_strpos, mb_stripos, mb_strrpos, etc. This is expected, since the only
real difference here (aside from shorter and simpler code) is that the
new text conversion code is used when converting non-UTF-8 input strings
to UTF-8. (This is done because internally, mb_strpos, etc. work only
on UTF-8 text.)

For ASCII, speed is boosted by 30-65%. For other legacy text encodings,
the degree of performance improvement will depend on how slow the
legacy conversion code was.

One other minor, but notable difference is that strings encoded using
UTF-8 variants from Japanese mobile vendors (SoftBank, KDDI, Docomo)
will not undergo encoding conversion but will be processed "as is". It
is expected that this will result in a large performance boost for
such input strings; but realistically, the number of users who work
with such strings is probably minute.

I was not originally planning to include mb_substr in this commit, but
fuzzing of the reimplemented mb_strstr revealed that mb_substr needed
to be reimplemented, too; using the old mbfl_substr, which was based
on the old text conversion filters, in combination with functions which
use the new text conversion filters caused bugs.

The performance boost for mb_substr varies from 10%-500%, depending
on the encoding and input string used.

show more ...

Add more tests for mb_substr

Migrate more SKIPIF -> EXTENSIONS (#7139)

This is a mix of more automated and manual migration. It should remove all applicable extension_loaded() checks outside of skipif.inc files.

Remove mbstring.func_overload

Deprecated in PHP 7.2 as part of
https://wiki.php.net/rfc/deprecations_php_7_2.

Sync leading and final newlines in *.phpt sections

This patch adds missing newlines, trims multiple redundant final
newlines into a single one, and trims redundant leading newlines in al

Sync leading and final newlines in *.phpt sections

This patch adds missing newlines, trims multiple redundant final
newlines into a single one, and trims redundant leading newlines in all
*.phpt sections.

According to POSIX, a line is a sequence of zero or more non-' <newline>'
characters plus a terminating '<newline>' character. [1] Files should
normally have at least one final newline character.

C89 [2] and later standards [3] mention a final newline:
"A source file that is not empty shall end in a new-line character,
which shall not be immediately preceded by a backslash character."

Although it is not mandatory for all files to have a final newline
fixed, a more consistent and homogeneous approach brings less of commit
differences issues and a better development experience in certain text
editors and IDEs.

[1] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206
[2] https://port70.net/~nsz/c/c89/c89-draft.html#2.1.1.2
[3] https://port70.net/~nsz/c/c99/n1256.html#5.1.1.2

show more ...

Sync leading and final newlines in *.phpt sections

This patch adds missing newlines, trims multiple redundant final
newlines into a single one, and trims redundant leading newlines in al

Sync leading and final newlines in *.phpt sections

This patch adds missing newlines, trims multiple redundant final
newlines into a single one, and trims redundant leading newlines in all
*.phpt sections.

According to POSIX, a line is a sequence of zero or more non-' <newline>'
characters plus a terminating '<newline>' character. [1] Files should
normally have at least one final newline character.

C89 [2] and later standards [3] mention a final newline:
"A source file that is not empty shall end in a new-line character,
which shall not be immediately preceded by a backslash character."

Although it is not mandatory for all files to have a final newline
fixed, a more consistent and homogeneous approach brings less of commit
differences issues and a better development experience in certain text
editors and IDEs.

[1] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206
[2] https://port70.net/~nsz/c/c89/c89-draft.html#2.1.1.2
[3] https://port70.net/~nsz/c/c99/n1256.html#5.1.1.2

show more ...

Sync leading and final newlines in *.phpt sections

This patch adds missing newlines, trims multiple redundant final
newlines into a single one, and trims redundant leading newlines in al

Sync leading and final newlines in *.phpt sections

This patch adds missing newlines, trims multiple redundant final
newlines into a single one, and trims redundant leading newlines in all
*.phpt sections.

According to POSIX, a line is a sequence of zero or more non-' <newline>'
characters plus a terminating '<newline>' character. [1] Files should
normally have at least one final newline character.

C89 [2] and later standards [3] mention a final newline:
"A source file that is not empty shall end in a new-line character,
which shall not be immediately preceded by a backslash character."

Although it is not mandatory for all files to have a final newline
fixed, a more consistent and homogeneous approach brings less of commit
differences issues and a better development experience in certain text
editors and IDEs.

[1] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206
[2] https://port70.net/~nsz/c/c89/c89-draft.html#2.1.1.2
[3] https://port70.net/~nsz/c/c99/n1256.html#5.1.1.2

show more ...

Sync leading and final newlines in *.phpt sections

This patch adds missing newlines, trims multiple redundant final
newlines into a single one, and trims redundant leading newlines in al

Sync leading and final newlines in *.phpt sections

This patch adds missing newlines, trims multiple redundant final
newlines into a single one, and trims redundant leading newlines in all
*.phpt sections.

According to POSIX, a line is a sequence of zero or more non-' <newline>'
characters plus a terminating '<newline>' character. [1] Files should
normally have at least one final newline character.

C89 [2] and later standards [3] mention a final newline:
"A source file that is not empty shall end in a new-line character,
which shall not be immediately preceded by a backslash character."

Although it is not mandatory for all files to have a final newline
fixed, a more consistent and homogeneous approach brings less of commit
differences issues and a better development experience in certain text
editors and IDEs.

[1] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206
[2] https://port70.net/~nsz/c/c89/c89-draft.html#2.1.1.2
[3] https://port70.net/~nsz/c/c99/n1256.html#5.1.1.2

show more ...

Remove the b prefix from literals on unrelated tests

MBString tests: checked on PHP 5.2.6, 5.3 and 6.0 (Windows, Linux and Linux 64 bit).

- Fix include_path.

fix tests.

Disabled output_handler in INI section

Clean up.

Renamed the test cases.