Error handling for UTF-8 complies with WHATWG specification

In 7502c86342, I adjusted the number of error markers emitted on
invalid UTF-8 text to be more consistent with mbstring's beha

Error handling for UTF-8 complies with WHATWG specification

In 7502c86342, I adjusted the number of error markers emitted on
invalid UTF-8 text to be more consistent with mbstring's behavior on
other text encodings (generally, it emits one error marker for one
unexpected byte). I didn't expect that anybody would actually care one
way or the other, but felt that it was better to be consistent than
not.

Later, Martin Auswöger kindly pointed out that the WHATWG encoding
specification, which governs how various text encodings are handled
by web browsers, does actually specify how many error markers should
be generated for any given piece of invalid UTF-8 text.

Until now, we have never really paid much attention to the WHATWG
specification, but we do want to comply with as many relevant
specifications as possible. And since PHP is commonly used for web
applications, compatibility with the behavior of web browsers is
obviously a good thing.

show more ...

New internal interface for fast text conversion in mbstring

When converting text to/from wchars, mbstring makes one function call
for each and every byte or wchar to be converted. Typica

New internal interface for fast text conversion in mbstring

When converting text to/from wchars, mbstring makes one function call
for each and every byte or wchar to be converted. Typically, each of
these conversion functions contains a state machine, and its state has
to be restored and then saved for every single one of these calls.
It doesn't take much to see that this is grossly inefficient.

Instead of converting one byte or wchar on each call, the new
conversion functions will either fill up or drain a whole buffer of
wchars on each call. In benchmarks, this is about 3-10× faster.

Adding the new, faster conversion functions for all supported legacy
text encodings still needs some work. Also, all the code which uses
the old-style conversion functions needs to be converted to use the
new ones. After that, the old code can be dropped. (The mailparse
extension will also have to be fixed up so it will still compile.)

show more ...

Add test suite for UTF-{7,8,16,32}

Also fix a couple small problems with UTF-32 and UTF-8 support:

- UTF-32 would pass very large codepoints (>= 0x80000000), which are
not val

Add test suite for UTF-{7,8,16,32}

Also fix a couple small problems with UTF-32 and UTF-8 support:

- UTF-32 would pass very large codepoints (>= 0x80000000), which are
not valid.
- UTF-8 would sometimes emit two error marker characters for a single
bad input byte.

show more ...

Migrate more SKIPIF -> EXTENSIONS (#7139)

This is a mix of more automated and manual migration. It should remove all applicable extension_loaded() checks outside of skipif.inc files.

Leading BOM is stripped for UTF-32

For consistency with UTF-16 and UCS-4.

Also, do some code cleanup.

Reindent phpt files

Trim trailing whitespace in *.phpt

Trim trailing whitespace in *.phpt

Trim trailing whitespace in *.phpt

Trim trailing whitespace in *.phpt

MFH: fixed #65045: mb_convert_encoding breaks well-formed character.

MFH: fixed #65045: mb_convert_encoding breaks well-formed character.

updated test script for bug #65045.

MFH: fixed test case failures.

fixed test case failures.

- MFH: strictly check UTF-8 and UTF-32 validity