mbstring no longer provides 'long' substitutions for erroneous input bytes

Previously, mbstring had a special mode whereby it would convert
erroneous input byte sequences to output like

mbstring no longer provides 'long' substitutions for erroneous input bytes

Previously, mbstring had a special mode whereby it would convert
erroneous input byte sequences to output like "BAD+XXXX", where "XXXX"
would be the erroneous bytes expressed in hexadecimal. This mode could
be enabled by calling `mb_substitute_character("long")`.

However, accurately reproducing input byte sequences from the cached
state of a conversion filter is often tricky, and this significantly
complicates the implementation. Further, the means used for passing
the erroneous bytes through to where the "BAD+XXXX" text is generated
only allows for up to 3 bytes to be passed, meaning that some erroneous
byte sequences are truncated anyways.

More to the point, a search of publically available PHP code indicates
that nobody is really using this feature anyways.

Incidentally, this feature also provided error output like "JIS+XXXX"
if the input 'should have' represented a JISX 0208 codepoint, but it
decodes to a codepoint which does not exist in the JISX 0208 charset.
Similarly, specific error output was provided for non-existent
JISX 0212 codepoints, and likewise for JISX 0213, CP932, and a few
other charsets. All of that is now consigned to the flames.

However, "long" error markers also include a somewhat more useful
"U+XXXX" marker for Unicode codepoints which were successfully
decoded from the input text, but cannot be represented in the output
encoding. Those are still supported.

With this change, there is no need to use a variety of special values
in the high bits of a wchar to represent different types of error
values. We can (and will) just use a single error value. This will be
equal to -1.

One complicating factor: Text conversion functions return an integer to
indicate whether the conversion operation should be immediately
aborted, and the magic 'abort' marker is -1. Also, almost all of these
functions would return the received byte/codepoint to indicate success.
That doesn't work with the new error value; if an input filter detects
an error and passes -1 to the output filter, and the output filter
returns it back, that would be taken to mean 'abort'.

Therefore, amend all these functions to return 0 for success.

show more ...

Remove useless constant MBFL_ENCTYPE_MBCS

This flag indicated that an encoding was 'multi-byte'; it can use a variable
number of bytes to encode each character. As it turns out, we don't

Remove useless constant MBFL_ENCTYPE_MBCS

This flag indicated that an encoding was 'multi-byte'; it can use a variable
number of bytes to encode each character. As it turns out, we don't actually
need to check this flag anywhere, so it's better to remove it.

show more ...

Combine MBFL_ENCTYPE_MWC2{BE,LE} constants

These constants indicate that a text encoding uses 2+ bytes for each character,
and is either big endian or little endian (respectively). But n

Combine MBFL_ENCTYPE_MWC2{BE,LE} constants

These constants indicate that a text encoding uses 2+ bytes for each character,
and is either big endian or little endian (respectively). But nothing in
mbstring cares about the difference between MBFL_ENCTYPE_MWC2BE and
MBFL_ENCTYPE_MWC2LE.

(Actually, nothing cares about whether these flags are set at all...
maybe we should just remove them?)

show more ...

Combine MBFL_ENCTYPE_WCS{2,4}{BE,LE} constants

These flags identify text encodings in mbstring which use a constant number of
bytes per character. While some parts of the code do use the

Combine MBFL_ENCTYPE_WCS{2,4}{BE,LE} constants

These flags identify text encodings in mbstring which use a constant number of
bytes per character. While some parts of the code do use these flags, usually
to detect cases which can be optimized due to constant-width encoding, nothing
cares whether the encodings are 'LE' (little-endian) or 'BE' (big-endian).

So we can simplify things by combining constants.

show more ...

Consolidate all single-byte encodings in one source file

We can squeeze out a lot of duplicated code in this way.

Fix mbstring support for ARMSCII-8

- Identify filter was completely wrong.
- Respect `mb_substitute_character` rather than converting invalid bytes to
Unicode 0xFFFD (generic repla

Fix mbstring support for ARMSCII-8

- Identify filter was completely wrong.
- Respect `mb_substitute_character` rather than converting invalid bytes to
Unicode 0xFFFD (generic replacement character).
- Don't convert Unicode 0xFFFD to a valid ARMSCII-8 character.
- When converting ARMSCII-8 to ARMSCII-8, don't pass invalid bytes through
silently.

show more ...

Remove dead code from mbfilter_koi8u.c (and do general code cleanup)

Remove dead code from mbfilter_iso8859_{2,4,5,9,10,13,14,15,16}.c

...Plus some dead code related to ISO-8859-1.

Remove dead code from mbfilter_koi8r.c

Remove dead code from mbfilter_cp850.c (and do general code cleanup)

Since there are no invalid bytes in CP850, these `if` conditions will never
be true.

Remove dead code from mbfilter_cp866.c (and do general code cleanup)

Since there are no invalid bytes in CP866, these `if` conditions will never
be true.

Fix mbstring support for CP1254 encoding

One funny thing: while the original author used Unicode 0xFFFD (generic
replacement character) for invalid bytes in CP1251 and CP1252, for CP1254

Fix mbstring support for CP1254 encoding

One funny thing: while the original author used Unicode 0xFFFD (generic
replacement character) for invalid bytes in CP1251 and CP1252, for CP1254
they used 0xFFFE, which is not a valid Unicode codepoint at all, but is a
reversed byte-order mark. Probably this was by mistake.

Anyways,

- Fixed identify filter, which was completely wrong.
- Don't convert Unicode 0xFFFE to a random (but valid) CP1254 byte.
- When converting CP1254 to CP1254, don't pass invalid bytes through silently.

show more ...

Fix mbstring support for CP1251 encoding

- Identify filter was as wrong as wrong can be.
- Invalid CP1251 byte 0x98 was converted to Unicode 0xFFFD (generic
replacement character),

Fix mbstring support for CP1251 encoding

- Identify filter was as wrong as wrong can be.
- Invalid CP1251 byte 0x98 was converted to Unicode 0xFFFD (generic
replacement character), rather than respecting `mb_substitute_character`.
- Unicode 0xFFFD was converted to some random CP1251 byte.
- When converting CP1251 to CP1251, don't pass invalid bytes through silently.

show more ...

Comment constants in mbfl_consts.h, remove unused ones

These were unused, and almost certainly will never be used:

- MBFL_ENCTYPE_MWC4BE
- MBFL_ENCTYPE_MWC4LE
- MBFL_ENCTYPE

Comment constants in mbfl_consts.h, remove unused ones

These were unused, and almost certainly will never be used:

- MBFL_ENCTYPE_MWC4BE
- MBFL_ENCTYPE_MWC4LE
- MBFL_ENCTYPE_SHFTCODE
- MBFL_ENCTYPE_ENC_STRM

For the latter two, there were some encodings which were marked with these flags;
but nothing ever _checked_ these particular flags.

show more ...

trailing whitespace removal

typo fixes - https://github.com/vlajos/misspell_fixer

updated libmbfl to 1.3.2 (JISX-0213:2004 support).

added gb18030 encoding to mbstring/libmbfl.~

* Refactor zend_multibyte facility.
Now mbstring.script_encoding is superseded by zend.script_encoding.

- Add support for CP850 encoding (patch by Denis Giffeler)

- MFH: strictly check UTF-8 and UTF-32 validity

- added koi8-u (Ukrainian) and CP1254 (Turkish).

- Add Armenian encoding / NLS (patch by Hayk Chamyan)

- Update libmbfl (fixes bug #30549 and #31911).
- Update oniguruma to 3.7.0

@mbfilter is changed to bundled version of libmbfl to prevent LGPL licence problem. mbregex is changed to oniguruma.