mbstring no longer provides 'long' substitutions for erroneous input bytes

Previously, mbstring had a special mode whereby it would convert
erroneous input byte sequences to output like

mbstring no longer provides 'long' substitutions for erroneous input bytes

Previously, mbstring had a special mode whereby it would convert
erroneous input byte sequences to output like "BAD+XXXX", where "XXXX"
would be the erroneous bytes expressed in hexadecimal. This mode could
be enabled by calling `mb_substitute_character("long")`.

However, accurately reproducing input byte sequences from the cached
state of a conversion filter is often tricky, and this significantly
complicates the implementation. Further, the means used for passing
the erroneous bytes through to where the "BAD+XXXX" text is generated
only allows for up to 3 bytes to be passed, meaning that some erroneous
byte sequences are truncated anyways.

More to the point, a search of publically available PHP code indicates
that nobody is really using this feature anyways.

Incidentally, this feature also provided error output like "JIS+XXXX"
if the input 'should have' represented a JISX 0208 codepoint, but it
decodes to a codepoint which does not exist in the JISX 0208 charset.
Similarly, specific error output was provided for non-existent
JISX 0212 codepoints, and likewise for JISX 0213, CP932, and a few
other charsets. All of that is now consigned to the flames.

However, "long" error markers also include a somewhat more useful
"U+XXXX" marker for Unicode codepoints which were successfully
decoded from the input text, but cannot be represented in the output
encoding. Those are still supported.

With this change, there is no need to use a variety of special values
in the high bits of a wchar to represent different types of error
values. We can (and will) just use a single error value. This will be
equal to -1.

One complicating factor: Text conversion functions return an integer to
indicate whether the conversion operation should be immediately
aborted, and the magic 'abort' marker is -1. Also, almost all of these
functions would return the received byte/codepoint to indicate success.
That doesn't work with the new error value; if an input filter detects
an error and passes -1 to the output filter, and the output filter
returns it back, that would be taken to mean 'abort'.

Therefore, amend all these functions to return 0 for success.

show more ...

Refactor mbfl_ident.c, mbfl_encoding.c, mbfl_memory_device.c, mbfl_string.c

- Make everything less gratuitously verbose
- Don't litter the code with lots of unneeded NULL checks (for thi

Refactor mbfl_ident.c, mbfl_encoding.c, mbfl_memory_device.c, mbfl_string.c

- Make everything less gratuitously verbose
- Don't litter the code with lots of unneeded NULL checks (for things which
will never be NULL)
- Don't return success/failure code from functions which can never fail
- For encoding structs, don't use pointers to pointers to pointers for the
list of alias strings. Pointers to pointers (2 levels of indirection)
is what actually makes sense. This gets rid of some extraneous
dereference operations.

show more ...

Remove redundant includes from mbstring (and make sure correct config.h is used)

Very interesting... it turns out that when Valgrind support was enabled,
`#include "config.h"` from withi

Remove redundant includes from mbstring (and make sure correct config.h is used)

Very interesting... it turns out that when Valgrind support was enabled,
`#include "config.h"` from within mbstring was actually including the file "config.h"
from Valgrind, and not the one from mbstring!!

This is because -I/usr/include/valgrind was added to the compiler invocation _before_
-Iext/mbstring/libmbfl.

Make sure we actually include the file which was intended.

show more ...

Remove unused functions from mbstring

- mbfl_buffer_converter_reset
- mbfl_buffer_converter_strncat
- mbfl_buffer_converter_getbuffer
- mbfl_oddlen
- mbfl_filter_output_pipe_

Remove unused functions from mbstring

- mbfl_buffer_converter_reset
- mbfl_buffer_converter_strncat
- mbfl_buffer_converter_getbuffer
- mbfl_oddlen
- mbfl_filter_output_pipe_flush
- mbfl_memory_device_output2
- mbfl_memory_device_output4
- mbfl_is_support_encoding
- mbfl_buffer_converter_feed2
- _php_mb_regex_globals_dtor
- mime_header_encoder_feed
- mime_header_decoder_feed
- mbfl_convert_filter_feed

show more ...

Inline unneeded indirection for mbstring memory management

All memory allocation and deallocation for mbstring bounces through a table of
function pointers before going to emalloc/efree/

Inline unneeded indirection for mbstring memory management

All memory allocation and deallocation for mbstring bounces through a table of
function pointers before going to emalloc/efree/etc. But this is unnecessary.
The allocators are never swapped out. Better to just call them directly.

show more ...

Spec mbfl allocators as infallible

And remove all NULL checks.

Revert "Remove dead code in libmbfl, memory device"

Stop trusting CLion's automatic dead code detection because it seems to be wrong
more often than not.
This reverts commit 612c86db

Revert "Remove dead code in libmbfl, memory device"

Stop trusting CLion's automatic dead code detection because it seems to be wrong
more often than not.
This reverts commit 612c86db3ea139e4ed4aa6a9aa28da343c9b39b8.

show more ...

Remove dead code in libmbfl, memory device

Remove HAVE_STDDEF_H

The `<stddef.h>` header file is part of the standard C89 headers [1] and
on current systems there is no need for a manual check if header is
present.

Si

Remove HAVE_STDDEF_H

The `<stddef.h>` header file is part of the standard C89 headers [1] and
on current systems there is no need for a manual check if header is
present.

Since PHP requires at least C89 the `HAVE_STDDEF_H` symbol isn't defined
by Autoconf anywhere else anymore [2] and accross the PHP source code
the header is included unconditionally already.

This patch syncs this also for the bundled libmbfl which is maintaned as
a fork in php-src.

Refs:
[1] https://port70.net/~nsz/c/c89/c89-draft.html#4.1.2
[2] https://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/autoconf/headers.m4

show more ...

Fix overflow checks in mbfl_memory_device

Also prune out some duplicate code and use strlen() and memcpy()
instead of ad-hoc reimplementations. Remove multiplications by
sizeof(unsig

Fix overflow checks in mbfl_memory_device

Also prune out some duplicate code and use strlen() and memcpy()
instead of ad-hoc reimplementations. Remove multiplications by
sizeof(unsigned char), which wrongly imply that this can be
anything but 1.

show more ...

Only write single terminating byte

As far as I could determine this is sufficient. It avoids
reallocating the buffer, if it was perfectly allocated beforehand.

Switch libmbfl to use size_t

Switch mbfl_string and related structures to use size_t lengths.

Quite likely that I broke some things along the way...

fix C89 compat

Add more mbfl string size checks (bug #73505)

Fixed multibyte related issues

@mbfilter is changed to bundled version of libmbfl to prevent LGPL licence problem. mbregex is changed to oniguruma.