Fix GH-15824 mb_detect_encoding() invalid "UTF8" (#15829)

I fixed from strcasecmp to strncasecmp.
However, strncasecmp is specify size to #3 parameter.
Hence, Add check length to mim

Fix GH-15824 mb_detect_encoding() invalid "UTF8" (#15829)

I fixed from strcasecmp to strncasecmp.
However, strncasecmp is specify size to #3 parameter.
Hence, Add check length to mime and aliases.

Co-authored-by: Niels Dossche <7771979+nielsdos@users.noreply.github.com>

show more ...

Use more optimal perfect hash table

Add mbstring support for GB18030-2022 text encoding

The previous version of the GB-18030 standard was published in 2005.
This commit adds support for the updated (2022) version of this t

Add mbstring support for GB18030-2022 text encoding

The previous version of the GB-18030 standard was published in 2005.
This commit adds support for the updated (2022) version of this text
encoding. The existing GB18030 implementation has been left unchanged
for backwards compatibility; users who want to use the new standard
must explicitly indicate the desired text encoding is 'GB18030-2022'.

The document which defines GB18030-2022, published by the government
of the People's Republic of China, defines three levels of standards
compliance. This implementation is intended to achieve Implementation
Level 3, which is the highest level of compliance.

Experts in the GB18030 standard are requested to assess this
implementation and report any deviation from the standard.

show more ...

Use optimized implementation of mb_strcut for Japanese mobile vendor UTF-8 variants

To facilitate sharing of mb_cut_utf8, I combined mbfilter_utf8.c and
mbfilter_utf8_mobile.c into a sin

Use optimized implementation of mb_strcut for Japanese mobile vendor UTF-8 variants

To facilitate sharing of mb_cut_utf8, I combined mbfilter_utf8.c and
mbfilter_utf8_mobile.c into a single source file.

show more ...

Avoid temporary string allocations in php_mb_parse_encoding_list() (#12714)

This brings execution time down from 0.91s to 0.86s on the reference
benchmark [1].

[1] https://githu

Avoid temporary string allocations in php_mb_parse_encoding_list() (#12714)

This brings execution time down from 0.91s to 0.86s on the reference
benchmark [1].

[1] https://github.com/php/php-src/issues/12684#issuecomment-1813799924

show more ...

Improve performance of mbfl_name2encoding() by using perfect hashing (#12707)

mbfl_name2encoding() uses a linear loop through the encodings, comparing
the name one by one, which is very

Improve performance of mbfl_name2encoding() by using perfect hashing (#12707)

mbfl_name2encoding() uses a linear loop through the encodings, comparing
the name one by one, which is very slow. For the benchmark [1] just
looking up the name takes about 50% of run-time.

By using perfect hashing instead, we no longer have to loop over the
list, and the number of string comparisons is reduced to just a single
one. The perfect hashing table is generated using GNU gperf and amended
manually to fit in with mbstring and manually changed to reduce the
cache size.

[1] https://github.com/php/php-src/issues/12684#issuecomment-1813799924

show more ...

Combine CJK encoding conversion code in a single source file

This will make it easier to combine duplicated code between all the
CJK text encodings (a significant amount is already combi

Combine CJK encoding conversion code in a single source file

This will make it easier to combine duplicated code between all the
CJK text encodings (a significant amount is already combined in this
commit, such as the repeated definitions of SJIS_DECODE and
SJIS_ENCODE), but I hope to remove even more redundancy in the future.

The table used to implement mb_strlen for CP932 has been changed to
the same table as "SJIS-win".

show more ...

Remove unneeded function mbfl_no2preferred_mime_name

Remove unneeded function mbfl_name2no_encoding

Reintroduce legacy 'SJIS-win' text encoding in mbstring

In e2459857af, I combined mbstring's "SJIS-win" text encoding
into CP932. This was done after doing some testing which appeared

Reintroduce legacy 'SJIS-win' text encoding in mbstring

In e2459857af, I combined mbstring's "SJIS-win" text encoding
into CP932. This was done after doing some testing which appeared
to show that the mappings for "SJIS-win" were the same as those
for "CP932".

Later, it was found that there was actually a small difference
prior to e2459857af when converting Unicode to CP932. The
mappings for the following two codepoints were different:

CP932 SJIS-win
U+203E 0x7E 0x81 0x50
U+00A5 0x5C 0x81 0x8F

As shown, mbstring's "CP932" mapped Unicode's 'OVERLINE' and
'YEN SIGN' to the ASCII bytes which have conflicting uses in
most legacy Japanese text encodings. "SJIS-win" mapped these
to equivalent JIS X 0208 fullwidth characters.

Since e2459867af was not intended to cause any user-visible
change in behavior, I am rolling back the merge of "CP932"
and "SJIS-win".

It seems doubtful whether these two text encodings should
be kept separate or merged in a future release. An extensive
discussion of the related historical background and
compatibility issues involved can be found in this
GitHub thread:

https://github.com/php/php-src/issues/8308

show more ...

Don't expose wchar encoding to users (#7415)

The "wchar" encoding isn't really an encoding -- it's what we
internally use as the representation of decoded characters.

In practic

Don't expose wchar encoding to users (#7415)

The "wchar" encoding isn't really an encoding -- it's what we
internally use as the representation of decoded characters.

In practice, it tends to behave a lot like the 8bit encoding when
used from userland, because input code units end up being treated
as code points.

This patch removes the wchar encoding from the public encoding
list and reserves it for internal use only.

show more ...

Remove duplicate implementation of CP932 from mbstring

Sigh. Double sigh. After fruitlessly searching the Internet for information on
this mysterious text encoding called "SJIS-open", I

Remove duplicate implementation of CP932 from mbstring

Sigh. Double sigh. After fruitlessly searching the Internet for information on
this mysterious text encoding called "SJIS-open", I wrote a script to try
converting every Unicode codepoint from 0-0xFFFF and compare the results from
different variants of Shift-JIS, to see which one "SJIS-open" would be most
similar to.

The result? It's just CP932. There is no difference at all. So why do we have
two implementations of CP932 in mbstring?

In case somebody, somewhere is using "SJIS-open" (or its aliases "SJIS-win" or
"SJIS-ms"), add these as aliases to CP932 so existing code will continue to
work.

show more ...

Remove useless mbstring encoding 'JIS-ms'

MicroSoft invented three encodings very similar to ISO-2022-JP/JIS7/JIS8, called
CP50220, CP50221, and CP50222. All three are supported by mbstr

Remove useless mbstring encoding 'JIS-ms'

MicroSoft invented three encodings very similar to ISO-2022-JP/JIS7/JIS8, called
CP50220, CP50221, and CP50222. All three are supported by mbstring.

Since these encodings are very similar, some code can be shared. Actually,
conversion of CP50220/1/2 to Unicode is exactly the same operation; it's when
converting from Unicode to CP50220/1/2 that some small differences arise in how
certain katakana are handled.

The most important common code was a function called `mbfl_filt_wchar_jis_ms`.
The `jis_ms` part doubtless refers to the fact that these encodings are modified
versions of 'JIS' invented by 'MS'. mbstring also went a step further and exported
'JIS-ms' to userland as a separate encoding from CP50220/1/2. If users requested
'JIS-ms' conversion, they got something like CP50220/1/2, minus their special
ways of handling half-width katakana when converting from Unicode.

But... that 'encoding' is not something which actually exists in the world outside
of mbstring. CP50220/1/2 do exist in MicroSoft software, but not 'JIS-ms'.

For a text encoding conversion library, inventing new variant encodings and
implementing them is not very productive. Our interest is in handling text
encodings which real people actually use for... you know, storing actual text
and things like that.

show more ...

Remove useless mbstring encoding 'CP50220-raw'

CP50220 is a variant of ISO-2022-JP invented by MicroSoft, which handles some
Unicode characters which are not representable in ISO-2022-JP

Remove useless mbstring encoding 'CP50220-raw'

CP50220 is a variant of ISO-2022-JP invented by MicroSoft, which handles some
Unicode characters which are not representable in ISO-2022-JP by converting
them to similar characters which are representable.

What, then, is CP50220-raw? An Internet search turns up absolutely nothing.
Reference works which I consulted don't say anything about it. Other text
conversion libraries don't support it.

From looking at the code: It's just the same as CP50220, but it accepts
unmapped JIS X 0208 characters passed through from other Japanese encodings
and silently encodes them using the usual ISO-2022-JP escape sequence and
representation for JIS X 0208 characters.

It's hard to see how this could be useful. OK, let me come out and say it:
it's _not_ useful. We can confidently jettison this (mis)feature.

show more ...

Consolidate all single-byte encodings in one source file

We can squeeze out a lot of duplicated code in this way.

Remove useless byte{2,4}{be,le} encodings from mbstring

There is no meaningful difference between these and UCS-{2,4}. They are
just a little bit more lax about passing errors silently.

Remove useless byte{2,4}{be,le} encodings from mbstring

There is no meaningful difference between these and UCS-{2,4}. They are
just a little bit more lax about passing errors silently. They also have
no known use.

Alias to UCS-{2,4} in case someone, somewhere is using them.

show more ...

Refactor mbfl_ident.c, mbfl_encoding.c, mbfl_memory_device.c, mbfl_string.c

- Make everything less gratuitously verbose
- Don't litter the code with lots of unneeded NULL checks (for thi

Refactor mbfl_ident.c, mbfl_encoding.c, mbfl_memory_device.c, mbfl_string.c

- Make everything less gratuitously verbose
- Don't litter the code with lots of unneeded NULL checks (for things which
will never be NULL)
- Don't return success/failure code from functions which can never fail
- For encoding structs, don't use pointers to pointers to pointers for the
list of alias strings. Pointers to pointers (2 levels of indirection)
is what actually makes sense. This gets rid of some extraneous
dereference operations.

show more ...

Remove redundant includes from mbstring (and make sure correct config.h is used)

Very interesting... it turns out that when Valgrind support was enabled,
`#include "config.h"` from withi

Remove redundant includes from mbstring (and make sure correct config.h is used)

Very interesting... it turns out that when Valgrind support was enabled,
`#include "config.h"` from within mbstring was actually including the file "config.h"
from Valgrind, and not the one from mbstring!!

This is because -I/usr/include/valgrind was added to the compiler invocation _before_
-Iext/mbstring/libmbfl.

Make sure we actually include the file which was intended.

show more ...

Remove unused functions from mbstring

- mbfl_buffer_converter_reset
- mbfl_buffer_converter_strncat
- mbfl_buffer_converter_getbuffer
- mbfl_oddlen
- mbfl_filter_output_pipe_

Remove unused functions from mbstring

- mbfl_buffer_converter_reset
- mbfl_buffer_converter_strncat
- mbfl_buffer_converter_getbuffer
- mbfl_oddlen
- mbfl_filter_output_pipe_flush
- mbfl_memory_device_output2
- mbfl_memory_device_output4
- mbfl_is_support_encoding
- mbfl_buffer_converter_feed2
- _php_mb_regex_globals_dtor
- mime_header_encoder_feed
- mime_header_decoder_feed
- mbfl_convert_filter_feed

show more ...

Only allow "pass" as input/output encoding

"pass" is not a real encoding, it just means "don't perform any
conversion". Using it as an internal encoding or passing it to
any of the m

Only allow "pass" as input/output encoding

"pass" is not a real encoding, it just means "don't perform any
conversion". Using it as an internal encoding or passing it to
any of the mbstring() function will not work (and on master commonly
assert).

show more ...

Merge branch 'PHP-7.3'

Remove the "auto" encoding

"auto" is only meaningful in functions which accept an encoding
*list* and support encoding detection. These functions have
explicit checks for "auto". It

Remove the "auto" encoding

"auto" is only meaningful in functions which accept an encoding
*list* and support encoding detection. These functions have
explicit checks for "auto". It cannot be used as a standalone
encoding in any meaningful capacity, so I'm dropping it entirely.

show more ...

Remove HAVE_STRING_H

The C89 standard and later defines the `<string.h>` header as part of
the standard headers [1] and on current systems it is always present.

Code included al

Remove HAVE_STRING_H

The C89 standard and later defines the `<string.h>` header as part of
the standard headers [1] and on current systems it is always present.

Code included also `<strings.h>` header as an alterinative in some
files. This kind of check was relevant on some older systems where the
`<strings.h>` file included definitions for the C89 compliant
`<string.h>`. Today such alternative check is not required anymore. The
`<strings.h>` file is part of the POSIX definition these days.

Also Autoconf suggests doing this and relying on C89 or above [2] and [3].

This patch also cleans few unused `<strings.h>` inclusions in the libmbfl.

[1]: https://port70.net/~nsz/c/c89/c89-draft.html#4.1.2
[2]: http://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/autoconf/headers.m4
[3]: https://www.gnu.org/software/autoconf/manual/autoconf-2.69/autoconf.html

show more ...

Remove HAVE_STDDEF_H

The `<stddef.h>` header file is part of the standard C89 headers [1] and
on current systems there is no need for a manual check if header is
present.

Si

Remove HAVE_STDDEF_H

The `<stddef.h>` header file is part of the standard C89 headers [1] and
on current systems there is no need for a manual check if header is
present.

Since PHP requires at least C89 the `HAVE_STDDEF_H` symbol isn't defined
by Autoconf anywhere else anymore [2] and accross the PHP source code
the header is included unconditionally already.

This patch syncs this also for the bundled libmbfl which is maintaned as
a fork in php-src.

Refs:
[1] https://port70.net/~nsz/c/c89/c89-draft.html#4.1.2
[2] https://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/autoconf/headers.m4

show more ...

Store input and output filters in mbfl encodings

For functions like mb_chr() and mb_ord() just looking up the
input/output filter for the encoding dominates the runtime. This
commit

Store input and output filters in mbfl encodings

For functions like mb_chr() and mb_ord() just looking up the
input/output filter for the encoding dominates the runtime. This
commit stores the input/output filter for an encoding in the
mbfl encoding structure, so it can be looked up directly, rather
than scanning through filter function lists.

show more ...