Fix GH-13968: Build failed when with --enable-mbregex on MSVC (#13971)

I added /utf-8 flag in CFLAGS_MBSTRING at config.w32

Advised from Hirokawa-san thanks

Fix GH-13789: build failed mbstring_arginfo.h when Visual C++ on Windows (#13906)

Probably CP932 environment can't compile. So add /utf-8 flag.

Use optimized implementation of mb_strcut for Japanese mobile vendor UTF-8 variants

To facilitate sharing of mb_cut_utf8, I combined mbfilter_utf8.c and
mbfilter_utf8_mobile.c into a sin

Use optimized implementation of mb_strcut for Japanese mobile vendor UTF-8 variants

To facilitate sharing of mb_cut_utf8, I combined mbfilter_utf8.c and
mbfilter_utf8_mobile.c into a single source file.

show more ...

Combine CJK encoding conversion code in a single source file

This will make it easier to combine duplicated code between all the
CJK text encodings (a significant amount is already combi

Combine CJK encoding conversion code in a single source file

This will make it easier to combine duplicated code between all the
CJK text encodings (a significant amount is already combined in this
commit, such as the repeated definitions of SJIS_DECODE and
SJIS_ENCODE), but I hope to remove even more redundancy in the future.

The table used to implement mb_strlen for CP932 has been changed to
the same table as "SJIS-win".

show more ...

Move mobile variants of SJIS into mbfilter_sjis.c

Move MacJapanese implementation into mbfilter_sjis.c

New implementation of mb_convert_kana

mb_convert_kana now uses the new text encoding conversion
filters. Microbenchmarking shows speed gains of 50%-150%
across various text encodings

New implementation of mb_convert_kana

mb_convert_kana now uses the new text encoding conversion
filters. Microbenchmarking shows speed gains of 50%-150%
across various text encodings and input string lengths.

The behavior is the same as the old mb_convert_kana
except for one fix: if the 'zero codepoint' U+0000 appeared
in the input, the old implementation would sometimes drop
it, not passing it through to the output. This is now
fixed.

show more ...

Implement fast text conversion interface for ISO-2022-JP-2004

There were bugs in the legacy implementation. Lots of them.

It did not properly track whether it has switched to JISX 0

Implement fast text conversion interface for ISO-2022-JP-2004

There were bugs in the legacy implementation. Lots of them.

It did not properly track whether it has switched to JISX 0213 plane 1
or plane 2. If it processes a character in plane 1 and then immediately
one in plane 2, it failed to emit the escape code to switch to plane 2.

Further, when converting codepoints from 0x80-0xFF to ISO-2022-JP-2004,
the legacy implementation would totally disregard which mode it was
operating in. Such codepoints would pass through directly to the output
without any escape sequences being emitted.

If that was not enough, all the legacy implementations of JISX 0213:2004
encodings had another common bug; their 'flush function' did not call
the next flush function in the chain of conversion filters. So if any
of these encodings were converted to an encoding where the flush
function was needed to finish the output string, then the output
would be truncated.

show more ...

Implement fast text conversion interface for EUC-JP-2004

All the legacy implementations of JISX 0213:2004 encodings had a
common bug; their 'flush function' did not call the next flush f

Implement fast text conversion interface for EUC-JP-2004

All the legacy implementations of JISX 0213:2004 encodings had a
common bug; their 'flush function' did not call the next flush function
in the chain of conversion filters. So if any of these encodings were
converted to an encoding where the flush function was needed to finish
the output string, then the output would be truncated.

show more ...

Remove duplicate implementation of CP932 from mbstring

Sigh. Double sigh. After fruitlessly searching the Internet for information on
this mysterious text encoding called "SJIS-open", I

Remove duplicate implementation of CP932 from mbstring

Sigh. Double sigh. After fruitlessly searching the Internet for information on
this mysterious text encoding called "SJIS-open", I wrote a script to try
converting every Unicode codepoint from 0-0xFFFF and compare the results from
different variants of Shift-JIS, to see which one "SJIS-open" would be most
similar to.

The result? It's just CP932. There is no difference at all. So why do we have
two implementations of CP932 in mbstring?

In case somebody, somewhere is using "SJIS-open" (or its aliases "SJIS-win" or
"SJIS-ms"), add these as aliases to CP932 so existing code will continue to
work.

show more ...

Consolidate all single-byte encodings in one source file

We can squeeze out a lot of duplicated code in this way.

Remove mbstring identify filters

mbstring had an 'identify filter' for almost every supported text encoding
which was used when auto-detecting the most likely encoding for a string.

Remove mbstring identify filters

mbstring had an 'identify filter' for almost every supported text encoding
which was used when auto-detecting the most likely encoding for a string.
It would run over the string and set a 'flag' if it saw anything which
did not appear likely to be the encoding in question.

One problem with this scheme was that encodings which merely appeared
less likely to be the correct one were completely rejected, even if there
was no better candidate. Another problem was that the 'identify filters'
had a huge amount of code duplication with the 'conversion filters'.

Eliminate the identify filters. Instead, when auto-detecting text
encoding, use conversion filters to see whether the input string is valid
in candidate encodings or not. At the same type, watch the type of
codepoints which the string decodes to and mark it as less likely if
non-printable characters (ESC, form feed, bell, etc.) or 'private use
area' codepoints are seen.

Interestingly, one old test case in which JIS text was misidentified
as UTF-8 (and this wrong behavior was enshrined in the test) was 'fixed'
and the JIS string is now auto-detected as JIS.

show more ...

Remove useless byte{2,4}{be,le} encodings from mbstring

There is no meaningful difference between these and UCS-{2,4}. They are
just a little bit more lax about passing errors silently.

Remove useless byte{2,4}{be,le} encodings from mbstring

There is no meaningful difference between these and UCS-{2,4}. They are
just a little bit more lax about passing errors silently. They also have
no known use.

Alias to UCS-{2,4} in case someone, somewhere is using them.

show more ...

Remove redundant includes from mbstring (and make sure correct config.h is used)

Very interesting... it turns out that when Valgrind support was enabled,
`#include "config.h"` from withi

Remove redundant includes from mbstring (and make sure correct config.h is used)

Very interesting... it turns out that when Valgrind support was enabled,
`#include "config.h"` from within mbstring was actually including the file "config.h"
from Valgrind, and not the one from mbstring!!

This is because -I/usr/include/valgrind was added to the compiler invocation _before_
-Iext/mbstring/libmbfl.

Make sure we actually include the file which was intended.

show more ...

Inline unneeded indirection for mbstring memory management

All memory allocation and deallocation for mbstring bounces through a table of
function pointers before going to emalloc/efree/

Inline unneeded indirection for mbstring memory management

All memory allocation and deallocation for mbstring bounces through a table of
function pointers before going to emalloc/efree/etc. But this is unnecessary.
The allocators are never swapped out. Better to just call them directly.

show more ...

Put oniguruma include path to proper CFLAGS

Move Oniguruma related config stuff to where it belongs

Oniguruma is exclusively used by ext/mbstring, and only if mbregex is
enabled. Therefore it is unnecessary and confusing to have

Move Oniguruma related config stuff to where it belongs

Oniguruma is exclusively used by ext/mbstring, and only if mbregex is
enabled. Therefore it is unnecessary and confusing to have Oniguruma
related config stuff scattered elsewhere.

While we're at it, we also remove the referral to the bundled libonig
which is removed as of PHP 7.4.0, and the duplicated call to
`PHP_INSTALL_HEADERS()`.

show more ...

Remove unused defines

Used in php-src the past and today removed and not used anymore:
- HAVE_CURL_EASY_STRERROR
- HAVE_CURL_MULTI_STRERROR
- HAVE_NEW_MIME2TEXT
- HAVE_MBSTR_

Remove unused defines

Used in php-src the past and today removed and not used anymore:
- HAVE_CURL_EASY_STRERROR
- HAVE_CURL_MULTI_STRERROR
- HAVE_NEW_MIME2TEXT
- HAVE_MBSTR_CN
- HAVE_MBSTR_JA
- HAVE_MBSTR_KR
- HAVE_MBSTR_RU
- HAVE_MBSTR_TW

Part of oniguruma which doesn't use these anymore
- NOT_RUBY
- HAVE_STDARG_PROTOTYPES

Unused:
- HAVE_MPIR

Closes GH-4427

show more ...

Sync with ZEND_ENABLE_STATIC_TSRMLS_CACHE enablement in ext/mbstring

Unbundle oniguruma in config.w32

Unbundle oniguruma

And also switch detection over to pkg-config.

Remove HAVE_STDLIB_H

The C89 and later standard defines the `<stdlib.h>` header as part of
the standard headers [1] and on current systems it is always present
and the `HAVE_STDLIB_H

Remove HAVE_STDLIB_H

The C89 and later standard defines the `<stdlib.h>` header as part of
the standard headers [1] and on current systems it is always present
and the `HAVE_STDLIB_H` symbol can be removed.

Also Autoconf suggests doing this and relying on C89 or above [2] and [3].

[1] https://port70.net/~nsz/c/c89/c89-draft.html#4.1.2
[2] http://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/autoconf/headers.m4
[3] https://www.gnu.org/software/autoconf/manual/autoconf-2.69/autoconf.html

show more ...

Remove unused Git attributes ident

The $Id$ keywords were used in Subversion where they can be substituted
with filename, last revision number change, last changed date, and last
use

Remove unused Git attributes ident

The $Id$ keywords were used in Subversion where they can be substituted
with filename, last revision number change, last changed date, and last
user who changed it.

In Git this functionality is different and can be done with Git attribute
ident. These need to be defined manually for each file in the
.gitattributes file and are afterwards replaced with 40-character
hexadecimal blob object name which is based only on the particular file
contents.

This patch simplifies handling of $Id$ keywords by removing them since
they are not used anymore.

show more ...

Update to Oniguruma 6.7.1

We also apply the still relevant parts of `oniguruma.patch` and update
the patch accordingly.

Remove --with-libmbfl configure option

The bundled libmbfl library is no longer API or ABI compatible with
the (currently unmaintained) upstream library. As such, building
against an

Remove --with-libmbfl configure option

The bundled libmbfl library is no longer API or ABI compatible with
the (currently unmaintained) upstream library. As such, building
against an external libmbfl is no longer possible.

show more ...