Combine CJK encoding conversion code in a single source file

This will make it easier to combine duplicated code between all the
CJK text encodings (a significant amount is already combi

Combine CJK encoding conversion code in a single source file

This will make it easier to combine duplicated code between all the
CJK text encodings (a significant amount is already combined in this
commit, such as the repeated definitions of SJIS_DECODE and
SJIS_ENCODE), but I hope to remove even more redundancy in the future.

The table used to implement mb_strlen for CP932 has been changed to
the same table as "SJIS-win".

show more ...

Move mobile variants of SJIS into mbfilter_sjis.c

Move MacJapanese implementation into mbfilter_sjis.c

New implementation of mb_convert_kana

mb_convert_kana now uses the new text encoding conversion
filters. Microbenchmarking shows speed gains of 50%-150%
across various text encodings

New implementation of mb_convert_kana

mb_convert_kana now uses the new text encoding conversion
filters. Microbenchmarking shows speed gains of 50%-150%
across various text encodings and input string lengths.

The behavior is the same as the old mb_convert_kana
except for one fix: if the 'zero codepoint' U+0000 appeared
in the input, the old implementation would sometimes drop
it, not passing it through to the output. This is now
fixed.

show more ...

Implement fast text conversion interface for ISO-2022-JP-2004

There were bugs in the legacy implementation. Lots of them.

It did not properly track whether it has switched to JISX 0

Implement fast text conversion interface for ISO-2022-JP-2004

There were bugs in the legacy implementation. Lots of them.

It did not properly track whether it has switched to JISX 0213 plane 1
or plane 2. If it processes a character in plane 1 and then immediately
one in plane 2, it failed to emit the escape code to switch to plane 2.

Further, when converting codepoints from 0x80-0xFF to ISO-2022-JP-2004,
the legacy implementation would totally disregard which mode it was
operating in. Such codepoints would pass through directly to the output
without any escape sequences being emitted.

If that was not enough, all the legacy implementations of JISX 0213:2004
encodings had another common bug; their 'flush function' did not call
the next flush function in the chain of conversion filters. So if any
of these encodings were converted to an encoding where the flush
function was needed to finish the output string, then the output
would be truncated.

show more ...

Implement fast text conversion interface for EUC-JP-2004

All the legacy implementations of JISX 0213:2004 encodings had a
common bug; their 'flush function' did not call the next flush f

Implement fast text conversion interface for EUC-JP-2004

All the legacy implementations of JISX 0213:2004 encodings had a
common bug; their 'flush function' did not call the next flush function
in the chain of conversion filters. So if any of these encodings were
converted to an encoding where the flush function was needed to finish
the output string, then the output would be truncated.

show more ...

Remove duplicate implementation of CP932 from mbstring

Sigh. Double sigh. After fruitlessly searching the Internet for information on
this mysterious text encoding called "SJIS-open", I

Remove duplicate implementation of CP932 from mbstring

Sigh. Double sigh. After fruitlessly searching the Internet for information on
this mysterious text encoding called "SJIS-open", I wrote a script to try
converting every Unicode codepoint from 0-0xFFFF and compare the results from
different variants of Shift-JIS, to see which one "SJIS-open" would be most
similar to.

The result? It's just CP932. There is no difference at all. So why do we have
two implementations of CP932 in mbstring?

In case somebody, somewhere is using "SJIS-open" (or its aliases "SJIS-win" or
"SJIS-ms"), add these as aliases to CP932 so existing code will continue to
work.

show more ...

Consolidate all single-byte encodings in one source file

We can squeeze out a lot of duplicated code in this way.

Remove mbstring identify filters

mbstring had an 'identify filter' for almost every supported text encoding
which was used when auto-detecting the most likely encoding for a string.

Remove mbstring identify filters

mbstring had an 'identify filter' for almost every supported text encoding
which was used when auto-detecting the most likely encoding for a string.
It would run over the string and set a 'flag' if it saw anything which
did not appear likely to be the encoding in question.

One problem with this scheme was that encodings which merely appeared
less likely to be the correct one were completely rejected, even if there
was no better candidate. Another problem was that the 'identify filters'
had a huge amount of code duplication with the 'conversion filters'.

Eliminate the identify filters. Instead, when auto-detecting text
encoding, use conversion filters to see whether the input string is valid
in candidate encodings or not. At the same type, watch the type of
codepoints which the string decodes to and mark it as less likely if
non-printable characters (ESC, form feed, bell, etc.) or 'private use
area' codepoints are seen.

Interestingly, one old test case in which JIS text was misidentified
as UTF-8 (and this wrong behavior was enshrined in the test) was 'fixed'
and the JIS string is now auto-detected as JIS.

show more ...

Remove useless byte{2,4}{be,le} encodings from mbstring

There is no meaningful difference between these and UCS-{2,4}. They are
just a little bit more lax about passing errors silently.

Remove useless byte{2,4}{be,le} encodings from mbstring

There is no meaningful difference between these and UCS-{2,4}. They are
just a little bit more lax about passing errors silently. They also have
no known use.

Alias to UCS-{2,4} in case someone, somewhere is using them.

show more ...

Remove redundant includes from mbstring (and make sure correct config.h is used)

Very interesting... it turns out that when Valgrind support was enabled,
`#include "config.h"` from withi

Remove redundant includes from mbstring (and make sure correct config.h is used)

Very interesting... it turns out that when Valgrind support was enabled,
`#include "config.h"` from within mbstring was actually including the file "config.h"
from Valgrind, and not the one from mbstring!!

This is because -I/usr/include/valgrind was added to the compiler invocation _before_
-Iext/mbstring/libmbfl.

Make sure we actually include the file which was intended.

show more ...

Inline unneeded indirection for mbstring memory management

All memory allocation and deallocation for mbstring bounces through a table of
function pointers before going to emalloc/efree/

Inline unneeded indirection for mbstring memory management

All memory allocation and deallocation for mbstring bounces through a table of
function pointers before going to emalloc/efree/etc. But this is unnecessary.
The allocators are never swapped out. Better to just call them directly.

show more ...

Normalize comments in *nix build system m4 files

Normalization include:
- Use dnl for everything that can be ommitted when configure is built in
favor of the shell comment characte

Normalize comments in *nix build system m4 files

Normalization include:
- Use dnl for everything that can be ommitted when configure is built in
favor of the shell comment character # which is visible in the output.
- Line length normalized to 80 columns
- Dots for most of the one line sentences
- Macro definitions include similar pattern header comments now

show more ...

Simplify ext/mbstring/libmbfl/config.h creation

- The config.h.in is part of the standalone libmbfl library and it is
forked and bundled.

Fixed ZTS cache usage

Add AS_HELP_STRING to *nix build configure options

The Autoconf's default AS_HELP_STRING macro can properly format help
strings [1] so watching out if columns are aligned manually is not

Add AS_HELP_STRING to *nix build configure options

The Autoconf's default AS_HELP_STRING macro can properly format help
strings [1] so watching out if columns are aligned manually is not
anymore.

[1] https://www.gnu.org/software/autoconf/manual/autoconf.html#Pretty-Help-Strings

show more ...

Unbundle oniguruma

And also switch detection over to pkg-config.

Remove local variables

This patch removes the so called local variables defined per
file basis for certain editors to properly show tab width, and
similar settings. These are mainly

Remove local variables

This patch removes the so called local variables defined per
file basis for certain editors to properly show tab width, and
similar settings. These are mainly used by Vim and Emacs editors
yet with recent changes the once working definitions don't work
anymore in Vim without custom plugins or additional configuration.
Neither are these settings synced across the PHP code base.

A simpler and better approach is EditorConfig and fixing code
using some code style fixing tools in the future instead.

This patch also removes the so called modelines for Vim. Modelines
allow Vim editor specifically to set some editor configuration such as
syntax highlighting, indentation style and tab width to be set in the
first line or the last 5 lines per file basis. Since the php test
files have syntax highlighting already set in most editors properly and
EditorConfig takes care of the indentation settings, this patch removes
these as well for the Vim 6.0 and newer versions.

With the removal of local variables for certain editors such as
Emacs and Vim, the footer is also probably not needed anymore when
creating extensions using ext_skel.php script.

Additionally, Vim modelines for setting php syntax and some editor
settings has been removed from some *.phpt files. All these are
mostly not relevant for phpt files neither work properly in the
middle of the file.

show more ...

Trim trailing whitespace in source code files

Trim trailing whitespace in source code files

Remove HAVE_STDARG_H

The C89 standard and later defines the `<stdarg.h>` header as part of
the standard headers [1]. On current systems it is always present and
can be included uncon

Remove HAVE_STDARG_H

The C89 standard and later defines the `<stdarg.h>` header as part of
the standard headers [1]. On current systems it is always present and
can be included unconditionally.

Checking for presence and functionality of the `<stdarg.h>` header and
variadic function is not relevant anymore on current systems since this
is always available.

Also Autoconf suggests relying on at least C89 or above [2] and [3].

The following files were regenerated with re2c 1.0.3:
- Zend/zend_language_scanner.c
- Zend/zend_language_scanner_defs.h

Refs:
[1] https://port70.net/~nsz/c/c89/c89-draft.html#4.1.2
[2] http://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/autoconf/headers.m4
[3] https://www.gnu.org/software/autoconf/manual/autoconf-2.69/autoconf.html

show more ...

Remove HAVE_STRING_H

The C89 standard and later defines the `<string.h>` header as part of
the standard headers [1] and on current systems it is always present.

Code included al

Remove HAVE_STRING_H

The C89 standard and later defines the `<string.h>` header as part of
the standard headers [1] and on current systems it is always present.

Code included also `<strings.h>` header as an alterinative in some
files. This kind of check was relevant on some older systems where the
`<strings.h>` file included definitions for the C89 compliant
`<string.h>`. Today such alternative check is not required anymore. The
`<strings.h>` file is part of the POSIX definition these days.

Also Autoconf suggests doing this and relying on C89 or above [2] and [3].

This patch also cleans few unused `<strings.h>` inclusions in the libmbfl.

[1]: https://port70.net/~nsz/c/c89/c89-draft.html#4.1.2
[2]: http://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/autoconf/headers.m4
[3]: https://www.gnu.org/software/autoconf/manual/autoconf-2.69/autoconf.html

show more ...

Remove HAVE_STDLIB_H

The C89 and later standard defines the `<stdlib.h>` header as part of
the standard headers [1] and on current systems it is always present
and the `HAVE_STDLIB_H

Remove HAVE_STDLIB_H

The C89 and later standard defines the `<stdlib.h>` header as part of
the standard headers [1] and on current systems it is always present
and the `HAVE_STDLIB_H` symbol can be removed.

Also Autoconf suggests doing this and relying on C89 or above [2] and [3].

[1] https://port70.net/~nsz/c/c89/c89-draft.html#4.1.2
[2] http://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/autoconf/headers.m4
[3] https://www.gnu.org/software/autoconf/manual/autoconf-2.69/autoconf.html

show more ...

Remove AC_FUNC_MEMCMP

Autoconf 2.59d (released in 2006) [1] started promoting several macros
as not relevant for newer systems anymore, including the `AC_FUNC_MEMCMP`.

On some o

Remove AC_FUNC_MEMCMP

Autoconf 2.59d (released in 2006) [1] started promoting several macros
as not relevant for newer systems anymore, including the `AC_FUNC_MEMCMP`.

On some old systems such as SunOS 4.1.3 (EOL in 2003) and NeXT x86
OpenStep (discontinued) the `memcmp` function wasn't present or it
didn't work properly. [2]

On current systems including at least Solaris 10+ this check is not
relevant anymore.

Refs:
[1] http://git.savannah.gnu.org/cgit/autoconf.git/tree/NEWS
[2] https://www.gnu.org/software/autoconf/manual/autoconf-2.69/autoconf.html

show more ...

Remove AC_HEADER_TIME

Autoconf 2.59d (released in 2006) [1] started promoting several macros
as not relevant for newer systems anymore, including the `AC_HEADER_TIME`.

This macr

Remove AC_HEADER_TIME

Autoconf 2.59d (released in 2006) [1] started promoting several macros
as not relevant for newer systems anymore, including the `AC_HEADER_TIME`.

This macro checks if both `<sys/time.h>` and `<time.h>` can be included
at the same time and defines the `TIME_WITH_SYS_TIME` and
`HAVE_SYS_TIME_H` symbols. On current system such check is not relevant
anymore because in case both headers are present both can be also
included at the same time.

This patch simplifies this checking.

Refs:
[1] http://git.savannah.gnu.org/cgit/autoconf.git/tree/NEWS
[2] https://www.gnu.org/software/autoconf/manual/autoconf-2.69/autoconf.html

show more ...