History log of /PHP-8.4/ext/tokenizer/tokenizer.c (Results 1 – 25 of 145)
Revision (<<< Hide revision tags) (Show revision tags >>>) Date Author Comments
# 11accb5c 25-Jun-2024 Arnaud Le Blanc

Preferably include from build dir (#13516)

* Include from build dir first

This fixes out of tree builds by ensuring that configure artifacts are included
from the build dir.

Preferably include from build dir (#13516)

* Include from build dir first

This fixes out of tree builds by ensuring that configure artifacts are included
from the build dir.

Before, out of tree builds would preferably include files from the src dir, as
the include path was defined as follows (ignoring includes from ext/ and sapi/) :

-I$(top_builddir)/main
-I$(top_srcdir)
-I$(top_builddir)/TSRM
-I$(top_builddir)/Zend
-I$(top_srcdir)/main
-I$(top_srcdir)/Zend
-I$(top_srcdir)/TSRM
-I$(top_builddir)/

As a result, an out of tree build would include configure artifacts such as
`main/php_config.h` from the src dir.

After this change, the include path is defined as follows:

-I$(top_builddir)/main
-I$(top_builddir)
-I$(top_srcdir)/main
-I$(top_srcdir)
-I$(top_builddir)/TSRM
-I$(top_builddir)/Zend
-I$(top_srcdir)/Zend
-I$(top_srcdir)/TSRM

* Fix extension include path for out of tree builds

* Include config.h with the brackets form

`#include "config.h"` searches in the directory containing the including-file
before any other include path. This can include the wrong config.h when building
out of tree and a config.h exists in the source tree.

Using `#include <config.h>` uses exclusively the include path, and gives
priority to the build dir.

show more ...


# fd2d8696 08-Jun-2024 Gina Peter Banyard

Clean-up some more headers (#14416)

Remove unused headers (such as php_ini.h for extensions that don't define INI settings)
Use more specific headers when possible


# 8e62e2b8 22-May-2024 Cristian Rodríguez

Mark multple functions as static (#13864)

* Mark many functions as static

Multiple functions are missing the static qualifier.

* remove unused struct sigactions

st

Mark multple functions as static (#13864)

* Mark many functions as static

Multiple functions are missing the static qualifier.

* remove unused struct sigactions

struct sigaction act, old_term, old_quit, old_int;
all unused.

* optimizer: minXOR and maxXOR are unused

show more ...


# 7936c808 23-Jan-2023 Máté Kocsis

Fix GH-8329 Print true/false instead of bool in error and debug messages (#8385)


Revision tags: php-8.2.0RC1, php-8.1.10, php-8.0.23, php-8.0.23RC1, php-8.1.10RC1, php-8.2.0beta3, php-8.2.0beta2, php-8.1.9, php-8.0.22
# 59d257d1 01-Aug-2022 Máté Kocsis

Declare ext/tokenizer constants in stubs (#9148)


Revision tags: php-8.1.9RC1, php-8.2.0beta1, php-8.0.22RC1, php-8.0.21, php-8.1.8, php-8.2.0alpha3, php-8.1.8RC1, php-8.2.0alpha2, php-8.0.21RC1, php-8.0.20, php-8.1.7, php-8.2.0alpha1, php-7.4.30, php-8.1.7RC1, php-8.0.20RC1, php-8.1.6, php-8.0.19, php-8.1.6RC1, php-8.0.19RC1, php-8.0.18, php-8.1.5, php-7.4.29, php-8.1.5RC1, php-8.0.18RC1, php-8.1.4, php-8.0.17, php-8.1.4RC1, php-8.0.17RC1, php-8.1.3, php-8.0.16, php-7.4.28, php-8.1.3RC1, php-8.0.16RC1, php-8.1.2, php-8.0.15, php-8.1.2RC1, php-8.0.15RC1, php-8.0.14, php-8.1.1, php-7.4.27, php-8.1.1RC1, php-8.0.14RC1, php-7.4.27RC1, php-8.1.0, php-8.0.13, php-7.4.26, php-7.3.33, php-8.1.0RC6, php-7.4.26RC1, php-8.0.13RC1, php-8.1.0RC5, php-7.3.32, php-7.4.25, php-8.0.12, php-8.1.0RC4, php-8.0.12RC1, php-7.4.25RC1, php-8.1.0RC3, php-8.0.11, php-7.4.24, php-7.3.31, php-8.1.0RC2, php-7.4.24RC1, php-8.0.11RC1, php-8.1.0RC1, php-7.4.23, php-8.0.10, php-7.3.30, php-8.1.0beta3
# 2e8904fe 11-Aug-2021 Dmitry Stogov

Use ZEND_HASH_FILL_* API.


Revision tags: php-8.0.10RC1, php-7.4.23RC1, php-8.1.0beta2, php-8.0.9, php-7.4.22, php-8.1.0beta1, php-7.4.22RC1, php-8.0.9RC1, php-8.1.0alpha3, php-7.4.21, php-7.3.29, php-8.0.8, php-8.1.0alpha2, php-7.4.21RC1, php-8.0.8RC1, php-8.1.0alpha1, php-8.0.7, php-7.4.20, php-8.0.7RC1, php-7.4.20RC1
# 01b3fc03 06-May-2021 KsaR

Update http->https in license (#6945)

1. Update: http://www.php.net/license/3_01.txt to https, as there is anyway server header "Location:" to https.
2. Update few license 3.0 to 3.01 as

Update http->https in license (#6945)

1. Update: http://www.php.net/license/3_01.txt to https, as there is anyway server header "Location:" to https.
2. Update few license 3.0 to 3.01 as 3.0 states "php 5.1.1, 4.1.1, and earlier".
3. In some license comments is "at through the world-wide-web" while most is without "at", so deleted.
4. fixed indentation in some files before |

show more ...


Revision tags: php-8.0.6, php-7.4.19, php-7.4.18, php-7.3.28, php-8.0.5, php-8.0.5RC1, php-7.4.18RC1, php-8.0.4RC1, php-7.4.17RC1, php-8.0.3, php-7.4.16, php-8.0.3RC1, php-7.4.16RC1, php-8.0.2, php-7.4.15, php-7.3.27
# 1954e597 26-Jan-2021 Máté Kocsis

Add support for generating class entries from stubs

Closes GH-6289

Co-authored-by: Nikita Popov <nikita.ppv@gmail.com>


# 3370b5fd 21-Jan-2021 Nikita Popov

Accept zend_string in zend_prepare_string_for_scanning


Revision tags: php-8.0.2RC1, php-7.4.15RC2, php-7.4.15RC1
# 3e01f5af 15-Jan-2021 Nikita Popov

Replace zend_bool uses with bool

We're starting to see a mix between uses of zend_bool and bool.
Replace all usages with the standard bool type everywhere.

Of course, zend_bool

Replace zend_bool uses with bool

We're starting to see a mix between uses of zend_bool and bool.
Replace all usages with the standard bool type everywhere.

Of course, zend_bool is retained as an alias.

show more ...


Revision tags: php-8.0.1, php-7.4.14, php-7.3.26, php-7.4.14RC1, php-8.0.1RC1, php-7.3.26RC1, php-8.0.0, php-7.3.25, php-7.4.13, php-8.0.0RC5, php-7.4.13RC1, php-8.0.0RC4, php-7.3.25RC1
# b1019f46 09-Nov-2020 Nikita Popov

Rename PhpToken::getAll() to PhpToken::tokenize()

See https://externals.io/message/112189.
Fixes bug #80328.


Revision tags: php-7.4.12, php-8.0.0RC3, php-7.3.24, php-8.0.0RC2, php-7.4.12RC1, php-7.3.24RC1, php-7.2.34, php-8.0.0rc1, php-7.4.11, php-7.3.23, php-8.0.0beta4, php-7.4.11RC1, php-7.3.23RC1, php-8.0.0beta3, php-7.4.10, php-7.3.22
# fa8d9b11 28-Aug-2020 George Peter Banyard

Improve type declarations for Zend APIs

Voidification of Zend API which always succeeded
Use bool argument types instead of int for boolean arguments
Use bool return type for functio

Improve type declarations for Zend APIs

Voidification of Zend API which always succeeded
Use bool argument types instead of int for boolean arguments
Use bool return type for functions which return true/false (1/0)
Use zend_result return type for functions which return SUCCESS/FAILURE as they don't follow normal boolean semantics

Closes GH-6002

show more ...


Revision tags: php-8.0.0beta2, php-7.3.22RC1, php-7.4.10RC1, php-8.0.0beta1, php-7.4.9, php-7.2.33, php-7.3.21, php-8.0.0alpha3, php-7.4.9RC1, php-7.3.21RC1, php-7.4.8, php-7.2.32, php-8.0.0alpha2, php-7.3.20
# 2b5de6f8 01-Jul-2020 Max Semenik

Remove proto comments from C files

Closes GH-5758


Revision tags: php-8.0.0alpha1, php-7.4.8RC1, php-7.3.20RC1, php-7.4.7, php-7.3.19
# 83a77015 08-Jun-2020 twosee

Add helper APIs for maybe-interned string creation

Add ZVAL_CHAR/RETVAL_CHAR/RETURN_CHAR as a shortcut for using
ZVAL_INTERNED_STRING and ZSTR_CHAR.

Add zend_string_init_fast()

Add helper APIs for maybe-interned string creation

Add ZVAL_CHAR/RETVAL_CHAR/RETURN_CHAR as a shortcut for using
ZVAL_INTERNED_STRING and ZSTR_CHAR.

Add zend_string_init_fast() as a helper for the empty string /
one char interned string / zend_string_init() pattern.

Also add corresponding ZVAL_STRINGL_FAST etc macros.

Closes GH-5684.

show more ...


# b03cafd1 05-Jun-2020 Nikita Popov

Fix bug #77966: Cannot alias a method named "namespace"

This is a bit tricky: In this cases we have "namespace as", which
means that we will only recognize "namespace" as an identifier w

Fix bug #77966: Cannot alias a method named "namespace"

This is a bit tricky: In this cases we have "namespace as", which
means that we will only recognize "namespace" as an identifier when
the lookahead token is already at the "as". This means that
zend_lex_tstring picks up the wrong identifier.

We solve this by actually assigning the identifier as the semantic
value on the parser stack -- as in almost all cases we will not
actually need the identifier, this is just an (offset, size)
reference, not a copy of the string.

Additionally, we need to teach the lexer feedback mechanism used
by tokenizer TOKEN_PARSE mode to apply feedback to something
other than the very last token. To that purpose we pass through
the token text and check the tokens in reverse order to find the
right one.

Closes GH-5668.

show more ...


Revision tags: php-7.4.7RC1, php-7.3.19RC1, php-7.4.6, php-7.2.31, php-7.4.6RC1, php-7.3.18RC1, php-7.2.30, php-7.4.5, php-7.3.17
# 1f48feeb 13-Apr-2020 Máté Kocsis

Improve some TypeError and ValueError messages

Closes GH-5377


# c5fb4f07 13-Apr-2020 Máté Kocsis

Generate function entries from stubs for a couple of extensions

Migrates ext/standard, ext/tidy, ext/tokenizer,
ext/xml, ext/xml_reader, and ext/xml_writer. Closes GH-5381.


# 80598f12 08-Apr-2020 Alex Dowad

Syntax errors caused by unclosed {, [, ( mention specific location

Aside from a few very specific syntax errors for which detailed exceptions are
thrown, generally PHP just emits the def

Syntax errors caused by unclosed {, [, ( mention specific location

Aside from a few very specific syntax errors for which detailed exceptions are
thrown, generally PHP just emits the default error messages generated by bison on syntax
error. These messages are very uninformative; they just say "Unexpected ... at line ...".

This is most problematic with constructs which can span an arbitrary number of lines, such
as blocks of code delimited by { }, 'if' conditions delimited by ( ), and so on. If a closing
delimiter is missed, the block will run for the entire remainder of the source file (which
could be thousands of lines), and then at the end, a parse error will be thrown with the
dreaded words: "Unexpected end of file".

Therefore, track the positions of opening and closing delimiters and ensure that they match
up correctly. If any mismatch or missing delimiter is detected, immediately throw a parse
error which points the user to the offending line. This is best done in the *lexer* and not
in the parser.

Thanks to Nikita Popov and George Peter Banyard for suggesting improvements.

Fixes bug #79368.
Closes GH-5364.

show more ...


Revision tags: php-7.4.5RC1, php-7.3.17RC1, php-7.3.18, php-7.4.4, php-7.2.29, php-7.3.16, php-7.4.4RC1, php-7.3.16RC1, php-7.4.3, php-7.2.28, php-7.3.15RC1, php-7.4.3RC1, php-7.3.15, php-7.2.27, php-7.4.2, php-7.3.14, php-7.3.14RC1, php-7.4.2RC1, php-7.4.1, php-7.2.26, php-7.3.13, php-7.4.1RC1, php-7.3.13RC1, php-7.2.26RC1, php-7.4.0, php-7.2.25, php-7.3.12, php-7.4.0RC6, php-7.3.12RC1, php-7.2.25RC1, php-7.4.0RC5, php-7.1.33, php-7.2.24, php-7.3.11, php-7.4.0RC4, php-7.3.11RC1, php-7.2.24RC1, php-7.4.0RC3, php-7.2.23, php-7.3.10, php-7.4.0RC2, php-7.2.23RC1, php-7.3.10RC1, php-7.4.0RC1, php-7.1.32, php-7.2.22, php-7.3.9, php-7.4.0beta4, php-7.2.22RC1, php-7.3.9RC1, php-7.4.0beta2, php-7.1.31, php-7.2.21, php-7.3.8, php-7.4.0beta1, php-7.2.21RC1, php-7.3.8RC1, php-7.4.0alpha3, php-7.3.7, php-7.2.20, php-7.4.0alpha2, php-7.3.7RC3, php-7.3.7RC2, php-7.2.20RC2, php-7.4.0alpha1, php-7.3.7RC1, php-7.2.20RC1, php-7.2.19, php-7.3.6, php-7.1.30, php-7.2.19RC1, php-7.3.6RC1, php-7.1.29, php-7.2.18, php-7.3.5, php-7.2.18RC1, php-7.3.5RC1, php-7.2.17, php-7.3.4, php-7.1.28, php-7.3.4RC1, php-7.2.17RC1, php-7.1.27, php-7.3.3, php-7.2.16, php-7.3.3RC1, php-7.2.16RC1, php-7.2.15, php-7.3.2, php-7.2.15RC1, php-7.3.2RC1, php-5.6.40, php-7.1.26, php-7.3.1, php-7.2.14, php-7.2.14RC1, php-7.3.1RC1, php-5.6.39, php-7.1.25, php-7.2.13, php-7.0.33, php-7.3.0, php-7.1.25RC1, php-7.2.13RC1, php-7.3.0RC6, php-7.1.24, php-7.2.12, php-7.3.0RC5, php-7.1.24RC1, php-7.2.12RC1, php-7.3.0RC4, php-7.1.23, php-7.2.11, php-7.3.0RC3, php-7.1.23RC1, php-7.2.11RC1, php-7.3.0RC2, php-5.6.38, php-7.1.22, php-7.3.0RC1, php-7.2.10, php-7.0.32, php-7.1.22RC1, php-7.3.0beta3, php-7.2.10RC1, php-7.1.21, php-7.2.9, php-7.3.0beta2, php-7.1.21RC1, php-7.3.0beta1, php-7.2.9RC1, php-5.6.37, php-7.1.20, php-7.3.0alpha4, php-7.0.31, php-7.2.8, php-7.1.20RC1, php-7.2.8RC1, php-7.3.0alpha3, php-7.3.0alpha2, php-7.1.19, php-7.2.7, php-7.1.19RC1, php-7.3.0alpha1, php-7.2.7RC1, php-7.1.18, php-7.2.6, php-7.2.6RC1, php-7.1.18RC1, php-5.6.36, php-7.2.5, php-7.1.17, php-7.0.30, php-7.1.17RC1, php-7.2.5RC1, php-5.6.35, php-7.0.29, php-7.2.4, php-7.1.16, php-7.1.16RC1, php-7.2.4RC1, php-7.1.15, php-5.6.34, php-7.2.3, php-7.0.28, php-7.2.3RC1, php-7.1.15RC1, php-7.1.14, php-7.2.2, php-7.1.14RC1, php-7.2.2RC1, php-7.1.13, php-5.6.33, php-7.2.1, php-7.0.27, php-7.2.1RC1, php-7.1.13RC1, php-7.0.27RC1, php-7.2.0, php-7.1.12, l, php-7.1.12RC1, php-7.2.0RC6, php-7.0.26RC1, php-7.1.11, php-5.6.32, php-7.2.0RC5, php-7.0.25, php-7.1.11RC1, php-7.2.0RC4, php-7.0.25RC1, php-7.1.10, php-7.2.0RC3, php-7.0.24, php-7.2.0RC2, php-7.1.10RC1, php-7.0.24RC1, php-7.1.9, php-7.2.0RC1, php-7.0.23, php-7.1.9RC1, php-7.2.0beta3, php-7.0.23RC1, php-7.1.8, php-7.2.0beta2, php-7.0.22, php-7.1.8RC1, php-7.2.0beta1, php-7.0.22RC1, php-5.6.31, php-7.0.21, php-7.1.7, php-7.2.0alpha3, php-7.1.7RC1, php-7.0.21RC1, php-7.2.0alpha2, php-7.1.6, php-7.2.0alpha1, php-7.0.20, php-7.1.6RC1, php-7.0.20RC1, php-7.1.5, php-7.0.19, php-7.0.19RC1, php-7.1.5RC1, php-7.1.4, php-7.0.18, php-7.1.4RC1, php-7.0.18RC1
# 5a09b9fb 23-Mar-2017 Nikita Popov

Add PhpToken class

RFC: https://wiki.php.net/rfc/token_as_object

Relative to the RFC, this also adds a __toString() method,
as discussed on list.

Closes GH-5176.


# 8a406898 14-Feb-2020 Nikita Popov

Clarify that token_get_all() never returns false

It can only fail in TOKEN_PARSE mode, in which case it will throw.


# 7e0f7b76 28-Sep-2019 Tyson Andre

Reduce memory used by token_get_all()

Around a quarter of all strings in array tokens would have a string that's one
character long (e.g. ` `, `\`, `1`)

For parsing a large numb

Reduce memory used by token_get_all()

Around a quarter of all strings in array tokens would have a string that's one
character long (e.g. ` `, `\`, `1`)

For parsing a large number of php files,
The memory increase dropped from 378374248 to 369535688 (2.5%)

Closes GH-4753.

show more ...


# 5d6e923d 24-Sep-2019 Gabriel Caruso

Remove mention of PHP major version in Copyright headers

Closes GH-4732.


# cfbcd4ca 11-Aug-2019 Stephen Reay

Arginfo stubs for tokenizer


# 7f72d771 28-Mar-2019 Nikita Popov

Revert "Switch to bison location tracking"

This reverts commit e528762c1c59bc0bd0bd6d78246c14269630cf0f.

Dmitry reports that this has a non-trivial impact on parsing
overhead, e

Revert "Switch to bison location tracking"

This reverts commit e528762c1c59bc0bd0bd6d78246c14269630cf0f.

Dmitry reports that this has a non-trivial impact on parsing
overhead, especially on 32-bit systems. As we don't have a strong
need for this change right now, I'm reverting it.

See also comments on
https://github.com/php/php-src/commit/e528762c1c59bc0bd0bd6d78246c14269630cf0f.

show more ...


# e528762c 15-Mar-2019 Nikita Popov

Switch to bison location tracking

Locations for AST nodes are now tracked with the help of bison
location tracking. This is more accurate than what we currently do
and easier to exte

Switch to bison location tracking

Locations for AST nodes are now tracked with the help of bison
location tracking. This is more accurate than what we currently do
and easier to extend with more information.

A zend_ast_loc structure is introduced, which is used for the location
stack. Currently it only holds the start lineno, but can be extended
to also hold end lineno and offset/column information in the future.

All AST constructors now accept a zend_ast_loc* as first argument, and
will use it to determine their lineno. Previously this used either the
CG(zend_lineno), or the smallest AST lineno of child nodes.

On the parser side, the location structure for a whole rule can be
obtained using the &@$ character salad.

show more ...


123456