[test][pkey_check] Add more invalid SM2 key tests

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/13359)