UTF-8 BOM handling

[hjtappe]

Hi!

The BOM handling should take into account the different encoding types indicated by the BOM. Just removing the BOM only works for the most common way. The hungarian translation of the widely-used ckeditor nevertheless is an example where this fails. A derived test file showing the exception is included in the patch.

Thanks for considering this patch.

[mrclay]

I think this is outside the scope of JSMin, but this logic could go in a different class whose only purpose is to detect/convert these particular non-UTF8 encodings. Ideally it would also support iconv.

[mrclay]

I'd recommend JSMin\EncodingDetector with method getEncoding($input) and it would return 'UTF-8' if no BOM was present.

[hjtappe]

Hi! Thanks for the feedback. I'll take care. Please give me a few days, I'm still learning git & github. :-)
Actually, the class would need to return either the iconv-modified $input or perform the mbstring settings. I'll do some research to extract the functionality it from the JSMin class.

[hjtappe]

Please review the new encoder class and advise it anything should be changed.
Thanks a lot.

[hjtappe]

Was it OK to merge with your master or does it cause merge problems? It seems, the already-merged changes are now also in the pull request - but I did want to use all new test files in my feature branch...

[glensc]

The get_encoding method relies on default_encoding, which is valid for PHP 5.6+, for earlier version should check for previous ini variables:

• https://wiki.php.net/rfc/default_encoding

and also the default value has changed to utf-8, so this would be breaking change because earlier versions the ini values default to to iso8859-1.

and new code does always decode even if mbstring and iconv both are missing, resulting fatal error.