project.py: Specify the manifest file to use UTF-8 encoding

[ylz0923]

Specify the manifest file to use UTF-8 encoding

[marc-hb]

Could you please file a bug with some reproduction steps?

Encoding issues are never as simple as they seem...

[marc-hb]

locale.getencoding() is used by default. If it does not already return utf8 on a system then why should west ignore the system's default and hardcode it?

https://docs.python.org/3/library/functions.html#open

Please try this interactively on your system and share the output:

python
import locale
locale.getencoding()

[marc-hb]

Just noticed 3.8 was different :-( https://docs.python.org/3.8/library/functions.html#open

Please also run this:

python
import locale
locale.getpreferredencoding()
locale.getdefaultlocale()
locale.getencoding()

[ylz0923]

>>> import locale
>>> locale.getencoding()
'cp936'
>>>

>>> import locale
>>> locale.getpreferredencoding()
'cp936'
>>> locale.getdefaultlocale()
<stdin>:1: DeprecationWarning: Use setlocale(), getencoding() and getlocale() instead
('zh_CN', 'cp936')
>>> locale.getencoding()
'cp936'
>>>

how to reproduction:
west init -m URL --mr main --mf west.yml myworkspace
zhe west.yml using UTF-8 encoding has Chinese annotation.
error out:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Python311\Scripts\west.exe_main.py", line 7, in
File "C:\Python311\Lib\site-packages\west\app\main.py", line 1085, in main
app.run(argv or sys.argv[1:])
File "C:\Python311\Lib\site-packages\west\app\main.py", line 244, in run
self.run_command(argv, early_args)
File "C:\Python311\Lib\site-packages\west\app\main.py", line 503, in run_command
self.run_builtin(args, unknown)
File "C:\Python311\Lib\site-packages\west\app\main.py", line 611, in run_builtin
self.cmd.run(args, unknown, self.topdir,
File "C:\Python311\Lib\site-packages\west\commands.py", line 194, in run
self.do_run(args, unknown)
File "C:\Python311\Lib\site-packages\west\app\project.py", line 224, in do_run
topdir = self.bootstrap(args)
^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\west\app\project.py", line 313, in bootstrap
manifest = Manifest.from_data(temp_manifest.read_text(),
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\pathlib.py", line 1059, in read_text
return f.read()
^^^^^^^^
UnicodeDecodeError: 'gbk' codec can't decode byte 0xaf in position 151: illegal multibyte sequence

[marc-hb]

Can you try $Env:PYTHONUTF8 = 1 (in powershell) and then try west again?

[ylz0923]

It process successfully, use $Env:PYTHONUTF8 = 1 than west init -m URL --mr main --mf west.yml myworkspace

[marc-hb]

Thanks for testing! So you don't need this PR, correct?

If you prefer to switch your entire Windows system to UTF8, you can do it like this:
https://stackoverflow.com/questions/57131654/using-utf-8-encoding-chcp-65001-in-command-prompt-windows-powershell-window
Then you won't have to choose $Env:PYTHONUTF8 = 1 (or not) for every application and west project.

There is a lot of other, useful information on that page, it's not just about powershell.

Also note the Python default will change in Python 3.15: https://peps.python.org/pep-0686/

[ylz0923]

When other places(manifest.py) call the read_text interface, they specify the encoding method, and I think it’s necessary here as well. 😊

[marc-hb]

When other places(manifest.py) call the read_text interface, they specify the encoding method,

Good point. Can you please fetch and test #711 instead?

[marc-hb]

Superseded by larger #711

[marc-hb]

I just merged larger #711.

@ylz0923 please close this one if confirmed that #711 works for you.

Thanks for your help and patience!