UTF-16 display

[datasynergyuk]

I'm not sure if this a bug or something I did wrong.

I created a UTF-16 (LE with BOM) test file and added text from several different languages (Chinese, Russian, Japanese, Yiddish and Hindi). The text looks like this in Notepad:

I tried viewing the file in the latest HexCtrl sample by setting the encoding to UTF-16. This only displayed the Russian text:

I wondered if this was a font problem, so I changed the font to Courier New. The result was similar but a little better:

Finally, I tried in another well-known hex application (also using Courier New) as UTF-16. This worked ok:

I have attached the test file to this post.

I'm not sure if this is a UTF-16 interpretation issue or a font problem?

Multilanguage.txt

[jovibor]

Definitely it's a font related. Here what I see with SimSun-ExtB font. My guess is that default Courier-New doesn't support some glyphs, or something like that.

This whole Unicode feature is just experimental, to make it behave properly is a lot of work.

[datasynergyuk]

Thank you. That explains things better.

I understand that HexCtrl support is experimental and it will need more work to improve this.

The thing I remained confused about is the font - Other applications can correctly display these characters in "Courier New" but this doesn't currently work in HexCtrl. I found that this can be partially fixed by removing the CF_NOSCRIPTSEL flag when calling the font dialog. This allows the script combo box to be used and different combinations (like Yiddish) to work. A similar outcome can be achieved with the default Consolas font by specifying the CHINESEBIG5_CHARSET flag when the font is initialized. This makes me think there may be a way to display multiple scripts with the same font but I haven't found how the other application does it yet.

Postscript: I found that the MingLiU font works well

[jovibor]

Another problem here though is glyph width. With non-standard charset it starts occupying two-letter width for some chars and one-letter width for others.
Even Winhex sometimes fails to render these Unicode chars properly.
Here you can see a full char width on the first picture, and only half of it on the second.

A similar outcome can be achieved with the default Consolas font by specifying the CHINESEBIG5_CHARSET flag

Yes, but font becomes very ugly, as opposed to Winhex where Consolas looks good even with chines glyph rendered. Would be nice to see how it's done.

All in all, making it all work and render properly with every encoding and every font is quite of a headache.

[datasynergyuk]

Another problem here though is glyph width. With non-standard charset it starts occupying two-letter width for some chars and one-letter width for others.

I thought the WinHex text display for UTF-16 looked a little further spaced apart. I wonder if it treats everything as two-letter width to ensure all characters align still. Presumably, variable width would mean HexCtrl also needed to store a map of each text character width so that it could correctly position the caret and highlighting correctly?

Yes, but font becomes very ugly, as opposed to Winhex where Consolas looks good even with chines glyph rendered. Would be nice to see how it's done.

I'm still thinking about this. The only way I can think of so far is that it re-creates the font for each character or when a change is needed. This would be a lot more intensive. I cannot (yet) see a way to change script dynamically without doing this. Maybe there is a way. I know very little about Windows font handling.

All in all, making it all work and render properly with every encoding and every font is quite of a headache.

Yes. All part of making HexCtrl work for the (at least) 4.5bn people who use these characters. Asia is the most populous continent and I'm sure there are lots of cases I cannot even think of.

[jovibor]

This inconsistency was resolved in 6e87ec3.