CJK line wrap

[SuibianP]

I am trying to convert an HTML file with mixed Chinese and English texts to Markdown, but line wrapping is performed exclusively at whitespaces. I enabled the east_asian_line_breaks extension, but from the documentation it seems mainly about insertion of space upon combination such as when rendering PDF, not the other way round. #3080 does not seem to have covered this either.

curl "https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.zh-hans" | pandoc -f html+east_asian_line_breaks-native_divs-native_spans -t markdown_strict+fancy_lists+east_asian_line_breaks-raw_html-native_divs-native_spans-fenced_divs-bracketed_spans

知识共享组织("Creative Commons"，"知识共享")
不是一家律师事务所，也不对外提供法律服务或建议。提供知识共享公共许可协议(Creative
Commons Public
License，以下缩写为CCPL)并不导致"律师—当事人"关系或其他法律关系的建立。知识共享(Creative
Commons)按其现状提供协议文本和相关信息。知识共享(Creative
Commons)对于其许可协议，或通过该协议提供的作品(material)或相关信息不提供任何担保，在最大可能程度内，也不对因使用其协议或通过该协议提供的作品(material)或信息而造成的损失承担损害赔偿责任。

Is there any way to have it wrapped correctly, possibly with implicit breaks between consecutive CJK characters?

[TomBener]

I'm not sure how you actually want to wrap the text. If I understand correctly, probably you should add the option --wrap=preserve:

curl "https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.zh-hans" | pandoc -f html+east_asian_line_breaks-native_divs-native_spans -t markdown_strict+fancy_lists+east_asian_line_breaks-raw_html-native_divs-native_spans-fenced_divs-bracketed_spans --wrap=preserve

The output is as follows:

知识共享组织("Creative Commons"，"知识共享") 不是一家律师事务所，也不对外提供法律服务或建议。提供知识共享公共许可协议(Creative Commons Public License，以下缩写为CCPL)并不导致"律师—当事人"关系或其他法律关系的建立。知识共享(Creative Commons)按其现状提供协议文本和相关信息。知识共享(Creative Commons)对于其许可协议，或通过该协议提供的作品(material)或相关信息不提供任何担保，在最大可能程度内，也不对因使用其协议或通过该协议提供的作品(material)或信息而造成的损失承担损害赔偿责任。

[SuibianP]

I am trying to have it wrap at the normal 72 columns (--wrap=auto --columns=72). The expected result is as follows,

知识共享组织("Creative Commons"，"知识共享")不是一家律师事务所，也不对外
提供法律服务或建议。提供知识共享公共许可协议(Creative Commons Public
License，以下缩写为CCPL)并不导致"律师—当事人"关系或其他法律关系的建立。
知识共享(Creative Commons)按其现状提供协议文本和相关信息。知识共享
(Creative Commons)对于其许可协议，或通过该协议提供的作品(material)或相关
信息不提供任何担保，在最大可能程度内，也不对因使用其协议或通过该协议提供
的作品(material)或信息而造成的损失承担损害赔偿责任。

This is produced with Vim formatoptions m and M.

[jgm]

I don't have any first-hand knowledge of CJK languages; is it the case that lines can be broken anywhere between two CJK characters? (Probably, it's more complicated, I would guess.)

[SuibianP]

There are characters that are not permitted at the start or end of line. Japanese seems to have additional requirements that differ even across standards (e.g. no English words at the end of line or otherwise).

More generally, there is UAX #14 Unicode Line Breaking Algorithm which could be the canonical solution for all languages, although I am unsure how feasible it would be to implement the standard in Pandoc.

For an initial implementation, breaking anywhere between CJK characters might be good enough for formats which are not presentational and where CJK whitespace is insignificant e.g. markdown_strict+east_asian_line_breaks. It would also make sense to just weld this behaviour into east_asian_line_breaks.