Newer versions of libxml2 (used by lxml) crash in tostring() when no encoding argument is present. Passing "unicode" as encoding makes tostring() returning already a Python unicode string, so we don't need to decode it anymore.
On Debian Sid where libxml2 2.9.12 is included, the following error occurs without the change: ``` /usr/bin/python3 ../scripts/gen-api-gtkdoc.py xml -d . -o geany-gtkdoc.h \ --sci-output geany-sciwrappers-gtkdoc.h Traceback (most recent call last): File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 460, in <module> sys.exit(main(sys.argv)) File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 389, in main e = DoxyStruct.from_compounddef(n0) File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 321, in from_compounddef e.add_member(p) File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 233, in add_member proc.process_element(xml.find("detaileddescription")) File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 136, in process_element s = self.__process_element(xml) File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 163, in __process_element s += self.__process_element(n) + "\n" File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 167, in __process_element ss = self.at.cb(n.get("kind"), self.__process_element(n)) File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 163, in __process_element s += self.__process_element(n) + "\n" File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 170, in __process_element s += self.get_program_listing(n) File "/build/geany-1.37.1-1+20210903gitb7bd5fa/doc/../scripts/gen-api-gtkdoc.py", line 126, in get_program_listing arr.append(" " + tostring(etree.HTML(html), method="text").decode("utf-8")) File "src/lxml/etree.pyx", line 3437, in lxml.etree.tostring File "src/lxml/serializer.pxi", line 103, in lxml.etree._tostring File "src/lxml/serializer.pxi", line 75, in lxml.etree._textToString UnicodeEncodeError: 'ascii' codec can't encode character '\xe1' in position 130970: ordinal not in range(128) ```
I'm not completely sure why this happens with libxml 2.9.12 (2.9.10 works fine), the XML contents which are processed here should be plain ASCII. Anyway, it might not be bad to set the encoding anyways.
To reproduce, start a Docker container with a Debian Sid image, like: `docker run --rm -it debian:sid` and within the container execute: ```bash apt-get update && apt-get install --no-install-recommends -y git intltool libtool build-essential libgtk-3-dev python3-docutils rst2pdf doxygen python3-lxml nano git clone https://github.com/geany/geany cd geany ./autogen.sh make -C doc ``` You can view, comment on, or merge this pull request online at:
https://github.com/geany/geany/pull/2885
-- Commit Summary --
* Use "encoding" keyword argument for lxml's tostring()
-- File Changes --
M scripts/gen-api-gtkdoc.py (6)
-- Patch Links --
https://github.com/geany/geany/pull/2885.patch https://github.com/geany/geany/pull/2885.diff
Does that work with older (yet not so ancient they don't matter) libxml?
I justed checked again with Debian Stable (Bullseye), Old-Old-Stable (Stretch) and Ubuntu 18.04.
Works as well on oldstable (Buster) here, so I guess we're safe enough. And Travis is happy as well.
Merged #2885 into master.
It turned out this change fixed only one symptom of the real underlying problem: https://gitlab.gnome.org/GNOME/libxml2/-/issues/255 - some changes in libxml2 2.9.12 in combination with lxml broke the `lxml.etree.tostring()` method which we also use. This leads to extra content in the extracted XML elements and this is why we suddenly got non-UTF8 content in the generated GTK doc header (and hence this change was necessary). Recent nightly builds failed at generating translation files with: ``` The following files contain translations and are currently not in use. Please consider adding these to the POTFILES.in file, located in the po/ directory.
doc/geany-gtkdoc.h ``` but this was also only a symptom because now the generated file contained more than the filtered content by the generation script.
Luckily, libxml2 got a workaround for this bug and Debian already included this workaround in the latest `libxml2` package, so future nightly builds on Debian Unstable should work again.
The change in this PR is good anyway and can be kept, I'd say.
github-comments@lists.geany.org