You can view, comment on, or merge this pull request online at:
https://github.com/geany/geany/pull/3315
-- Commit Summary --
* CI: Use MSYS2 for Windows build
-- File Changes --
M .github/workflows/build.yml (131) A scripts/ci_mingw64_geany.sh (327)
-- Patch Links --
https://github.com/geany/geany/pull/3315.patch https://github.com/geany/geany/pull/3315.diff
@eht16 pushed 1 commit.
2a4f14667aa47bc5e78a34a0c09cfc05d82635e1 CI: Use MSYS2 for Windows build
@eht16 pushed 1 commit.
94b6e948360cd7a6bcc1dbe2e52a12ec737fda5e Update actions to latest version for Node 16 update
@kugel- commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
Isn't that something that should be done in geany build itself? Anyone who builds Geany from source on Windows should do this right?
@elextr commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
Git should convert line endings on checkout since the default value of `core.eol` is `native` but maybe MSYS2 doesn't register "native" as CRLF? Maybe the scripts in MSYS2 should do `git config core.eol crlf`?
@eht16 commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
I don't expect from Geany's build system to know whether it need to convert text files to the target system's preferences. IMO this is very Windows specific and basically only necessary to be able to read our text files with Windows' Notepad which is not able to handle different line endings.
In Geany, we would need to properly integrate it in Autotools and Meson and also depend on "unix2dos".
Regarding Git's `core.eol`: I'm not what happens on a native Windows clone but here we use the already cloned repository from the CI job and even if we would clone it ourselves, it would be LF as everything in this script is executed on Linux.
Changing the setting to `crlf` might have other side effects we do not want.
@elextr commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
Windows' Notepad which is not able to handle different line endings.
Works on my windows 11 tablet[^1]. Google says it was fixed in 2018, so any _supported_ version of Windows should be ok ;-P
Regarding Git's core.eol: I'm not what happens on a native Windows clone but here we use the already cloned repository from the CI job and even if we would clone it ourselves
Ok, so its necessary for the CI job, but I was really addressing @kugel- "Anyone who builds Geany from source on Windows should do this right?" but IIUC their checkout would do it for them with default git settings. If tarballs are going to be buildable on windows I guess its needed there as well since the tarball files are LF endings unless we make a special Windows tarball ... or just say they are *x or modern Windows only.
Changing the setting to crlf might have other side effects we do not want.
Yes, IIUC "Bad things" will happen if we merge anything back on Linux with that setting.
[^1]: No, its a _tablet_ so Geany is no use as I don't develop on it, sorry can't help with windows builds ;-P
@kugel- commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
Well the CI builds shouldn't have "more features" than normal builds from source - source code is king!! If a hard dependency in the normal build is not wanted then there could be an optional dependency that is _coincidentally_ met by the CI.
I wouldn't want manual git or tarball builds to regress compared to CI builds (e.g. on Windows 7). Stuff like this is hard to track down even for experienced contributors.
IMO: Let's open an issue regarding the line ending issue. Then we either deal with it in the normal build or just live with it as suggested by @elextr . I can help with meson if that's a problem.
@kugel- commented on this pull request.
- sed -i -E "s/^!define PRODUCT_VERSION_ID "@VERSION@.0.0"/!define PRODUCT_VERSION_ID "${MAJOR}.${MINOR}.${PATCH}.90"/" ${GEANY_BUILD_DIR}/geany.nsi.in
+} + + +build_geany() { + cd ${GEANY_BUILD_DIR} + log "Running autogen.sh" + ./autogen.sh + log "Running configure" + ./configure ${CONFIGURE_OPTIONS} + log "Running make" + make + log "Running install-strip" + make install-strip + + cd /
????
@kugel- commented on this pull request.
-ts http://zeitstempel.dfn.de/ \
+ -h sha512 \ + -in ${1} \ + -out ${1}-signed + mv ${1}-signed ${1} + else + echo "Skip signing due to missing certificate" + fi +} + + +sign_geany_binaries() { + log "Signing Geany binary files" + for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file}
Why not sign all files?
Can you add a stub with an (perhaps commented out) example on how to build new dependencies for CI builds? I'm still afraid that it's too hard for a random guy's PR to add a dependency without breaking CI.
@elextr commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
IMO: Let's open an issue regarding the line ending issue.
Agreed, and I would have thought the tarball problem would exist already, can anyone test it?
@eht16 commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
Could any of you open that issue? I still don't get the problem you see, as outlined above, a GIT clone on Windows probably gives the correct line endings and so I'm not really able to create an issue for an issue I don't understand :).
Here we are talking about a native Linux clone with native LF line endings and those we need to convert. IMO this is special to the CI build and so needs to be done here.
@eht16 commented on this pull request.
-ts http://zeitstempel.dfn.de/ \
+ -h sha512 \ + -in ${1} \ + -out ${1}-signed + mv ${1}-signed ${1} + else + echo "Skip signing due to missing certificate" + fi +} + + +sign_geany_binaries() { + log "Signing Geany binary files" + for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file}
osslsigncode (and probably the original "signcode.exe" tool) are made for signing binary files, see https://github.com/mtrojnar/osslsigncode#what-can-it-do.
To get sure things haven't changed, I tried to sign an arbitary text file: ``` root@9214a3c76917:/build# osslsigncode sign -certs /certs/cert.pem -key /certs/key.pem -n "Test file" -i "https://www.geany.org/" -ts http://zeitstempel.dfn.de/ -h sha512 -in /etc/wgetrc -out wgetrc-s Unrecognized file type: /etc/wgetrc 140619785353856:error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149: 140619785353856:error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:309:Type=PKCS8_PRIV_KEY_INFO 140619785353856:error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149: 140619785353856:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:572: 140619785353856:error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149: 140619785353856:error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:309:Type=RSAPrivateKey 140619785353856:error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:142: 140619785353856:error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149: 140619785353856:error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:309:Type=PKCS8_PRIV_KEY_INFO Failed ```
Note, for the CI builds nothing is signed because no certificates are available to the build container. It wouldn't make sense to sign automatically built code since we have no control over the build system and its integrity, how should we tell the user that the generated binaries are really clean? Additionally, we had to give the build system access to the private key of a certificate which would mean Github can access and in the worst case manipulate the certificate. So, CI builds should never be trusted.
@eht16 pushed 1 commit.
ef2b6e5039e021b1b5d613eff90996f9c53da709 Remove unnecessary directory changes
Can you add a stub with an (perhaps commented out) example on how to build new dependencies for CI builds?
I don't get it :(. Where should I add the example and what do you mean by "build new dependencies"?
@eht16 commented on this pull request.
- sed -i -E "s/^!define PRODUCT_VERSION_ID "@VERSION@.0.0"/!define PRODUCT_VERSION_ID "${MAJOR}.${MINOR}.${PATCH}.90"/" ${GEANY_BUILD_DIR}/geany.nsi.in
+} + + +build_geany() { + cd ${GEANY_BUILD_DIR} + log "Running autogen.sh" + ./autogen.sh + log "Running configure" + ./configure ${CONFIGURE_OPTIONS} + log "Running make" + make + log "Running install-strip" + make install-strip + + cd /
Removed, thanks for the pointer.
Nevermind, the G-P PR shows how to install additional build dependencies.
@kugel- commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
A tarball created on Linux will have wrong line endings on Windows. The normal build system should take care of that instead of being restricted to CI.
And how do you create Win32 releases if not using CI (as you mentioned in another comment)?
@eht16 commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
A tarball created on Linux will have wrong line endings on Windows. The normal build system should take care of that instead of being restricted to CI.
As said above, if you want to do this, please create an issue or even better a PR to make the changes to the build system(s). Then we can remove the line conversion here. I won't do it because I still think it's not such a big deal and I won't spend the time effort it needs.
And how do you create Win32 releases if not using CI (as you mentioned in another comment)?
As described on https://wiki.geany.org/howtos/win32/msys2, especially using the [attached](https://wiki.geany.org/_media/howtos/win32/geany-release.py.txt) script which also does the line ending conversion to CRLF. In short, using a clean GIT clone with the release tag. At least with my GIT config the relevant files all have LF line endings and therefore need to be converted. Using a release tarball wouldn't help here because it is created on Linux and so also has LF line endings.
@eht16 pushed 1 commit.
47d397e52155d4510452030de3d7a966b29a22dc Change infrastructure builders branch
@kugel- commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
I opened #3350
I would like to see the conversion to be removed from this PR since I'm not even sure it's necessary at all. Don't recent Windows version deal fine with LF line endings. I don't know if notepad alone is an indication but see https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/, this fix is included in Windows 10 and 11.
@eht16 pushed 1 commit.
4ed3402359d70b0d63afafd62e96945bc95ff6ab Remove CRLF text conversion
@eht16 commented on this pull request.
- log "Signing Geany binary files"
+ for binary_file_pattern in ${GEANY_RELEASE_BINARY_PATTERNS[@]}; do + for binary_file in $(ls ${binary_file_pattern}); do + sign_file ${binary_file} + done + done +} + + +convert_text_files_to_crlf() { + log "Converting line endings to CRLF in text files" + for text_file_pattern in ${GEANY_RELEASE_TEXTFILE_PATTERNS[@]}; do + for text_file in $(ls ${text_file_pattern}); do + mime_type=$(file --brief --mime-type ${text_file}) + case $mime_type in text/*) + unix2dos ${text_file}
Thanks for creating the issue and the investigation. Nice that Notepad now can handle non-CRLF line endings. This was the main reason I convert the line endings.
So I just removed the code from the build script here and in the G-P counterpart.
@eht16 pushed 1 commit.
30becfd0c152969510490f13e3b42778e74e8d4d Remove CRLF text conversion
@eht16 pushed 1 commit.
4bdd15c1aa86f916a8dc362c4d09af570c51b344 GH Actions: Replace deprecated "set-output" command
@eht16 pushed 1 commit.
6cb951f3510ffb6b4d84b7c5d78010cd8ffe1f8f Remove test settings
@eht16 pushed 1 commit.
f146c2eb70715e94d1e31931cdebb08c86f81799 CI: Use MSYS2 for Windows build
I just cleaned up the commit history and removed temporary test settings, so this can be merged.
@kugel- approved this pull request.
@eht16 pushed 1 commit.
e2066d4f5f9600b0d8acd0ebeaeddebc93d1f15a Use Ubuntu 20.04 for Linux CI builds
Merged #3315 into master.
github-comments@lists.geany.org