Freeciv

Software is hard. Consulting with my Unix expert friend to help improve homebrew-appbundle.sh I discovered that the meson build doesn't work the way I thought it did. I thought it would include all the homebrew installed packages into the libfreeciv.dylib file but it doesn't, so my goal of making a self-contained app that doesn't require the end user to install homebrew can't be done with meson. So no point in trying to put a zip file on the servers for the download page.

The homebrew-install-packages.sh file is still needed. And I still need to finish working on homebrew-appbundle.sh - what I want is to dynamically set the version string in the Info.plist file, starting with the output of "freeciv-server -v".

homebrew-appbundle.sh now done. Using output of "freeciv-server -v" didn't work, better to use output of fc_version.

What's weird is I discovered the output of "freeciv-server -v" is written to the screen in such a way that it cannot be redirected or piped.

./freeciv-server -v > file

writes the info to the screen just the same, and the resulting file is empty. Didn't think that was possible to do.

Reply To ddeanbrown

./freeciv-server -v > file writes the info to the screen just the same, and the resulting file is empty.

Yeah, seems it writes to stderr. Which might count as a bug - showing version number when it's not requested is not an error message.

So you should be able to just redirect stderr instead of stdout to file: './freeciv-server -v 2> file'. But it's probably better to use ./fc_version regardless. For one thing, there should not be anything else than the version number on its output.

Reply To ddeanbrown

my goal of making a self-contained app that doesn't require the end user to install homebrew can't be done with meson.

Umh, do you have some meson related problem in this? Of course just switching the freeciv build system from autotools to meson doesn't mean that the dependency packaging would implement itself on its own. Our plan was just that it will be better to implement it to work together with meson, than spending effort on autotools side.

Thinking some more - it may be possible to do what I want with meson, have to experiment with setting environment variables. The issue is dealing with installing the homebrew packages inside the app bundle instead of the default location = /usr/local. That's what I've been doing with my autotools built version. Homebrew says installing into non default location is not officially supported, but it has been working OK. OTOH, the other Mac package manager, MacPorts, does officially support arbitrary install locations, but it doesn't actually work. Go figure...

The other way to do what I want would be if there were a linker flag that caused the contents of dylibs to get copied into the linker output, instead of a reference to the dylib (and recursively). I have not been able to find such a linker flag, and maybe it doesn't exist. Tho in theory it could be done.

Reply To ddeanbrown

the contents of dylibs to get copied into the linker output, instead of a reference to the dylib (and recursively).

Umh, don't you simply mean static linking? Though that's not a dylib thing by the very definition of it being dy(namic)lib and not a static one. For freeciv build part (libfreeciv) you can make static build with '-Ddefault_library=static', but that does not rebuild recursively rebuild homebrew libraries to static ones.

Yeah, static linking with static libraries is what I want. I tried experimenting with '-Ddefault_library=static' and that changes libfreeciv.dylib to libfreeciv.a and also eliminates the @rpath problem, which is good for me. But it doesn't help my other problem. I also found that meson has options "-Dprefer_static=true", and "find_library(... , static: true)", which would solve my problem if it worked. But I can't get it to work on my Mac - I think it may be a Mac meson bug. I couldn't find a bug database for meson, so I opened a question on StackOverflow. https://stackoverflow.com/questions/75000597/is-this-a-bug-with-meson-using-static-libs-on-mac

Found the bug repo for meson and submitted a bug report. https://github.com/mesonbuild/meson/issues/11237

Note that we already have bunch of iconv handling stuff in meson.build, to work around issues like the 'iconv_' vs 'libiconv_' namespaces macro magic (on some platforms, but not all) and whether it's part of c-library, separate gnu libiconv, or another kind of separate libiconv. Multiply the complexity by supporting different combinations on build host and cross-compile target (cross-compile to Windows, with no native iconv, isn't exactly trivial)

You may want to do your changes to that part of the meson.build, and in any case you should be careful not to break anything it already does, by hardcoding something that's already correctly detected.

This ticket seems now to try to implement multiple patches (/steps) -> likely needs to be split to multiple tickets. Can you do that, the way you see them properly organized, or do you want me to try to figure out how it should go?

Do you need #44814 already, for the things you're trying to achieve now? I hope to (finally) get to it soon.

About homebrew-install-packages.sh:

https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh

While first version of the script is the first version to iteratively improve from, this part will have a high priority to get improved - for build reproducibility you really should have a pinned version, and not something that (potentially) changes between each build. Not having it pinned is also extremely likely to break build of any release really quickly (leaving users out of *any* buildable release until a new one, updated to build against homebrew's current HEAD, comes)

I fixed the 'iconv_' vs 'libiconv_' namespace problem by adding to the CPPFLAGS and LDFLAGS.

#44814 is not holding me up.

I agree about splitting out a new ticket for making a stand-alone Mac app bundle. New ticket #46474. With what I have learned it now looks like it is possible. This ticket can be only for homebrew-install-packages.sh and homebrew-appbundle.sh.

Agree about adding pins to homebrew-install-packages.sh. Except I think meson shouldn't be pinned. New version attached.

Reply To ddeanbrown

This ticket can be only for homebrew-install-packages.sh and homebrew-appbundle.sh.

This should be part of the release/distribution, I think. So add to Makefile.am.

Agree about adding pins to homebrew-install-packages.sh. Except I think meson shouldn't be pinned. New version attached.

So, "HEAD" in that github URL should be replaced by a specific version.

So add to Makefile.am.

I assume in the EXTRA_DIST = section?

So, "HEAD" in that github URL should be replaced by a specific version.

I suspect that "HEAD" only affects the version of homebrew, not the versions of the packages. But I'm not sure, manual doesn't say. Also, manual has "...Homebrew/install/master/install.sh" instead of "...Homebrew/install/HEAD/install.sh".

Just to be clear, what I think the potential problem is - that a newer version of some package breaks something. So, unless that "HEAD" does affect the versions of the packages, what we would need to do, for each release, is note the version of each package used to build it, have the script download that version and pin it. Script would also have to check if it's a newer freeciv release than what is currently installed, and if so, then un-pin everything and download the maybe newer versions of the packages and then pin them.

Reply To ddeanbrown

Just to be clear, what I think the potential problem is - that a newer version of some package breaks something. So, unless that "HEAD" does affect the versions of the packages, what we would need to do, for each release, is note the version of each package used to build it, have the script download that version and pin it. Script would also have to check if it's a newer freeciv release than what is currently installed, and if so, then un-pin everything and download the maybe newer versions of the packages and then pin them.

How long does homebrew keep old versions available for download?

This is the reason we store snapshots of msys2 environment on our own servers. So ten years from now we will still have the environment to build freeciv-2.6 Windows packages, no matter what happens to msys2 upstream. It was actually just some months ago that msys2 project migrated their repositories to new servers. This would have broken all our past releases, had they been depending on msys2 upstream repository URL.

But maybe this is another can of worms that we should not be too concerned about in the initial implementation. Maybe have it in the RoadMap to have more setup more robust in this sense by final 3.1.0 release?

How long does homebrew keep old versions available for download?

Let's find out by trying for version 2.6.3 of freeciv - woops, there's no support for getting versions of anything. Maybe you can get HEAD instead of master. Oh well, so much for that idea...

So it looks like we have no choice but to live with the risk that a newer version of some package breaks something. I wouldn't think it's a big risk, but I don't really know. Not sure if it is still worthwhile to do the pins - if user ran the script when there was a package version that breaks something, they would have to un-pin it to get a newer version that fixes it. In any case I expect very few users will be using this script.

Updated the homebrew-install-packages.patch to include change to makefile.am

Noticed that the "docs = " section of makefile.am has a couple of entries which are not in the

"install_data(

section of meson.build. Is that OK?

Reply To ddeanbrown

Noticed that the "docs = " section of makefile.am has a couple of entries which are not in the "install_data( 'COPYING',..." section of meson.build. Is that OK?

Likely mostly dummy files that the autotools *demand* to be present in a project. Not sure if people would find them useful or not.

- "if ! [ -e /usr/local/bin/brew ] then" missing ";" before "then" -> will likely always parse as false (command "/usr/local/bin(brew then" isn't legal)
- "#install packages" - I would add a space between "#" and the text, and have a capital "I" (capital letter also in the beginning of the other comment)

Lack of pinning means that we don't have reproducible builds. Which is bad even in the cases when the build is not really broken. Reduces both security and debuggability. Anyway, let's try to get the initial version of this in, and continue iterative process from it (and in a new ticket).

Reply To cazfi

Lack of pinning means that we don't have reproducible builds. Which is bad even in the cases when the build is not really broken. Reduces both security and debuggability. Anyway, let's try to get the initial version of this in, and continue iterative process from it (and in a new ticket).

Mac runner updates on github this week have alleviated this problem a bit. It's third time the mac CI build seems to have broken in a couple of days, with no regressions on the freeciv side involved.

In a way this sets even the official release build at risk. What if things change between your last test before the release, and the time you are about to build the release? We can take the risk with 3.1 betas, of course, but I'd say it's a high priority to get this right by the stable releases.

Reply To cazfi

Reply To cazfi

Lack of pinning means that we don't have reproducible builds. Which is bad even in the cases when the build is not really broken. Reduces both security and debuggability. Anyway, let's try to get the initial version of this in, and continue iterative process from it (and in a new ticket).

One answer to https://apple.stackexchange.com/questions/101090/list-of-all-packages-installed-using-homebrew seems like what we want for pinned/reproducible environments (I originally used term "pin" without knowing that homebrew has such a term with a bit different semantics from what we want):

brew bundle may also be interesting if you are asking because you want to manage your brew installation. This includes casks, which brew list does not. It is aimed at having reproducible Homebrew setups.

# creates Brewfile in the current directory from currently-installed packages
brew bundle dump
# edit Brewfile
# install everything from the Brewfile
brew bundle

| Mac runner updates on github this week have alleviated this problem a bit. It's third time the mac CI build seems to have broken in a couple of days, with no regressions on the freeciv side involved.

Possibly related - I noticed Homebrew changed the package dependencies for the sdl2_mixer package recently.

Removing the homebrew-appbundle.sh.patch from this ticket because there is now one attached to https://osdn.net/projects/freeciv/ticket/47392

Agree that homebrew's "pin" is not what we want. I will research the brew bundle command.

I have researched the brew bundle command and it doesn't do what we want. The line about "reproducible Homebrew setups." does not mean specific versions of packages (formulas in homebrew-speak). That stackexchange entry has a link "brew bundle" which goes to https://github.com/Homebrew/homebrew-bundle, where it says "Note

Homebrew does not support installing specific versions of a library, only the most recent one, so there is no good mechanism for storing installed versions in a .lock file.

If your software needs specific versions then perhaps you'll want to look at using Vagrant to better match your development and production environments."

I think the only way to preserve specific versions of packages would be to compress and save the "Cellar" folder, along with the "include", "opt" and "lib" folders that have aliases into it. It's 1.1 gig, 377 meg compressed. I'm dubious about adding a 377 meg file to the git repo.

Reply To ddeanbrown

I'm dubious about adding a 377 meg file to the git repo.

For windows we have https://files.freeciv.org/packages/windows/msys2/ and under version control doc/README.msys2 telling which environment package is correct (for that specific version)

If the giant file isn't in the repo but just on the server then this approach looks doable. A bit of testing shows need to compress the "homebrew" folder after deleting 1 un-needed bit, resulting zip file is 478 meg. This new approach allows a better way to implement https://osdn.net/projects/freeciv/ticket/46262, can replace the current repo scripts "platforms/macos/homebrew-appbundle.sh" and "platforms/macos/homebrew-install-packages.sh" with 1 script (with a better name) which lets user build freeciv.app. The current approach does a normal homebrew install, new approach puts the homebrew stuff inside the app bundle, which gives 2 plusses -

1 doesn't interfere with any homebrew installation the user might already have.

2 the resulting user built freeciv.app will be identical to the official released one, and use the same versions of the homebrew packages.

1 At time of release, I use a script file (based on what I already have) to build the release zip file containing freeciv.app & readme.txt. This script also creates a zip file of the "homebrew" folder needed to do the build.

2 The zip file containing the release app gets uploaded to sourceforge.net and freeciv.org.

3 The zip file of the "homebrew" folder gets uploaded to https://files.freeciv.org/packages/...something...

4 The new script file in the repo (for #46262), would download that zip file instead of installing homebrew and homebrew packages.

5 Users can run that new script file at any time and get a freeciv.app that's identical to the official released one.

I have that new way of doing things written, only problem is where to upload/download the archived homebrew package versions. The homebrew-install-packages.sh is no longer needed, so I deleted that patch. I want to rename the homebrew-appbundle.sh to buildMacApp.sh - is it possible to rename a file in the git repo? Or to delete the old file and add the new one?

The work in tickets #46474 & #46262 is now interdependent, so I think it would be best to close out those tickets as superseded by this one, and have one patch for all these changes -

1 platforms/macos/buildStandaloneApp.sh - New script to build release self-contained Mac app

2 platforms/macos/MacAppREADME.txt - readme to accompany above

3 platforms/macos/buildMacApp.sh - New script for user to build Mac app

4 doc/README.MacInstall - readme to explain how to use above

5 doc/Makefile.am - add README.MacInstall

6 meson.build - add README.MacInstall to install_data()

Reply To ddeanbrown

is it possible to rename a file in the git repo?

'git mv'