Merging homebrew-core and linuxbrew-core

[MikeMcQuaid]

Eventually we may want to merge homebrew-core and linuxbrew-core. Even if not, minimising the diff between these repositories makes life easier for the linuxbrew-core maintainers.

To do this will require a few things (not in any particular order):

• Migrate Homebrew/homebrew-core CI to GitHub Actions CI Migrate Homebrew/homebrew-core CI to GitHub Actions CI #6255 (which linuxbrew-core already is)
• Have bottled as many linuxbrew-core formulae as possible (in linuxbrew-core)
• Add more Linux DSLs to avoid if OS.linux? blocks (partly for readability and partly so these blocks can be parsed on macOS) like uses_from_macos e.g.
  • a way to applying patches only on a single platform
  • a way to have dependencies only on a single platform
  • a block in def install that's only run on a single platform
• Finish the xorg migration, a few formulae need to be migrated
• Allow applying cellar :any to bottles on a single platform Allow platform-specific cellars in bottle blocks #9315: Reading (OK, bottle: tag specific cellars #10186), Writing (OK, Allow platform-specific cellars in bottle blocks #9315)
• Fix ENV.cc/ENV.cxx handling with gcc on linux: ENV.cc and ENV.cxx is not correctly set when depending on gcc formula #10170, https://github.com/Homebrew/linuxbrew-core/pull/21380
• Reduce the diff between the two repositories to 0
• Add linux CI that has no timeout (self-hosted?)

Notes

• Revisions will stay the same between linux and Mac. If we notice that we have to do too many revision bumps, then we will need to split them: Discussion around linux/mac revision handling #10099

[iMichka]

@sjackman I would like to make gcc@5 the main gcc formula on linux.
brew install gcc would then install gcc 9, like it does on mac. This would allow to cleanup the gcc formula on linux, and ease the transition. Do you think this is doable?

[issyl0]

As of #7280 we've got an audit for legitimate uses_from_macos usage.

[sjackman]

@sjackman I would like to make gcc@5 the main gcc formula on linux.
brew install gcc would then install gcc 9, like it does on mac. This would allow to cleanup the gcc formula on linux, and ease the transition. Do you think this is doable?

My intuition is that on Linux gcc should be an alias to gcc@5.

[sjackman]

As of #7280 we've got an audit for legitimate uses_from_macos usage.

Thanks, Issy!

[iMichka]

My intuition is that on Linux gcc should be an alias to gcc@5

This means that we need os-dependent aliases. Not sure I like that idea. And I still think it is surprising to brew install gcc and get something different than on mac.

[MikeMcQuaid]

I'm with @iMichka that I think the cleanest solution is probably to make the gcc formula the same on both OSs. It's easier to have Homebrew/brew compiler selection code pick a different compiler when needed than it's going to be to have homebrew-core have different logic for GCC versions with the same name.

[sjackman]

Makes sense. I'm on board.

[iMichka]

Short status update, as some huge progress has been made lately:

• I update the checklist on the top of this issue
• Most of the important xorg formulae have been migrated to homebrew-core (which had the nice side-effect that some of maintainers started to work on getting rid of the x11 requirement :)). Around 13 remaining xorg formulae need to be migrated.
• 13 linux-only formulae (out of 43) have been added to homebrew-core and have linux CI there.
• When I started reducing the diff between both repos, there were more than 2000 OS.mac? statments. We are now below 1400.

[iMichka]

There was a proposal from @SMillerDev to start adding linux bottles and more CI to homebrew-core as the migration is ongoing. Initially I was thinking to add all the bottles in one step at the end of the migration; but we can also take a step-by-step approach. We can not re-use linuxbrew-core bottles as-is, as the revisions are sometimes different there.

• Where do we store the new linux bottles on bintray for homebrew-core? I guess in https://bintray.com/homebrew, and we keep the old linux bottles in https://bintray.com/linuxbrew?
• I think we do not need to adapt the merge script from homebrew-core to linuxbrew-core, as the linux sha256 line will conflict anyway, so this is a good thing.
• We will do double bottle builds for some time, in homebrew-core and then in linuxbrew-core. I hope that's fine
• Remains the final question: how will the transition happen from linuxbrew-core to homebrew-core for our users, knowing that revisions might be different between both repos. We do not need to answer this question right now.
• What is the impact of the migration on the shelf-life of old linux bottles? Is this still a thing? I know that @sjackman always thought it was important, but I personally do not have any need for this.

Thoughts?

[SMillerDev]

I think the best option to have would be:

We bottle the formula that are explicitly enabled to be tested in homebrew-core and use those bottles in linuxbrew where we can having less and less bottles in the linuxbrew bintray.

Second best option:

We bottle the formula that are explicitly enabled to be tested in homebrew-core and rebuild the bottles in linuxbrew. Until we flip the switch and we disable linuxbrew.

Worst option:

We don't bottle in homebrew-core, but we keep the same flow until all formulae are upstreamed from linuxbrew and then we start bottles in homebrew-core.

[dawidd6]

The first option looks great, but how can we know on which bintray account the Linux bottle actually is? This will become irrelevant when we merge the core taps together, but for now I think it would be a problem that needs to be sorted out first.

[sjackman]

We bottle the formula that are explicitly enabled to be tested in homebrew-core and rebuild the bottles in linuxbrew. Until we flip the switch and we disable linuxbrew.

I like this option.

[SMillerDev]

The first option looks great, but how can we know on which bintray account the Linux bottle actually is? This will become irrelevant when we merge the core taps together, but for now I think it would be a problem that needs to be sorted out first.

I'd imagine we can try the main one and fall back if it fails?

[iMichka]

I would go for second option. It's less work, and probably easier to handle.

We just need to think about the final migration when we flip the switch, but that's all.

[MikeMcQuaid]

• Where do we store the new linux bottles on bintray for homebrew-core? I guess in https://bintray.com/homebrew, and we keep the old linux bottles in https://bintray.com/linuxbrew?

Yup 👍🏻

• We will do double bottle builds for some time, in homebrew-core and then in linuxbrew-core. I hope that's fine

Fine with me.

• Remains the final question: how will the transition happen from linuxbrew-core to homebrew-core for our users, knowing that revisions might be different between both repos. We do not need to answer this question right now.

Will likely be some custom brew update logic, eventually. I can handle that when the time comes.

• What is the impact of the migration on the shelf-life of old linux bottles? Is this still a thing? I know that @sjackman always thought it was important, but I personally do not have any need for this.

Personally: I don't think it's worth worrying about this. We should probably avoid deleting these bottles but I don't see any need to write code to make them work better.

[MikeMcQuaid]

Oh and: great work on this @iMichka!

[iMichka]

Ok. I'll work on setting up linux bottling in homebrew-core soon for a few main formulae. I think the same rule will apply to homebrew-core as linuxbrew-core: if a formula has a linux bottle, any revision bump or version bump should build a new bottle. If there is no bottle, nothing happens (no linux CI) to that formula.

I'll start working on this once I am done with the self-hosted runners project, unless someone wants to pick this up earlier.

[iMichka]

The xorg migration is done!

[iMichka]

Another step done: cellar information for specific platforms in bottle blocks is now merged!

[iMichka]

All done here. Users were migrated to homebrew-core with release 3.3.0.

There is still an issue open at Homebrew/formulae.brew.sh#566 but it is being worked on.

Closing here.

[MikeMcQuaid]

Congratulations on the completion of this huge effort. Great work @iMichka and all.