Travis: Consider only testing the latest point releases of each major GHC version

[RyanGlScott]

Currently, base-compat's Travis setup tests no fewer than 32 (!) different configurations. This seems somewhat excessive, especially given that there are only 11 different major versions of GHC represented among the various configurations. This means that 21 of the configurations simply test different minor releases.

While there is some value in ensuring that base-compat works with every minor release of GHC, it causes the CI turnaround time to be quite high. Moreover, most of the CPP used in base-compat only depends on the major version of GHC being used (it might be all of the CPP, in fact, but I haven't tested this). Therefore, the amount of value we gain from testing every minor release is quite low compared to the significant build times they add in each CI run.

I propose that we simply test against the latest point release of each major GHC version. What do others think?

[sjakobi]

Is the motivation to improve developer ergonomics for you?

Also, has CI ever been useful in detecting a change between minor versions?

[RyanGlScott]

Is the motivation to improve developer ergonomics for you?

Yes. That, and I have to maintain a large collection of hacks to support old minor versions of GHC due to old compiler bugs:

base-compat/cabal.haskell-ci

Lines 1 to 18 in 9bf3fc6

	-- 7.8.1 is here because contravariant-1.5.1+ fails to build with 7.8.1
	-- due to https://gitlab.haskell.org/ghc/ghc/issues/8978. Ugh.
	env: 7.0.1:BATTERIES=NO,
	7.2.1:BATTERIES=NO,
	7.8.1:BATTERIES=NO
	-- TODO: Have `BATTERIES=NO` imply no tests
	tests: (>=7.0.2 && <7.2.1) || (>=7.2.2 && <7.4.1) || (>=7.4.2 && <7.8.1) || >=7.8.2
	travis-patches: travis.yml.patch
	-- travis_wait interacts poorly with color_cabal_output
	color: False
	-- Old versions of Haddock have trouble building the documentation for
	-- base-compat-batteries' Data.Type.Equality.Compat module:
	--
	-- * GHC 7.2.2 runs into "synifyType: PredTys are not, in themselves, source-level types."
	-- This bug is apparently so obscure that no issue was ever filed about it!
	--
	-- * GHC 7.6 runs into https://github.com/haskell/haddock/issues/242
	haddock: <7.2.2 || (>=7.4.1 && <7.6.1) || >=7.8.1

Also, has CI ever been useful in detecting a change between minor versions?

There are a handful of changes to base that were introduced in minor versions, and we might consider backporting those changes to base-compat (see, for instance, the base-4.5.1.0 and base-4.7.0.2 sections in #24). If we did do that, it might be beneficial to test the corresponding versions of GHC that these minor releases of base are shipped with. At present, however, I don't believe testing all minor releases of GHC pays its weight.

[sjakobi]

Seems reasonable! 👍

[phadej]

If someone writes a small script which searchs out for all the MIN_VERSION_base used, so one can audit that GHC versions on the both side of the conditional one are tested, then I'd be happy.

BTW, disabling base-compat-batteries can be done using jobs-selections: any and tweaking tested-with in base-compat-batteries.

[RyanGlScott]

If someone writes a small script which searchs out for all the MIN_VERSION_base used, so one can audit that GHC versions on the both side of the conditional one are tested, then I'd be happy.

I'm not sure how fancy of a script you're envisioning, but this crude one-liner tells you how many occurrences of each variant of MIN_VERSION_base are currently used in base-compat:

$ grep -rh "MIN_VERSION_base" base-compat/src/ base-compat-batteries/src/ | sort | uniq -c
      1 #elif MIN_VERSION_base(4,10,0)
      1 # if !(MIN_VERSION_base(4,10,0))
      1 # if MIN_VERSION_base(4,10,0)
      5 #if !(MIN_VERSION_base(4,10,0))
     11 #if MIN_VERSION_base(4,10,0)
      2 #if MIN_VERSION_base(4,10,0) && !(MIN_VERSION_base(4,11,0))
      1 #  if MIN_VERSION_base(4,10,0) && !(MIN_VERSION_base(4,12,0))
      4 #if MIN_VERSION_base(4,10,0) && !(MIN_VERSION_base(4,12,0))
      7 #if !(MIN_VERSION_base(4,11,0))
      1 #if MIN_VERSION_base(4,11,0)
      4 #if MIN_VERSION_base(4,12,0)
      1 # if !(MIN_VERSION_base(4,13,0))
      1 #if !(MIN_VERSION_base(4,4,0))
      1 #if !(MIN_VERSION_base(4,4,0)) || MIN_VERSION_base(4,8,0)
      2 #if MIN_VERSION_base(4,4,0) && !(MIN_VERSION_base(4,8,0))
      1 #if !(MIN_VERSION_base(4,4,0)) || MIN_VERSION_base(4,9,0)
      2 #if MIN_VERSION_base(4,4,0) && !(MIN_VERSION_base(4,9,0))
      3 #if !(MIN_VERSION_base(4,5,0))
      1 #if !(MIN_VERSION_base(4,5,0)) && !(MIN_VERSION_base(4,9,0))
      1 # if !(MIN_VERSION_base(4,6,0))
      6 #if !(MIN_VERSION_base(4,6,0))
      4 #if MIN_VERSION_base(4,6,0)
     12 #if !(MIN_VERSION_base(4,7,0))
     11 #if MIN_VERSION_base(4,7,0)
      1 #if MIN_VERSION_base(4,7,0) && !(MIN_VERSION_base(4,8,0))
      1 #if !(MIN_VERSION_base(4,7,0)) || MIN_VERSION_base(4,9,0)
      2 # if !(MIN_VERSION_base(4,8,0))
     13 #if !(MIN_VERSION_base(4,8,0))
     13 #if MIN_VERSION_base(4,8,0)
      2 # if MIN_VERSION_base(4,9,0)
     10 #if !(MIN_VERSION_base(4,9,0))
     27 #if MIN_VERSION_base(4,9,0)
      1 #if MIN_VERSION_base(4,9,0) && !MIN_VERSION_base(4,10,0)
      1 #if MIN_VERSION_base(4,9,0) && !(MIN_VERSION_base(4,11,0))
      1 #if MIN_VERSION_base(4,9,0) && !(MIN_VERSION_base(4,13,0))

There's some duplication in there, but the point I want to make shines through: every use of MIN_VERSION_base at present uses 0 as the tertiary version number.

[phadej]

@RyanGlScott so the hacks are actually more like,

base-compat-batteries doesn't work with GHC-7.8.1

and not that we have some code written differently specifically for GHC-7.8.1 (or some other minor version)?

[RyanGlScott]

Right, the issues I have to work around are exclusively old GHC bugs, not API differences in base-compat{-batteries}. For instance, I run into https://gitlab.haskell.org/ghc/ghc/issues/8978 when building base-compat-batteries with GHC 7.8.1, which does not affect any other GHC release.

[RyanGlScott]

It sounds like there is a consensus to proceed with this idea, so I have implemented this in #66.