> Seems like nobody can get anywhere near the functionality of Jenkins.
I've heard Jenkins is a nightmare to administer (not sure about the new versions). Seems like an opportunity: the power of Jenkins with modern sensibility.
Jenkins is a nightmare because everyone decided to spend months customising it with groovy and it became overly specific to your team/dept. The servers themselves are quite simple (single .jar, etc). These days we have containers, and it's made CI a much better thing.
I agree with this. The quality of your Jenkins experience will be pretty much inversely proportional to how much groovy script is in your Jenkinsfiles.
Every new permutation of parameters is a new chance for something to go wrong, and there's basically no sane way to unit test or validate any of your "helper" functions, so they're all write-once-change-never.
Which is not necessarily a problem inherent in Jenkins— Jenkins just gives you more than enough rope to hang yourself. GitLab CI can get silly too, but culturally it inherits from the much simpler Travis model, where you're basically expected to just be wrapping some other build tool (eg, tox from the Python world) rather than rolling the whole thing yourself.
Groovy is arguably the wrong language choice, even if it worked fine in other aspects. The string handling situation alone is crazy, which is compounded if you are using it with bash (which has its own crazy ideas about string escaping). That's a not too uncommon scenario.
To be fair, basically everything that interops/embeds bash like this ends up in a similar place— you see the same double-escape ridiculousness in CMakelists files and Nix package definitions.
> Jenkins is a nightmare because everyone decided to spend months customising it with groovy and it became overly specific to your team/dept
Yes. One thousand times this.
If you are running simple scripts it's fine. If you are using the declarative pipeline, it's fine. The moment you start adding Groovy you'll be down a path that is filled with sadness and anger. Mind you, even the folks behind Jenkins will advise you not to use any complex Groovy scripts(including for performance reasons - you can easily overwork the jenkins master).
I've been focusing on Concourse because it forces the usage of containers for everything. You don't have to care about what's installed in the worker node, you just use a container that has the stuff you want. Simple inputs and outputs.
You _can_ do the same sort of thing with Jenkins (but be aware of all the bugs still open regarding containers). But Jenkins doesn't force you to do anything, nor it gives you an easy and out of the box solution to string containers together. Left unchecked, you have your reproducible builds running in completely unreproduceable magical build machines – that noone really understands how it all works.
Did a migration of a few hundreds of pipelines from one server to another and it exposed a lot of dependencies we didn't know we had. Plugins, jars, packages installed in build machines, you name it.
If you must use Jenkins, please try to avoid Groovy to the max, write anything that's non-trivial as an executable (even if it is a bash script) and call it from the main pipeline. Use containers if you can to avoid build machine dependencies. Try to use declarative pipelines too unless your jobs are very simple, and avoid the 'script' blocks. Do not use the scripting pipeline to avoid inviting groovy to your home.
I worked on a Jenkins cluster with around 150 agents and many, many builds per day. It all ran so smoothly and I swear it was because there was zero groovy.
Just to contradict the complaining about groovy scripts: I reduced something like 100 jobs to 2 using groovy in jenkins, and it has been a godsend. I'm not having any problems with it.
Mind you, the Groovy stuff was implemented as an afterthought, and if you aren't handy with Groovy (I am) it can be challenging to learn. But compared to the horrible GUI-driven alternative, it was a no-brainer in the end.
Really, the more you're doing CI, the more you want scripts - doesn't have to be Groovy. I'd honestly be perfectly fine just doing things with a bash shell on a vanilla linux install. I only use Jenkins because my team insists on it, for what? So we can put a web UI in front of CI. I'd even tend to agree that with supply-chain attacks being what they are, and jenkins being a never-ending fountain of security patches, putting a web UI in front of CI isn't worth it.
I've heard Jenkins is a nightmare to administer (not sure about the new versions). Seems like an opportunity: the power of Jenkins with modern sensibility.