As always my take on this is that if you don't care about portability, why even bother with bash?
My rule of thumb is that if your script becomes too complicated that you find POSIX sh limiting, it's probably a good hint that you should rewrite it in a proper programming language. Bash scripting is an anti-pattern in my opinion, it's the worst of both worlds since it's neither portable nor that much better than POSIX.
I don’t really care about targeting, say, IRIX, or z/OS, or the early-2000s-era Windows POSIX Subsystem. But I do care about targeting Linux, macOS/iOS, all the modern living BSDs, and Android (where applicable.) I also maybe care about letting my software run under a Linux Busybox userland.
It’d be great to know what the lowest-common-demoninator standard is for just that set of targets. I don’t care about POSIX, but if that set of targets had a common formal named standard, I’d adhere to it rigorously.
Since it doesn’t, I just mostly adhere to POSIX — except where it gets to be too much (like having to write 1000-line Bourne Shell scripts.) When that happens, I look around to see if the set of targets I care about all do something the same way. If they do (such as all happening to ship some — perhaps ancient — version of Bash), then I break away from POSIX and take advantage of that commonality.
I don’t know enough about all the things these OSes do or do not have in common to truly code to the implicit standard 100% of the time. I wish I did. So, in most respects, wherever I haven’t done independent research, I have to hold myself to a much stricter standard — POSIX. But it’s only ignorance keeping me doing it!
>I don’t really care about targeting, say, IRIX, or z/OS, or the early-2000s-era Windows POSIX Subsystem.
Sure, those are very niche, but not all BSDs or every Linux distros have bash in the base distribution. And if you work with embedded system bash is very much a luxury. You mention busybox but this one is very much a crap shot because everything in busybox is opt-in. You want to use `seq`? Better make sure that it's configured. On the other hand you almost certainly won't have bash on these systems either.
At any rate if the alternative is between having to install python, ruby, perl or lua on my machine or having to maintain and hack on ginormous bash scripts I know what I'll chose.
But in general I like your approach, it's pragmatic. It's just pretty hard these days to find a system that ships bash but can't give you some decent scripting support, at least in my experience.
not just limited to embedded environments, some docker images used in cloud setups also don't have bash - alpine doesn't have it. The alpine image is used a lot, due to its small size; you can add bash easily, but bash is not part of the base image (there is a difference if you are in an ssh session to a container that doesn't have access to the net)
docker run --rm -it alpine
/ # bash
/bin/sh: bash: not found
Note that targeting bash on macOS means targeting bash v3, as Apple is not updating to bash v4+ (due to GPLv3) and has changed the default interactive shell to zsh in macOS 11.0+.
> it's the worst of both worlds since it's neither portable nor that much better than POSIX
Writing modern bash using the updated syntax, it is easy to make robust scripts and the syntax around things like quoting is simple and consistent.
It seems odd to me to suggest that one should either stick to the old brittle /bin/sh syntax or not use bash at all. It has it's place in the world and it should be done the modern way.
You'll find 10s of thousands of lines of bash in the kubernetes repo and all the files I checked are written using modern bash syntax. There must be a reason they choose bash for these parts.
Of course, in any case everyone should use the tools they like and find most suited to the task at hand.
Because it's everywhere anyone cares about and isn't some weird thing like zsh or Lord forgive us tcsh.
Also all of these modern are basically fancy wrappers around bash scripts. Even dockerfiles is just a way to run some shell commands at image build time and other shell commands at launch time. Much less puppet or ansible or user-data.sh ....
Shellcheck specifically supports POSIX, BASH, and Korn. Each of these has an important role in scripting.
Development of the formal ksh93 was halted and rewound by AT&T after David Korn's departure, precisely because the user community remains extremely intolerant to changes in functionality and/or performance. Korn shell development that requires a "living" shell should likely target both ksh93 and the MirBSD Korn shell (the default shell in Android).
BASH has diverged from the Korn shell in a few (annoying) ways, but remains core to the GNU movement and rightly commands its own audience. It is much larger than the MirBSD Korn shell.
POSIX shells were developed to target truly minimal systems - POSIX rejected Korn likely due to (Microsoft) XENIX on 80286 systems that only allowed 64K text segments. Korn was able to run on such machines, but the code was not maintainable. Clean C source for a POSIX shell is far easier to achieve for XENIX, and remains a better fit for embedded environments. This is likely why POSIX is not Korn.
For small scripts, bash is still the easier way to go. Other programming languages are not that immediate to do simple things. How many lines of code take piping a command into another in python, for example?
Also bash is not portable, but you can assume to find a recent version of bash on any Linux system. And for BSD systems or macOS, you can install it with a command. And yes, there are other ancient UNIX systems, but who uses them? But what other programming language can you be sure to find on any system?
Finally, bash it's quite efficient. I mean that it doesn't have the weight that a full programming language like Python has, especially the time that it takes to activate the program interpreter. That could make a difference in scripts that are executed a lot of time from other script or other programs. Or even in wrapper scripts, since they doesn't add noticeable delay when executing a command.
There is still a reason to use and know bash to me. Of course implementing whole programs in bash doesn't make sense, but for simple scripts it does.
Sometimes you write POSIX shell (or very nearly POSIX) for portability. Sometimes you write in "a proper programming language" for more powerful/flexible features. And sometimes you write Bash to get both.
Bash is ported to basically every system, it's a small binary (by today's grotesque standards), and it gives you some added functionality that would be otherwise annoying to re-implement in POSIX sh (or Bourne sh). But it's also far less complex than Python. In today's world, if someone has any shell, it's a safe bet that they either already have Bash, or they can install it just like they'd have to install Python.
It's less painful in the long term to use Bash rather than Python. Python is more time-consuming and costly. It's a larger space requirement, you have to manage more runtime dependencies and a virtual environment, debugging is worse, and there's more opportunity for bugs because it's more complex. Bash scripts also rarely get bugs down the line due to dependencies changing, whereas this happens in Python so frequently we have to pin versions in a virtualenv. When's the last time you pinned the version of a Bash script dependency?
There are two specific cases where you do not have BASH.
1. Ubuntu/Debian /bin/sh is the Debian variant of the Almquist shell, known as dash. The only non-POSIX element it includes is the "local" keyword (afaik).
2. Busybox, with the Almquist shell. Busybox also has a "bash" which offers a few cosmetic improvements to ash (an alias of [[ to [ is one that I can see in the source). I write POSIX shell scripts on Windows with busybox quite often.
The only other shell that omits everything outside of the POSIX specification is mrsh. I don't think mrsh is widely-deployed in any major distribution.
(You also aren't going to have bash in AIX/HP-UX and maybe a Solaris base load, but as the parent article says, we aren't talking about dinosaur herders.)
For #1, you can just put #!/bin/bash at the top of the file to use Bash. Bash is still available, it’s just not the default for scripts that specify #!/bin/sh.
#2 is still currently tricky, but Rob Landley (former Busybox maintainer) is working on a full bug-for-bug compatible Bash clone called toysh which will be included in an upcoming release of Toybox[1]. Once that’s released, I’m looking forward to (hopefully) never writing a script for BusyBox ash again.
Bash is ugly and rife with pitfalls, but it can be incredibly productive for certain classes of problems if you know what you’re doing. Trying to use Python or Go to whack some external commands together feels very cumbersome to me.
For pure programming, “real” languages are absolulely preferable in almost every way, but all of them fail when it comes to running external programs and redirecting their output in a way that doesn’t make me pull my hair out. As a heavy user of both “real” programming languages and shell scripting languages, I’m left craving something that brings the best of both worlds.
There are a number of newer shell projects in this vein that are very exciting, like Oil [0], Elvish [1], and Xonsh [2]. I was also hopeful that Neugram [3] would go somewhere, but the project seems to have died out. While many people cite Bash’s lack of portability as a reason not to use it, I find Bash to be very portable for my use cases and avoid using these newer shells for their lack of portability. Maybe one day we can have nice things.
Superficially it's quite similar to elvish (purely by coincidence) but it's aimed around local machine use (eg by developers and devops engineers) so is as much inspired by IDEs as it is by shells.
Sad that none of those projects took the easy route. If you want to replace Bash, do it the way Bash replaced Bourne: Make it backwards compatible.
Bash acts like 'historical versions of sh' if it was invoked as sh. So, a Bash replacement could act like Bash if it was invoked like bash. Then implement all the extra crazy shit if you invoke it as slash or something.
Assuming it was a small, fast, compiled binary, this would take off in all the distros pretty much immediately. And if you really want it to succeed, implement a spec first, and implementations second. Add a slash implementation to Busybox, and then it's on every embedded Linux system in the world.
It would certainly be nice, but I wouldn’t call that the easy route. Creating a new shell that’s fully backwards compatible with the monstrosity that is Bash sounds like a massive undertaking. POSIX or the Bourne shell is probably an easier target, and there are numerous alternative shells that are POSIX-compatible and add nice new features or optional alternate syntaxes. But I agree, to properly dethrone Bash would probably require backwards compatibility with it.
Personally I don't see the point dethroning bash. I mean Bash still hasn't completely dethroned the Bourne shell. And now there is a lot more competition in shells than there ever was.
Personally I think the better approach is to accept that bash/sh will always be about for legacy stuff and for alternative shells to carve out a niche elsewhere. Particularly because to some of bash/sh's pain points can't be addressed without break compatibility in the first place (like handling file names with spaces).
This used to be my take, but I'm not sure any more. Now, I don't think Python is ideal, and I am looking for something that maybe fits better than it in that gray zone of programming language vs "quick" script, but I can't see your point about debugging and bugs: bash is notoriously annoying to write bug free, and I can't see what's the issue with the Python debugger.
Also if we're in that gray zone of bash vs Python, you can pretty much stick to the stdlib.
I'm hopeful that Deno can be just that runtime. Since its a single binary, its trivial to install and script for. TBH, I don't have any technical reasons for JS/TS over python besides that I have coworkers that have never had to learn python and would prefer not to.
/**
* cat.ts
*/
for (let i = 0; i < Deno.args.length; i++) {
const filename = Deno.args[i];
const file = await Deno.open(filename);
await Deno.copy(file, Deno.stdout);
file.close();
}
A distribution that removes Python 2.7 once upstream security support has been cut. Since Python is frequently used on network-facing services, it is unconscionable to allow its use past January 2020. Heck, one might even have setuid programs in Python and you don't really want to risk an unfixed local privilege escalation either.
> AWS Linux 2
A butchered version of CentOS
> and macOS
Not a distribution, and Apple doesn't exactly have a good track record with keeping things up to date or secure.
> Or more likely, the world isn't as clear cut as you might think
Given that the vast majority of the Python ecosystem is no longer compatible with 2.7, there isn't even a technical reason for kicking around 2.7 anymore.
Python 2.7 is trivial to build and run if you really, really want or need it, but since all the major upstream distributions have removed it, it sends a clear signal that you must take the risks into serious consideration before doing it.
Where size of data >> RAM (but less than free space in /tmp). It of course possible, but I suspect would require more effort and would work slower than a shell line above.
If you want to compare memory efficiency. The sort+uniq -c would likely be more optimal in a high level language as you would there keep only the count+value in a defaultdict or alike instead of sorting the entire dataset, which requires keeping it all in mem.
As for brevity, yeah, this particular example would require 5 lines instead of 1.
How say you? To even ask the system a question in python, you have to "import os". Is that not a library? I'm not a python-first type of person, but I have hacked enough scripts to be functional. My one take away is how python core is not very useful, and to even remotely do anything one must have a list of imports.
then why must it be imported as a 3rd party? this is my question on python in general. i get needing to explicitly import things that are not part of the core/standard, but why are core/standard required to be imported? why can't the methods they provide not just immediately available.
*clearly, i've never taken a python CS style class, but it this importing core functions is just strange to me
"Third party" means a third party provided it: not yourself, and not the Python distribution, but someone else.
"os" is a second-party library. You didn't write it; it was included in the Python distribution for you.
What you're asking, I think is why the "os" functions are not, in Python terminology, "builtins." The builtins are functions that are available without any kind of import: like len(), max(), sorted(), etc.
Why do you have to "import os"? Python was designed that way, that's all. So were the majority of other languages that come with standard libraries. (JavaScript is one exception that comes to mind, but there aren't many such exceptions.)
The discussion above relates to "virtual environments", which are ways to manage third-party dependencies. My claim is that virtualenvs, while handy for general-purpose development, are basically pointless for replacing shell scripts. You don't need them -- the core language and its standard library are sufficient for most shell scripting purposes. I'm basing this opinion on my ~20 years of using Python for this kind of work.
You can have other shells that are not bash, true. But you are guaranteed to find a bash shell in any Linux distribution (well, except minimal distros like OpenWRT or Alpine Linux that for saving space use busybox). Let's say that in any server or desktop Linux installation you find bash.
You would probably find also python, but what version? Python3? These days you are probably guaranteed to find them, except that there are old Red Hat servers still in production where is not installed. And Python3, what version? 3.7? Older? What features are safe to use and what are not? Do you have to research to find out? Or you are stuck with using Python 3.2 just to be safe? Well, in the time you wasted thinking about that, you would have finished writing the script in bash.
3. the ability to run parts of scripts and build up into a bigger script
Currently I write a lot of small-ish scripts using either bash or bash plus a text editor. I would like to use a less obtuse language than bash, but the interactive part is really important. I don't want a system where I have to run exploratory and test commands in bash and then translate them into a different language, because that means figuring out how to say it twice and doing extra debugging.
My rule of thumb is that if your script becomes too complicated that you find POSIX sh limiting, it's probably a good hint that you should rewrite it in a proper programming language. Bash scripting is an anti-pattern in my opinion, it's the worst of both worlds since it's neither portable nor that much better than POSIX.