2406 stories
·
1 follower

The terminal, the console and the shell

1 Comment

Published on 2021-01-13. Modified on 2021-01-14.

The other day, as I was going through some of my old notes, I stumbled upon something I had written about the console, the terminal and the shell on Unix-like operating systems. I have decided to rewrite these notes in order to share them here on my website. So without further ado we will now stroll down memory lane and take a quick look at the origins of the Unix terminal and shell. And I will also give my advice to new users on Linux or BSD regarding the choice of terminal emulator and shell.

Table of contents

The terminal and the console

Early computers where huge machines that consisted of multiple cabinets, e.g. one cabinet for the CPU, one or more cabinets for tape drives, one cabinet for each disk drive, one cabinet for a punched card reader and one cabinet for a high speed printer. The image below is a Univac 9400 system from 1967 consisting of several cabinets.

A Univac 9400 Mainframe
A Univac 9400 mainframe computer.

The "console" was the "control console". In the pictures below is a "UNIVAC 1" control console and a "UNIVAC 2" control console.

UNIVAC 1 control console
A UNIVAC 1 control console.

UNIVAC 2 control console
A UNIVAC 2 control console.

The word terminal comes from the Latin "terminus", meaning "an end, a limit, boundary line", indicating that it's the terminating end or "terminal" end of a communications process. You will sometimes hear the description "a dumb terminal" when referring to a text-based environment where the computer is just taking input and showing text while the real work happens at the other end, typically in a mainframe.

The teleprinter or TTY was the first kind of terminal. Rather than a monitor you would have a literal typewriter in front of you. When you typed on it, you would see the text on a piece of paper and that text would be send to the computer. When the computer replied, you would see the typewriter print on the paper. Some models could also be used to create punched tape for data storage (either from typed input or from data received from a remote source) and to read back such tape for local printing or transmission.

In the picture below is a Teletype 33 ASR Teleprinter. It was introduced in 1963 as an electro-mechanical teleprinter and it was one of the most popular terminals in the communications industry. The ASR stands for Automatic Send and Receive. The ASR 33 had a built-in punched tape reader and tape punch which allowed the user to save programs.

33 ASR Teleprinter
A Teletype Model 33 ASR teleprinter, usable as a terminal.

NOTE:
If you're interested I can highly recommend that you take a look at Charles Baetsen's Hobby Page as he has a wealth of information about the Teletype 33 ASR in the form of both user manuals, installations manuals and videos.

In the picture below Ken Thompson (sitting) and Dennis Ritchie (standing) is using a 33 ASR teleprinter connected to a PDP-11, the picture is from 1972.

Ken Thompson and Dennis Ritchie

Later, as computers became much smaller, it was possible to integrate multiple components into one single unit, with both a video monitor and a keyboard put together inside it, and the "console" now became more or less synonymous to the "terminal". Both the "console" and the "terminal" now referred to the physical video terminal that had replaced both the teleprinter and the old control console.

The video terminal provides a way for the kernel and other processes to send text output on the monitor to the user, and to receive text input from the user via the keyboard.

The VT100 is a video terminal, introduced in August 1978 by Digital Equipment Corporation (DEC). It was one of the first terminals to support ANSI escape codes for cursor control and other tasks, and added a number of extended codes for special features like controlling the status lights on the keyboard. This led to rapid uptake of the ANSI standard, becoming the de facto standard for terminal emulators.

DEC VT100
A DEC VT100

The VT220 is an ANSI standard video terminal introduced by Digital Equipment Corporation (DEC) in November 1983. The VT240 added monochrome vector graphics support to the base model, while the VT241 did the same in color. The 200 series replaced the VT100 series. Among its major upgrades was a number of international character sets, as well as the ability to define new character sets, and a much lighter keyboard.

DEC VT220
A DEC VT220

Today, in the software world, "console" and "terminal" has become completely synonymous.

The virtual terminal

A virtual terminal or virtual console is a program that simulates a physical terminal. For example, both the Linux kernel and BSD kernels support virtual terminals - terminals that are logically separate, but which access the same physical keyboard and monitor.

The virtual terminal gives the impression that several independent terminals are running concurrently. Each virtual terminal can be logged in with a different user and it can run its own shell and have its own font settings. The virtual terminals each use a device /dev/ttyX, and you can switch between them by pressing Alt+FX (where X is equal to the virtual terminal number, beginning with 1).

The terminal emulator

Emulation refers to the ability of a computer program to emulate, i.e. imitate, another program or device. A terminal emulator is a computer program that emulates a physical terminal within some other display architecture, such as the X Window System.

Many different terminal emulators have been developed that emulate terminals such as the VT52, VT100, VT220, VT320, IBM 3270/8/9/E, IBM 5250, and many others.

The terminal emulator takes the input you type at the keyboard and convert those to ASCII characters which it sends to the shell, or to a program running under the shell (more about the shell later). The terminal emulator also takes the stream of output characters from the various programs you run via the shell and displays them on the monitor.

The purpose of the terminal emulator is to allow access to the command line while working in a graphical user interface, such as the X Window System. Since the shell is "expecting" to interface with a human through a terminal, and we don't use a physical terminal while in a graphical environment, we need the terminal emulator.

You can see what terminal types are available on most Linux distributions by running ls /lib/terminfo/* (the path may be different on your system). On OpenBSD it's ls /usr/share/terminfo/*

xterm

xterm was originally written as a stand-alone terminal emulator for the VAXStation 100 (VS100) by Mark Vandevoorde, a student of Jim Gettys, who worked at DEC's Cambridge Research Laboratory. It became clear that xterm would be more useful as part of X Window System than as a standalone program, so it was retargeted to the X Window System.

The xterm program is the default terminal emulator for the X Window System. It provides DEC VT102/VT220 features and other selected features from higher-level terminals such as VT320, VT420 and VT520. It also provides Tektronix 4014 emulation for programs that cannot use the window system directly. If the underlying operating system supports terminal resizing (for example, the SIGWINCH signal in systems derived from 4.3bsd), xterm will use the facilities to notify programs running in the window whenever it is resized.

Around 1996 the main line of development shifted to the XFree86 implementation of the X Window System and xterm is now maintained by Thomas Dickey, who is also the current lead developer of Lynx, a popular customizable text-based web browser.

Early versions of xterm emulated the VT102 and Tektronix 4014. Later versions added control sequences for DEC and other terminals such as:

  • VT220: Added in patch 24.
  • VT320: Added in patch 24.
  • VT420: DECSTR (soft terminal reset) was added in patch 34.
  • VT520: Although not officially emulated, parts of VT520 features were implemented. Controls DECSMBV and DECSWBV for setting the margin- and warning-bell volume was added in patch 254.

As with most X applications, xterm can be customized via global X resources files (e.g. /usr/lib/X11/app-defaults/XTerm), per-user resource files (e.g. ~/.Xresources), or command-line arguments. Most of the command-line options correspond to resource settings, as noted in the xterm manual page.

While the name of the program is xterm, the X resource class in ~/.Xresources is XTerm, e.g.:

XTerm*utf8: 1

The xterm manual page provides a full list of features and options.

xterm is still being actively developed, it works really great across many different systems, it has extremely low input latency, it has many hidden gems and it is my favorite terminal emulator! For that reason I'm going to share some settings you can use in your ~/.Xresources file:

XTerm*faceName: "DejaVu Sans Mono"
XTerm*faceSize: 12
XTerm*renderFont: true
! Dynamically change font size with CTRL+SHIFT+PageUp/PageDown
XTerm*faceSize1: 12
XTerm*faceSize2: 14
XTerm*faceSize3: 16
XTerm*faceSize4: 18
XTerm*faceSize5: 20
XTerm*faceSize6: 22
XTerm*utf8: 1
XTerm*termName: xterm-256color
XTerm*borderWidth: 0
XTerm*autohint: true
XTerm*backarrowKey: false
XTerm*bellIsUrgent: false
XTerm*cursorBlink: false
XTerm*ScrollKey: true
! Fix ALT key (check in mc with Alt+h)
XTerm*metaSendsEscape: true
XTerm*eightBitInput: false
XTerm*ttyModes: erase ^?
XTerm*fastScroll: true

! I like a lot of scrollback.
XTerm*saveLines: 100000

! Use CLIPBOARD by default.
XTerm*selectToClipboard: true
! Hack xterm to add selection to both PRIMARY and CLIPBOARD.
<Btn1Up>: select-end(PRIMARY, CLIPBOARD, CUT_BUFFER0)

! xterm defines a whole suite of "actions" for manipulating the terminal e.g.
! copy-selection(), hard-reset(), scroll-back(), etc. These actions can be
! mapped to mouse/key combinations using the translations resource.
!
! Normally, selected text is stored in PRIMARY, to be pasted with Shift+Insert
! or by using the middle mouse button. With these settings you can use
! CTRL+SHIFT+v/c and you can make Xterm copy to clipboard so that you can use
! CTRL+v in GUI applications.
XTerm*translations: #override \n\
    Shift Ctrl <Key>Prior: larger-vt-font() \n\
    Shift Ctrl <Key>Next: smaller-vt-font() \n\
    Shift Ctrl <Key>C: copy-selection(CLIPBOARD) \n\
    Shift Ctrl <Key>V: insert-selection(CLIPBOARD) \n\
    <Key>BackSpace: string(0x7f) \n\
    <Key>Delete: string(0x1b) string("[3~")

! Theme.
! <a href="https://github.com/logico-dev/Xresources-themes" rel="nofollow">https://github.com/logico-dev/Xresources-themes</a>
#include ".dotfiles/xresource-themes/ubuntu.Xresources"

Last, but not least, many people become surprised when they discover that xterm has a menu.

  • CTRL+left mouse button - for the main options menu.
  • CTRL+right mouse button - for the VT fonts menu.
  • CTRL+mouse wheel - for the VT options menu.

st

st is a simple terminal implementation for Xorg developed by the suckless project. It is intended to serve as a lightweight replacement for xterm.

st currently supports:

  • Most VT10X escape sequences
  • Serial line support
  • XIM support
  • utmp via utmp(1)
  • Clipboard handling
  • Mouse and keyboard shortcuts (via config.h)
  • UTF-8
  • Wide-character support
  • Resize
  • 256 colors and true colors
  • Antialiased fonts (using fontconfig)
  • Fallback fonts
  • Line drawing

Configuration of st is done by manually editing the config.h file and then recompiling st. Because st is very small and very well designed, it is compiled very quickly, even on something like a Raspberry Pi. The configuration file is very well documented and most issues is answered in the FAQ.

Alacritty

Alacritty is an open-source GPU accelerated terminal emulator written in Rust for Linux, BSD, macOS and Windows. Alacritty focuses on performance and simplicity.

Alacritty supports scrollback, truecolor, copy/paste, opening URLs by clicking with the mouse, custom key bindings, and more.

Alacritty is configured by editing its configuration file alacritty.yml that contains documentation for all available fields. The GitHub releases page for Alaritty contains a alacritty.yml file for each release that can be used as a boilerplate.

Alacritty does not create the alacritty.yml file for you, but it looks for one in the following locations:

  • $XDG_CONFIG_HOME/alacritty/alacritty.yml
  • $XDG_CONFIG_HOME/alacritty.yml
  • $HOME/.config/alacritty/alacritty.yml
  • $HOME/.alacritty.yml

Other terminal emulators

Many other terminal emulators exist. This is a short list of some of them.

What terminal emulator should I use?

Whether you're new to the world of open source operating systems, and whether you use Linux or BSD, you have most likely run into some of the more "fanatical" people in the community telling you that this or that terminal emulator is bad, bloated, slow, outdated, or something else.

The fact is that unless you find yourself in some specific and rare edge case, it generally doesn't matter what terminal emulator you use!

Even though xterm is considered bloated, because it contains tons of code that is able to emulate strange and outdated terminals, it is still one of the terminal emulators with the lowest input latency and it is very customizable.

However, terminals such as the GNOME terminal, Konsole and the Xfce terminal are considered more user friendly by some as they can be easier to setup and manage. They also have very good out-of-the-box integration to the desktop system or window manager they support and they add many features that can make it easier to use the terminal and be productive in the X window environment.

One point worth mentioning is that if you depend on your terminal emulator to run your shell scripts faster, you're doing something wrong! Shell scripts are not something you deploy when speed matters!

My advice is that you don't worry about which terminal emulator is the fastest or most popular, try out some of the different ones and use the one that gets the most out of your way and that makes you the most productive. If you have to spend hours getting basic functionality working because you're dealing with some exotic or outdated terminal, then that's just not worth it.

Terminal multiplexer

A terminal multiplexer can be thought of as a console based version of a graphical window manager, or as a way of putting virtual terminals into any login session. It is a wrapper that allows multiple terminal based programs to run at the same time, and it provides features that allow the user to use the programs within a single interface productively. This enables some of the following features: persistence, multiple windows, and session sharing.

tmux and GNU Screen are two popular terminal multiplexers for Linux and BSD.

tmux vs GNU Screen

Theo de Raadt, the founder and project leader for OpenBSD, was impressed with the security of the tmux design:

The most impressive thing about tmux, in my view, is how frustrating the code audit was. In 2 hours, I found only one or two nits that had very minor security consequences. It was not accepted into the tree based on license alone. It is high quality code.

In terms of functionality, screen and tmux both perform similarly and offer the same main features. However, it is the way you access those features that is very different. tmux offers incredibly flexible scripting capabilities.

The environment variable TERM

The environment variable TERM tells applications the name of a terminal description to read from the terminfo database (see man terminfo). Each description consists of a number of named capabilities which tell applications what to send to control the terminal. For example, the cup capability contains the escape sequence used to move the cursor up.

You can read more about TERM in the manual page.

Escape sequences

ANSI escape sequences are a standard for in-band signaling to control cursor location, color, font styling, and other options on video text terminals and terminal emulators. Certain sequences of bytes, most starting with an ASCII escape character and a bracket character, are embedded into text. The terminal interprets these sequences as commands, rather than text to display verbatim.

An escape sequence is a combination of characters that has a meaning other than the literal characters contained therein. It is marked by one or more preceding (and possibly terminating) characters.

As mentioned above, the VT100 terminal implemented the more sophisticated ANSI escape sequences standard for functions such as controlling cursor movement, character set, and display enhancements.

ANSI sequences were introduced in the 1970s to replace vendor-specific sequences and became widespread in the computer equipment market by the early 1980s. They are used in development, scientific, commercial text-based applications as well as bulletin board systems to offer standardized functionality.

Although hardware text terminals have become increasingly rare in the 21st century, the relevance of the ANSI standard persists because a great majority of terminal emulators and command consoles interpret at least a portion of the ANSI standard.

The shell

A shell is a computer program that serves as a command-line interpreter. The shell is both an interactive command language and a scripting language, and is used by the operating system to control the execution of the system using shell scripts. The shell exposes the operating system's services to a human user or other programs.

Operating system shells use either a command-line interface (CLI), i.e. a terminal, or graphical user interface (GUI), i.e. a terminal emulator, depending on a computer's role and particular operation. It is named a shell because it is the outermost layer around the operating system.

A shell process is the program that prompts you for input, takes your commands, and runs them for you. The shell implements a read-eval-print loop (REPL).

The most generic sense of the term "shell" means any program that users employ to type commands. A shell hides the details of the underlying operating system and manages the technical details of the operating system kernel interface, which is the lowest-level, or "inner-most" component of most operating systems.

The shell knows nothing about displaying characters on the monitor or about handling input keystroke codes from the keyboard - that is up to the hardware and software that is implementing the terminal. That is why we interact with the shell using the terminal, however, direct operation via serial hardware connections or Secure Shell are common for server systems. All Unix shells provide filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration.

A shell script is a computer program run by the shell. Instead of direct user input via the keyboard the shell script basically contains a list of such input. This means that you can create a "recipe" of the commands you wish to run and then have the shell perform these commands automatically. One benefit of such a script is that you can reuse it across multiple systems, and you can also save it for later repeated usage.

Because the various shells also provide different control structures for condition-testing, iteration, variables and other programming related functionality the various dialects of shell scripts are considered to be scripting languages.

Typical operations performed by shell scripts include program execution and file manipulation. A script which sets up the system environment, runs programs, and does any necessary cleanup, logging, etc. is typically called a wrapper script.

When a user logs into the system, a shell program is automatically executed for the duration of the session. The type of shell, which may be customized for each user, is typically stored in the user's profile, for example in the local /etc/passwd file or in a distributed configuration system such as NIS or LDAP. However, the user may execute any other available shell interactively.

On computers with a windowing system, such as Microsoft Windows or macOS, some users may never use the shell directly. On Unix systems, the shell has historically been the implementation language of system startup scripts, including the program that starts a windowing system, configures networking, and many other essential functions. However, some system vendors have replaced the traditional shell-based startup system (init) with different approaches, such as systemd.

The first Unix shell was the Thompson shell (sh), written by Ken Thompson at Bell Labs and distributed with Versions 1 through 6 of Unix, from 1971 to 1975. The Thompson shell introduced many of the basic features common to all later Unix shells, including piping, simple control structures using if and goto, and filename wildcarding.

The Thompson shell was modeled after the Multics shell, developed in 1965 by American software engineer Glenda Schroeder, who is noted for implementing the first command-line user interface shell while working as a member of the staff at the MIT Computation Center. The Multics shell was itself modeled after the RUNCOM. The rc suffix on some Unix configuration files, e.g. .vimrc, is a remnant of the RUNCOM ancestry of Unix shells. The term stands for the phrase "run commands". rc may also be expanded as "run control", because an rc file controls how a program runs. In The Art of Unix Programming, Eric S. Raymond consistently refers to rc files as "run-control" files.

Tom Van Vleck, a Multics engineer, has also reminisced about the extension rc:

The idea of having the command processing shell be an ordinary slave program came from the Multics design, and a predecessor program on CTSS by Louis Pouzin called RUNCOM, the source of the .rc suffix on some Unix configuration files.

This is also the origin of the name of the Plan 9 shell by Tom Duff, "The rc shell". It is called "rc" because the main job of a shell is to "run commands".

Popular shells

The most widely distributed and influential of the early Unix shells were the Bourne shell and the C shell. Both shells have been used as the coding base and model for many derivative and work-alike shells with extended feature sets.

This is a short list of some of the more popular shells.

Bourne shell

The Bourne shell was the default shell for Version 7 Unix. Unix-like systems continue to have /bin/sh - which will be the Bourne shell, or a symbolic link or hard link to a compatible shell - even when other shells are used by most users.

The Bourne shell was developed by Stephen Bourne at Bell Labs and it was a replacement for the Thompson shell, whose executable file had the same name - sh. The Bourne shell was released in 1979 in the Version 7 Unix release distributed to colleges and universities. Although it is used as an interactive command interpreter, it was also intended as a scripting language and contains most of the features that are commonly considered to produce structured programs.

It gained popularity with the publication of The Unix Programming Environment by Brian Kernighan and Rob Pike - the first commercially published book that presented the shell as a programming language in a tutorial form.

Features of the Version 7 UNIX Bourne shell include:

  • Scripts can be invoked as commands by using their filename.
  • May be used interactively or non-interactively.
  • Allows both synchronous and asynchronous execution of commands.
  • Supports input and output redirection and pipelines.
  • Provides a set of built-in commands.
  • Provides flow control constructs, quotation facilities, and functions.
  • Typeless variables.
  • Provides local and global variable scope.
  • Scripts do not require compilation before execution.
  • Does not have a goto facility, so code restructuring may be necessary.
  • Command substitution using backquotes: `command`.
  • Here documents using << to embed a block of input text within a script.
  • for ~ do ~ done loops, in particular the use of $* to loop over arguments, as well as for ~ in ~ do ~ done loops for iterating over lists.
  • case ~ in ~ esac selection mechanism, primarily intended to assist argument parsing.
  • sh provided support for environment variables using keyword parameters and exportable variables.
  • Contains strong provisions for controlling input and output and in its expression matching facilities.

The Bourne shell was the first to feature the convention of using file descriptor 2> for error messages, allowing much greater programmatic control during scripting by keeping error messages separate from data.

Stephen Bourne's coding style was influenced by his experience with the ALGOL 68C compiler that he had been working on at Cambridge University. In addition to the style in which the program was written, Bourne reused portions of ALGOL 68's if ~ then ~ elif ~ then ~ else ~ fi, case ~ in ~ esac and for/while ~ do ~ od (using done instead of od) clauses in the common Unix Bourne shell syntax. Although the v7 shell is written in C Bourne took advantage of some macros to give the C source code an ALGOL 68 flavor. These macros (along with the finger command distributed in Unix version 4.2BSD) inspired the International Obfuscated C Code Contest (IOCCC).

C shell

The C shell (csh) is a Unix shell created by Bill Joy, the founder of Sun Microsystems, while he was a graduate student at University of California, Berkeley in the late 1970s. The C shell has been widely distributed, beginning with the 2BSD release of the Berkeley Software Distribution (BSD) which Joy first distributed in 1978. Other early contributors to the ideas or the code were Michael Ubell, Eric Allman, Mike O'Brien and Jim Kulp.

The main design objectives for the C shell were that it should look more like the C programming language and that it should be better for interactive use. And the C shell's built-in expression grammar and support for arrays were all strongly influenced by C.

Like all Unix shells, the C shell supports filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration. What differentiated the C shell from others, especially in the 1980s, were its interactive features and overall style. Its new features made it easier and faster to use. The overall style of the language looked more like C and was seen as more readable.

This following example illustrates the C shell's more C like conventional expression operators and syntax.

Bourne shell:

#!/bin/sh
if [ $days -gt 365 ]
then
   echo "This is over a year"
fi

C shell:

#!/bin/csh
if ( $days > 365 ) then
   echo "This is over a year."
endif

tcsh

tcsh is a Unix shell based on and backward compatible with the C shell (csh). It is essentially the C shell with programmable command-line completion, command-line editing, and a few other features. Unlike the other common shells, functions cannot be defined in a tcsh script and the user must use aliases instead (as in csh). It is the default shell for FreeBSD.

Ksh

The Korn shell (ksh), written by David Korn and based on the original Bourne Shell source code, was a middle road between the Bourne shell and the C shell.

The Korn shells syntax was chiefly drawn from the Bourne shell, while its job control features mostly resembled those of the C shell. The functionality of the original Korn Shell (known as ksh88 from the year of its introduction) was used as a basis for the POSIX shell standard. A newer version, ksh93, has been open source since 2000 and is used on some Linux distributions.

NOTE:
The Portable Operating System Interface (POSIX) is a family of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems. POSIX defines the application programming interface (API), along with command line shells and utility interfaces, for software compatibility with variants of Unix and other operating systems.

There are several other varians related to the original KornShell, such as:

  • dtksh - a fork of ksh93 included as part of CDE.
  • tksh - a fork of ksh93 that provides access to the Tk widget toolkit.
  • pdksh - the OpenBSD clone of ksh88. It is the default shell in OpenBSD.
  • oksh - a port of pdksh, intended to be maximally portable across operating systems. It was used as the default shell in DeLi Linux 7.2.
  • mksh - the MirBSD Korn Shell, a free implementation of the KornShell language, also forked from pdksh. In addition to its usage on BSD, this variant has replaced pdksh on Debian, and is the default shell on Android. mksh is available on most Linux distributions.

Bash

Bash is a Unix shell and command language written by Brian Fox for the GNU Project as a free software replacement for the Bourne shell. First released in 1989, it has been used as the default login shell for most Linux distributions and all releases of Apple's macOS prior to macOS Catalina. A version is also available for Windows 10 via the Windows Subsystem for Linux. It is also the default user shell in Solaris 11.

The name Bash is an acronym for "Bourne Again Shell", a pun on the name of the Bourne shell that it replaces and the notion of being "born again".

A security hole in Bash dating from version 1.03 (August 1989), dubbed Shellshock, was discovered in 2014 and led to a range of attacks across the Internet. Patches to fix the bugs were made available soon after the bugs were identified. Because of the potential to compromise millions of unpatched systems, Shellshock was compared to the Heartbleed bug in its severity.

Bash is the default interactive command-line shell on most Linux distributions. It is feature rich, actively maintained and generally considered very stable.

Zsh

Zsh is an extended Bourne shell with many features, including features from Bash, ksh, and tcsh. Paul Falstad wrote the first version of Zsh in 1990 while a student at Princeton University.

Zsh includes features such as:

  • Programmable command-line completion that can help the user type both options and arguments for most used commands, with out-of-the-box support for several hundred commands.
  • Sharing of command history among all running shells.
  • Extended file globbing allows file specification without needing to run an external program such as find.
  • Extended variable/array handling.
  • Editing of multi-line commands in a single buffer.
  • Spelling correction and autofill of command names (and optionally arguments, assumed to be file names).
  • Themeable prompts, including the ability to put prompt information on the right side of the screen and have it auto-hide when typing a long command.
  • Loadable modules, providing among other things: full TCP and Unix domain socket controls, an FTP client, and extended math functions.
  • The built-in where command. Works like the which command but shows all locations of the target command in the directories specified in $PATH rather than only the one that will be used.
  • Named directories. This allows the user to set up shortcuts such as ~mydir, which then behave the way ~ and ~user do.

Ash

The Almquist shell (ash) is a lightweight Unix shell originally written by Kenneth Almquist in the late 1980s. Initially a clone of the System V.4 variant of the Bourne shell, it replaced the original Bourne shell in the BSD versions of Unix released in the early 1990s.

DASH

In 1997 Herbert Xu ported ash from NetBSD to Debian Linux. In September 2002, with release 0.4.1, this port was renamed to DASH (Debian Almquist shell). Xu's main priorities are POSIX conformance and slim implementation.

Like its predecessor, DASH implements support for neither Internationalization and localization nor multi-byte character encoding (both required in POSIX). Line editing and history support based on GNU Readline is optional.

Because of its slimness, Ubuntu decided to adopt DASH as the default /bin/sh implementation in 2006. The reason for using DASH is faster shell script execution, especially during boot up of the operating system, compared to previous versions of Debian and Ubuntu that used Bash for this purpose, although Bash is still the default login shell for interactive use. DASH became the default /bin/sh in Ubuntu starting with the 6.10 release in October 2006. DASH replaced ash and became the default /bin/sh in Debian 6 (Squeeze).

A result of the shift is that many shell scripts were found making use of bash-specific functionalities, referred to as "bashisms", without properly declaring it in the shebang line. The problem was first spotted in Ubuntu and the Ubuntu maintainers decided to make all the scripts comply with the POSIX standard. The changes were later upstreamed to Debian, which soon adopted DASH as its default /bin/sh too. As a result, all /bin/sh scripts in Debian and Ubuntu are guaranteed to be POSIX-compliant, save for the extensions merged into DASH for convenience. A similar transition has happened in Slackware Linux, although their version of ash is only partially based on DASH.

DASH is also very popular in embedded Linux systems. DASH version 0.3.8-5 was incorporated into BusyBox, the catch-all executable often employed in this area, and is used in distributions like DSLinux, Alpine Linux, Tiny Core Linux and Linux-based router firmware such as OpenWrt, Tomato and DD-WRT.

fish

fish is a Unix shell that attempts to be more interactive and "user-friendly" than those with a longer history. The design goal of fish is to give the user a rich set of powerful features in a way that is easy to discover, remember, and use. fish has its own syntax that is derived neither from the Bourne shell (ksh, Bash, zsh) nor the C shell (csh, tcsh). Also unlike previous shells, which disable certain features by default to save system resources, fish enables all features by default.

Fish has "search as you type" automatic suggestions based on history and current directory. This is essentially like Bash's Ctrl+r history search, but because it is always on instead of being a separate mode, the user gets continuous feedback while writing the command line, and can select suggestions with the arrow keys, or as in Bash, press Tab for a tab completion instead. Tab-completion is feature-rich, expanding file paths (with wildcards and brace expansion), variables, and many command specific completions. Command-specific completions, including options with descriptions, can to some extent be generated from the commands man pages.

Fish has few syntactic rules, preferring features as commands rather than syntax. This makes features discoverable in terms of commands with options and help texts. Functions can also carry a human readable description. A special help command gives access to all the fish documentation in the user's web browser.

What shell should I use?

Unless you have a need for a specific shell, then I recommend you use the default one on the operating system of your choice as it has normally been well integrated into the system.

If you have begun to use shell scripting and you share your scripts with others, e.g. on GitHub or Codeberg, I recommend that you develop your scripts according to the POSIX standard rather than something specific like Bash. Using POSIX will help ensure that your scripts can run on most systems. But if you only write scripts for yourself, then it doesn't matter. You can use the tool ShellCheck (a shell script static analysis tool) to find bugs or "bashisms" in your scripts.

It is also worth keeping in mind that if you are writing large scripts, or use scripts that use complex control flow logic, then you're much better of rewriting such scripts in a more structured language.

I don't recommend you get into the habit of using command auto completion. Auto completion for filenames and directories is fine, but auto completion for command options can become a bad habit. It's much better to get into the habit of reading the man page for the relevant command and typing it out manually. Not only will it help you memorize the command options better, but sometimes what the command option does isn't obvious and you might end up doing something wrong. Also the documentation might contain important information which you tend to skip if you just use command auto completion without reading the man page.

Oh, and by the way, did you know that most shells has a Vi mode and an Emacs mode?

Choose the shell that gets out of your way and makes you the most productive.

My personal preference is:

  • On OpenBSD: I use the default Ksh for both root and regular user. For the regular user I keep an export HISTFILE="$HOME/.ksh_history" in my ~/.profile because I like to keep a command history.
  • On FreeBSD: I use the default tcsh for root, but install Bash for the regular user. When tcsh gets in the way, which it occasionally does for me, I just type bash and run with Bash during that specific session only.
  • On Linux: I just use Bash.
  • When I do shell scripts I generally only do POSIX sh.

Further reading

Read the whole story
emrox
2 days ago
reply
Nice explanation what is what
Hamburg, Germany
Share this story
Delete

2020 Game

1 Share

Play through all the major events of 2020: the Australia wildfires, Covid-19, the stock market crash, quarantine, the rise of TikTok, the USA elections, etc.

Nicely made little 2D game 🙂

2020 Game →

Read the whole story
emrox
2 days ago
reply
Hamburg, Germany
Share this story
Delete

PETAL – the end-to-end web stack

1 Share

There’s a new stack in town. PETAL. It destructures to

  • Phoenix
  • Elixir
  • Tailwind
  • Alpine
  • LiveView

(PETAL was coined by Patrick Thompson to describe a particular set of tools he was combining with good results. I’m just here to tell you about it.)

So what is it? Well, it helps you build web applications.

The “P”

So we take Phoenix, a web framework with heritage from Rails and the same kind of focus on developer experience and productivity without the kind of magic I’ve understood causes a lot of issues and confusion around Rails. I’ll have to take witness reports on that because I did Python and Django in the before-times. I find Phoenix more powerful and explicit than Django as well.

The “E”

Then we take Elixir. A developer-friendly high-level language in the Functional Programming school. Because of the people involved, Elixir has stylistic heritage from Ruby, but it is very much its own language.

(Changelog’s platform is built with Elixir and José Valim, the creator of Elixir, has been on the podcast 3 times.)

Elixir is built on an extremely well-respected runtime, the BEAM, which traditionally powers Erlang. The BEAM was built to do distributed systems at Ericsson, primarily for telecom use, which meant handling concurrency and maintaining good latency. It does concurrency very well for both multi-core machines as well as clustered distributed systems. It is an enormously capable virtual machine built for providing soft real-time performance. Whatsapp has built with Erlang to great success running an enormous system on a very small crew. With Elixir, we see companies like Discord that are repeating that success.

The “T”

Tailwind CSS is a well-liked CSS framework. I haven’t gone deep with Tailwind so I can’t speak to the power of it. But the JS Party podcast has covered it in the weirdest way possible and Thinking Elixir had a very good conversation around all of PETAL that I’d recommend.

The “A”

Alpine.js is a JavaScript framework by Caleb Porzio who built Livewire for PHP (inspired by Phoenix LiveView of Elixir fame). It is built to work well when you need strictly client-side interactivity and these live server-side rendering solutions just won’t do. It should give you something more convenient than vanilla JavaScript while mostly staying out of the way. Patrick Thompson covered a lot of LiveView + Alpine.js in his ElixirConf talk. Chris McCord (of Phoenix fame) also covers some of this at DockYard.

The “L”

LiveView. I dare say that this is the star of the show. It’s an increasingly important yet optional part of the Phoenix web framework.

It allows extremely efficient real-time interactivity and UI without writing frontend JS for the majority of your use-cases. This idea is not entirely new, but since Elixir/Erlang and the BEAM build on the actor model and have powerful facilities for managing in-memory state, events, and message passing… the actual implementation is absolutely outstanding. It is an immensely gratifying and productive way of building interactivity.

So, what it does is basically server-side rendering (SSR) and shipping updated parts over a WebSocket connection. Chris McCord explains it way better in his keynote at ElixirConf EU. Like most things built by the Elixir and Phoenix teams, it is much better in execution than it has any right to be.

Now, to give some external validation to the idea of LiveView I want to draw on a favored entity of The Changelog. I will invoke Basecamp. Because not only did Caleb Porzio feel inspired to create Livewire because of LiveView. I imagine the recently released Hotwire from Basecamp pulls some of its name from Livewire… suddenly we can trace the lineage. I think DHH has been working along this thinking for a while, so I’m not sure if LiveView had anything to do with its creation but they are similar solutions for the same problems: building an SPA is too much damn work for a lot of use-cases.

Whether you have a beef with modern JavaScript, just find the tooling frustrating, or simply don’t find the time investment makes sense for your project… there are many reasons why people reach for these solutions. With the release of Hotwire, DHH went on Full Stack Radio and spoke at length about it. I thought he did a good job of not saying “I hate JS and never want to write it”. Rather he emphasized that he really likes and enjoys Ruby and wants to focus on and use that language. I think he made a lot of good points in that conversation that applies quite closely to the PETAL stack. If you prefer Rails, that Basecamp stack is probably great.

Now I like and enjoy Elixir. I also happen to think it is technically superior. That isn’t the most important thing in the world, but it makes me happy. I find it gives me more tools than any other stack I’ve ever worked in. It is a fairly common story that Ruby shops rebuild a system that was performing poorly in Ruby with Elixir and suddenly forget it exists because it never caused trouble again and only required a slice of the previous power to perform at the same pace. In his talk on crawling, Adam Mokan covers some of that ground.

Why PETAL?

I think the PETAL stack is very compelling. Why? Because it has opinionated solutions to the common problems of selecting these layers, but doesn’t paint you into a corner.

Phoenix does your core web server work and basic rendering and forms. LiveView brings your system to the client and provides real-time interactivity for the 80-90% case. Alpine is there for the remaining 10-20% of high-touch interactivity. Tailwind CSS gives you the visual and styling baseline and tools you will need. This should give you all the power you need for a web application UI and all the way back to persisting data to a DB.

Personally I find what lies underneath this stack even more compelling.

Elixir is incredible

Hiding in the “E” for Elixir is an incredible set of capabilities that you won’t find in most languages. You get all of Erlang OTP which is a ridiculously powerful standard library of sorts. See, when your project grows the fun really begins. Because you can grow in this ecosystem like nowhere else. Adding your second host? You have the option of clustering them together allowing transparent message exchange between the nodes. One application spread across many hosts. Phoenix, Channels, LiveView and friends are already cool with that. It just works. You can, of course, run a more typical shared nothing if you like as well. Do you need to do background work? No need to offload that to another process. Your application is ready to manage countless parallell processes.

It’s nice to run on a runtime that can do more than one thing at a time.

There are things you can build without ever leaving your application or creating an artificial boundary because of runtime concerns. That would be really hard to argue for in a Python web application, or a Node.js backend. You can skip your Celery or Sidekiq workers for many use-cases. There’s also a very neat library for setting up LetsEncrypt automatically for you called site_encrypt. It works without certbot or a single line of cron. Because it can.

I think this stack has the potential to be quite approachable to new learners while not really setting any limits to future growth. You can go from simple forms and just work in this stack until you’re wielding CRDTs and applying hybrid logical clocks, or doing event-driven architecture and ingesting huge volumes of data. A traditional trade-off in high level dynamic languages such as Python, Node.js or Ruby is that you sacrifice performance for developer convenience and quick iteration. Elixir works at the same high level of abstraction but builds on very different fundamentals. The problem of distribution and concurrency is the primary bottleneck where these languages struggle. They generally attempt to tackle it with cooperative multitasking, which requires threading a needle of what kind of work can be done without delaying the time to yielding or you will impact other parts of the system. You can handle many requests in a concurrent non-parallel fashion. You run more instances for parallelism.

With Elixir and the BEAM you get another paradigm which uses preemptive scheduling (enabled by Functional Programming and the Actor model) to run things concurrently and in parallel. It has a scheduling system that prevents a heavy piece of work from holding up the line and as such protects system latency so you don’t have to be laser-focused on hitting that IO spot where you can yield. It scales across CPU cores by default.

Elixir won’t beat C++ for performance in number crunching, just like other high-level languages won’t. But it is very well suited to long-running applications, concurrency, state management and distributed computing. And that’s what we do as backend web developers.

Erlang has been doing this for quite some time already. Elixir makes it more approachable and brings better tooling. Phoenix gives all the web conveniences. And now, forming around LiveView, in line with the trend of server-side rendering returning, we can see the shape of a systematic approach to the final parts, all the way into the CSS and last mile of JS interactivity.

Yeah, I think PETAL might just be the stack to watch.


From the Editor

Lars Wikman is a big time contributor to Changelog’s codebase, news feed, and developer community. You can (and should) read more from Lars on underjord.io.

Discuss on Changelog News

Read the whole story
emrox
3 days ago
reply
Hamburg, Germany
Share this story
Delete

Progressive Web Apps in 2021

1 Share

The Progressive Web App term is now five years old, and it’s time to sit down and understand where we are at 2021 within the platform, what has changed during 2020 and what we are expecting for the upcoming months.

It shouldn’t surprise you while reading that iOS is the bottleneck here …

Progressive Web Apps in 2021 →

Read the whole story
emrox
4 days ago
reply
Hamburg, Germany
Share this story
Delete

A brief history of Responsive Web Design

1 Share

Abhishek Chaudhary:

Websites are like a canvas. You have complete freedom to design them the way you want. But unlike a painting, not all people will view your site the way you want. This article discusses how Responsive Web Design (RWD) evolved.

Discuss on Changelog News

Read the whole story
emrox
4 days ago
reply
Hamburg, Germany
Share this story
Delete

containerd development with Linux and multipass

1 Share

About 18 months ago I started a project which had to develop directly against containerd with a full Linux system.

This presented a problem which I’d not really encountered before - Docker and Kubernetes on my Mac were no longer enough, I needed a full Linux environment, and so did the community.

This is how it went and what we learned along the way.

Discuss on Changelog News

Read the whole story
emrox
4 days ago
reply
Hamburg, Germany
Share this story
Delete
Next Page of Stories