Markdown Shell Scripting.md

Shell scripting with Markdown documents

alias ~~~=":<<'~~~sh'";:<<'~~~sh'

This Markdown files is also an executable Bourne Shell script. Here’s the shell command that it will run:

vim -M "$0"

Which will view this file in vi.

❗️ If you don’t know vi don’t panic. Just type :q and press return. Without the back-ticks.

1. type a colon
2. then type q
3. then press return

Then you can continue reading here. It it fails, your probably were still holding down the shift key when you typed the q. It must be lowercase. Try again…

The key is the first two lines of the file.

# Shell scripting with Markdown documents

	alias ~~~=":<<'~~~sh'";:<<'~~~sh'

![][xkcd-2054]

[xkcd-2054]:  https://imgs.xkcd.com/comics/data_pipeline_2x.png

The first line, the file header, is treated as a comment by the shell. If fact headers are the only Markdown that can come before the second line.

The alias ~~~=":<<'~~~sh'";:<<'~~~sh' is the key to the whole file.¹ It causes anything encased in a code block like this

~~~sh
# code goes here…
~~~

to run as part of the shell script; and everything else to be ignored.

Note that the code block must look exactly like that (three tildes; not backticks) to work. Using a different number of tildes or a different code block descriptor (like bash) requires changing the alias statement accordingly and is left as an exercise for the reader.

echo That’s it! Thanks…

How does it work?

Looking at each part of the command:

alias ~~~=":<<'~~~sh'";:<<'~~~sh'

• : is a built-in shell command that ignores all of its arguments (and standard input) and simply returns true.
• << indicates a Here document — passing all of the lines that follow it into the standard input of that command until it encounters its delimiting identifier on its own line.
• ’~~~sh’ is the delimiting identifier being used to indicate the end of the current Here document. Specifically, the identifier is ~~~sh. Surrounding it with single quotes ensures that the shell does no substitution or command execution on the Here document content.
• alias sets this sequence to be executed whenever the shell see ~~~.

The result is that when the shell encounters the expression ~~~ it begins discarding every line that follows until it encounters a fenced code block that begins with ~~~sh. It then resumes execution with the content of that code block.

• ; ends the alias command so that we may then execute the next one.
• :<<'~~~sh' is the exact same as the alias we just set. But, because shells execute their scripts line-by-line, the alias won’t take effect until after this line. To work around this limitation we just spell out the command to skip to the first code block that we wish to run.

Any code block that you do not want executed can simply use a different code-block style. For example, when discussing the alias command, above, it was written as

## How does it work?

Looking at each part of the command:

```sh
alias ~~~=":<<'~~~sh'";:<<'~~~sh'
```

to ensure that it wasn’t mistaken for part of the script.

Running the script

If you have done a chmod +x on the file then it can be run like any other command. Although doing so might do strange things to file’s icon in your file browser.

Normally these script are run by prefixing them with the shell name to run them in²

sh 'Markdown Shell Scripting.md'

Note that using the bash source command will not work as the file must be interpreted as a sh script. Not a bash script. Even if the sh executable on your system is just a symlink to bash it makes a difference.

Or you could just add a

#! /usr/bin/env sh

at the top..

Shebang lines

Many people will want to include a #! (shebang) line at the top, but it is not strictly necessary.

Some people may wish to point out that a file without a shebang line is not technically a valid executable and is therefore not portable. Not only are they missing the point of how frickin' awesome this is, but they are also technically incorrect. (The best kind of incorrect!)

You see, POSIX 2008.1-compliant shells are required to treat shebang​less text files as shell scripts — and you wouldn't want to use a non-POSIX-compliant shell, would you? (Even busybox is compliant enough!) For that matter, since POSIX.2008-1, the "treat other files as a shell script" behavior applies to execvp() and execlp() as well.

So the only time a non-shebang file is in any danger of failing execution is when it's being executed in a non-POSIX environment or by a program that doesn't use those functions to do the exec. (And in such cases, you can simply prefix your command string with env or sh or whatever!) So don't let this detail get in the way of how awesome it can be to mix markdown and shell scripting.

(You can also just use a shebang line, if it bothers you that much or yours or your users' environment sucks that hard and you don't mind your markdown file having a huge heading saying !/usr/bin/env or some such. The point is, think about your use cases instead of just blindly opting for one path or another.)

— Mixed Markdown and Shell Scripting

Also note that when posting Markdown files as a Github gist, if the file begins with a #! it will be rendered as a shell script. Regardless of the .md extension in the file name.

Note that the source of the quote above has a slightly more complicated (but more flexible?) version of this technique.

Notes

1. Putting an xkcd comic at the top of your script is not required, but highly recommended.³
2. Apologies for any time lost trying to find the perfect comic.
3. And for any time lost reading through explanations of various xkcd comics.
4. At least it wasn’t a TV Tropes link…
5. Apologies for any time lost browsing TV Tropes.

Footnotes

1. It does not need to be indented, but doing so increases it’s readability as Markdown. ↩

2. I like to use the dash shell when developing the scripts just to ensure that no “bash-isms” creep into my code. Even when running in compatibility mode, certain extensions are allowed by bash. If that happens, your script will seem to be fine until one day it is run on a system that does not use a symlink to bash as its sh ↩

3. xkcd comics are licensed under a Creative Commons Attribution-NonCommercial 2.5 License. ↩

best markdown tutorial ever

very helpful

Thanks. I wanted to document the techniques in a single, easily sharable, place.

I change ~~~sh to ~~~bash. But it does not work.

/usr/local/bin/removeDuplicates.md: line 45: ~~~: command not found

As long as your change all of the instances of ~~~sh to ~~~bash it will work. The alias needs to match the code blocks.

Good

Wow. This is incredible. Literate programming in posix shell. This should be a standard for all sysadmins lmao.

I like this a lot, but I can't get it to work, I get ~~~: command not found errors (in Bash or in Debian's Dash shells).

Apparently aliases are not expanded in non-interactive situations. The bash documentation explicitly states that shopt expand_aliases should be used for that (but it is a bash-ism), the POSIX documentation is unclear on the topic, other documentations explicitly rule out alias substitution in scripts.

In what environment are you getting this to work?

Interesting. I use dash to test my scripts in macOS, but run them directly (with sh in Linux). Are you calling it correctly?

From above:

Note that using the bash source command will not work as the file must be interpreted as a sh script. Not a bash script. Even if the sh executable on your system is just a symlink to bash it makes a difference.

Indeed somehow I did overlook this detail, sorry. I made my markdown/script executable and since bash is my default shell, it failed.

I actually found a way around to make the script work in bash too, but it is arguably less elegant since it uses two lines after the title :

command -v shopt >/dev/null && shopt -s expand_aliases
alias ~~~=":<<'~~~sh'";:<<'~~~sh'

Another detail : bash also needs a line ~~~sh at the end of the file, otherwise it complains with

here-document at line X delimited by end-of-file (wanted `~~~sh')

whereas dash silently accepts the here-document to end with the file. POSIX states that in this situation "the shell should, but need not, treat this as a redirection error" (dash does not). The downside of putting this at the end is that the markdown document has an unterminated code block, which may produce garbage at the end when formatting.

Or you could just add a

#! /usr/bin/env sh

at the top..