Balthazarhttps://blog.balthazar-rouberol.com/2023-08-30T00:00:00+02:00Just enough Makefile to be dangerous2023-08-30T00:00:00+02:002023-08-30T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2023-08-30:/just-enough-makefile-to-be-dangerous<p>Over the years, I've developed a mix of appreciation and frustration for <code>make</code>. While it's conveniently ubiquitous across UNIX systems and widely used, its syntax often feels perplexing and unwieldy, posing debugging challenges. In this article, I share best practices I've embraced to make working with <code>make</code> a more satisfying experience.</p><div class="toc"><span class="toctitle">Table of Contents</span><ul>
<li><a href="#getting-started-with-make">Getting started with make</a><ul>
<li><a href="#the-step-structure">The step structure</a></li>
<li><a href="#phony-targets">Phony targets</a></li>
<li><a href="#default-target">Default target</a></li>
</ul>
</li>
<li><a href="#my-best-practices">My best practices</a><ul>
<li><a href="#makefile-auto-documentation-as-the-default-step">Makefile auto-documentation as the default step</a></li>
<li><a href="#tell-whats-happening-not-how">Tell what's happening, not how</a></li>
<li><a href="#define-commonalities-in-variables">Define commonalities in variables</a></li>
<li><a href="#keep-all-paths-in-the-makefile">Keep all paths in the Makefile</a></li>
<li><a href="#generate-a-visual-representation-of-the-makefile">Generate a visual representation of the Makefile</a></li>
<li><a href="#keep-things-readable">Keep things readable</a></li>
</ul>
</li>
</ul>
</div>
<p>Over the years, I have developed a bit of a love-hate relationship with <code>make</code>. On the plus side, it is ubiquitous, preinstalled on most UNIX systems, and widely used. On the other hand, its syntax can feel arcane and clunky, and it can prove hard to debug.
In this article, I will go over the basic <code>make</code> concepts, and the set of best practices I've come to embrace as my own, to make <code>make</code> enjoyable to use.</p>
<p>Let's start with the beginning.</p>
<h2 id="getting-started-with-make">Getting started with <code>make</code></h2>
<h3 id="the-step-structure">The step structure</h3>
<p><code>make</code> is a build system: a piece of tooling allowing you to define steps to build your project. It should make sure to only rebuild what needs to be rebuilt, to keep build time as short as possible. All these steps are defined in a file named <code>Makefile</code>, usually located at the root of your project.</p>
<p>A <code>make</code> step has the following syntax:</p>
<div class="highlight"><pre><span></span><code><span class="nf">target</span><span class="o">:</span><span class="w"> </span>[<span class="n">space</span> <span class="n">separated</span> <span class="n">dependencies</span>]
<span class="w"> </span>shell<span class="w"> </span>instructions
<span class="w"> </span>...
</code></pre></div>
<p>By default, <code>make</code> assumes that a target is a <em>file</em>, and will build it by executing the shell instructions associated with that target, after it has executed the shell instructions associated with the possible target dependencies (if any).</p>
<p>Let's have a look at a simple example in which we will build this <code>hello.c</code> file into a <code>hello</code> binary, using the <code>gcc</code> compiler.</p>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"hello world</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>We define the following <code>Makefile</code>:</p>
<div class="highlight"><pre><span></span><code><span class="nf">hello</span><span class="o">:</span><span class="w"> </span><span class="n">hello</span>.<span class="n">c</span>
<span class="w"> </span>gcc<span class="w"> </span>hello.c<span class="w"> </span>-o<span class="w"> </span>hello
</code></pre></div>
<p>We can then run <code>make hello</code> to compile the <code>hello</code> binary, after which we run it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>make<span class="w"> </span>hello
gcc<span class="w"> </span>hello.c<span class="w"> </span>-o<span class="w"> </span>hello
$<span class="w"> </span>./hello
hello<span class="w"> </span>world
</code></pre></div>
<p>When we ran <code>make hello</code>, <code>make</code> detected that the <code>hello</code> file wasn't found on disk, and built it by running <code>gcc hello.c -o hello</code>.</p>
<p>What happens if we re-run the same command now?</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>make<span class="w"> </span>hello
make:<span class="w"> </span><span class="sb">`</span>hello<span class="err">'</span><span class="w"> </span>is<span class="w"> </span>up<span class="w"> </span>to<span class="w"> </span>date.
</code></pre></div>
<p><code>make</code> detected that <code>hello.c</code> hadn't changed since last time <code>hello</code> was built, and thus did nothing. If we change <code>hello.c</code> to print <code>hello bobbytables</code> instead of <code>hello world</code>, <code>make</code> will see that the file had changed and will happily rebuild the binary:</p>
<div class="highlight"><pre><span></span><code>#include <stdio.h>
int main() {
<span class="gd">- printf("hello world\n");</span>
<span class="gi">+ printf("hello bobbytables\n");</span>
<span class="w"> </span> return 0;
}
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>make<span class="w"> </span>hello
gcc<span class="w"> </span>hello.c<span class="w"> </span>-o<span class="w"> </span>hello
$<span class="w"> </span>./hello
hello<span class="w"> </span>bobbytables
</code></pre></div>
<h3 id="phony-targets">Phony targets</h3>
<p>Say now that you'd like to define a <code>run</code> step, that will simply run the binary:</p>
<div class="highlight"><pre><span></span><code><span class="nf">hello</span><span class="o">:</span>
<span class="w"> </span>gcc<span class="w"> </span>-o<span class="w"> </span>hello<span class="w"> </span>hello.c
<span class="nf">run</span><span class="o">:</span><span class="w"> </span><span class="n">hello</span>
<span class="w"> </span>./hello
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>make<span class="w"> </span>run
./hello
hello<span class="w"> </span>world
</code></pre></div>
<p>The issue here, is that <code>run</code> does not represent a file on disk. To avoid confusing <code>make</code>, we mark this step as being <code>PHONY</code>, aka not a file <code>make</code> needs to build. This will make sure the associated shell instructions are always executed.</p>
<div class="highlight"><pre><span></span><code><span class="nf">hello</span><span class="o">:</span>
<span class="w"> </span>gcc<span class="w"> </span>-o<span class="w"> </span>hello<span class="w"> </span>hello.c
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">run</span>
<span class="nf">run</span><span class="o">:</span><span class="w"> </span><span class="n">hello</span>
<span class="w"> </span>./hello
</code></pre></div>
<h3 id="default-target">Default target</h3>
<p>We can define what step should be run when invocating <code>make</code> without any argument by using <code>.DEFAULT_GOAL</code>:</p>
<div class="highlight"><pre><span></span><code><span class="nv">.DEFAULT_GOAL</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>run
<span class="nf">hello</span><span class="o">:</span>
<span class="w"> </span>gcc<span class="w"> </span>-o<span class="w"> </span>hello<span class="w"> </span>hello.c
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">run</span>
<span class="nf">run</span><span class="o">:</span><span class="w"> </span><span class="n">hello</span>
<span class="w"> </span>./hello
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>make
./hello
hello<span class="w"> </span>world
</code></pre></div>
<div class="Note">
<p>We can hide the command being executed by prefixing it with <code>@</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">.DEFAULT_GOAL</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>run
<span class="nf">hello</span><span class="o">:</span>
<span class="w"> </span>gcc<span class="w"> </span>-o<span class="w"> </span>hello<span class="w"> </span>hello.c
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">run</span>
<span class="nf">run</span><span class="o">:</span><span class="w"> </span><span class="n">hello</span>
<span class="w"> </span>@./hello
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>make
hello<span class="w"> </span>world
</code></pre></div>
</div>
<p>And with that, we now know just enough to get started for real.</p>
<h2 id="my-best-practices">My best practices</h2>
<div class="Note">
<p>The examples are taken from the <a href="https://github.com/brouberol/5esheets"><code>5esheets</code></a> <a href="https://github.com/brouberol/5esheets/blob/main/Makefile">Makefile</a>.</p>
</div>
<h3 id="makefile-auto-documentation-as-the-default-step">Makefile auto-documentation as the default step</h3>
<p>Ever since I stumbled on this <a href="https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html">article</a>, I have made sure to auto-document all my <code>Makefile</code>s, to help with discoverability. This works by adding a one-liner explanation of the "public" targets (the one a contributor might find themselves executing) after a <code>##</code>. We then define a <code>help</code> target that will parse the current <code>Makefile</code>, extract all the target names and associated comments, and format them nicely. The finishing touch is to make <code>help</code> the default target, to make it extra easy for a newcomer to understand what can be built with your <code>Makefile</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">.DEFAULT_GOAL</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">help</span>
<span class="err">...</span>
<span class="nf">run</span><span class="o">:</span><span class="w"> </span><span class="n">admin</span>-<span class="n">statics</span> <span class="n">build</span> <span class="c">## Run the app</span>
<span class="w"> </span>...
<span class="nf">help</span><span class="o">:</span><span class="w"> </span><span class="c">## Display help</span>
<span class="w"> </span>@grep<span class="w"> </span>-E<span class="w"> </span><span class="s1">'^[a-zA-Z_-]+:.*?## .*$$'</span><span class="w"> </span><span class="k">$(</span>MAKEFILE_LIST<span class="k">)</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sort<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'</span>
</code></pre></div>
<p>This is what the output looks like for the <code>5esheets</code> project:</p>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/makefiles/autodoc.webp"></p>
<h3 id="tell-whats-happening-not-how">Tell what's happening, not how</h3>
<p>I personally like to have each step include a short explanation of what it is doing, and hide the actual shell command, which I find of low value.</p>
<div class="highlight"><pre><span></span><code><span class="nf">deps-python</span><span class="o">:</span><span class="w"> </span><span class="n">poetry</span>.<span class="n">lock</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"\n[+] Installing python dependencies"</span>
<span class="w"> </span>@poetry<span class="w"> </span>install
</code></pre></div>
<p>In that example, when the target executes, I see <code>[+] Installing python dependencies</code>, as well at the command output, but not the <code>poetry install</code> command itself. I find that communicating the <em>intent</em> is clearer and more self-explanatory than taking screen real-estate by displaying the nitty-gritty details.</p>
<h3 id="define-commonalities-in-variables">Define commonalities in variables</h3>
<p>When I find myself repeating things too much in various rules, this is when I start using variables. For example, instead of writing many rules that hardcode a given directory name in them, I define that directory name in a variable. This makes it easier to keep the <code>Makefile</code> valid when the project structure evolves.</p>
<div class="highlight"><pre><span></span><code><span class="nv">app-root</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>dnd5esheets
<span class="nf">black</span><span class="o">:</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"\n[+] Reformatting python files"</span>
<span class="w"> </span>@poetry<span class="w"> </span>run<span class="w"> </span>black<span class="w"> </span>--check<span class="w"> </span><span class="k">$(</span>app-root<span class="k">)</span>/
<span class="nf">mypy</span><span class="o">:</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"\n[+] Checking Python types"</span>
<span class="w"> </span>@poetry<span class="w"> </span>run<span class="w"> </span>mypy<span class="w"> </span><span class="k">$(</span>app-root<span class="k">)</span>/
<span class="nf">ruff</span><span class="o">:</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"\n[+] Running linter"</span>
<span class="w"> </span>@poetry<span class="w"> </span>run<span class="w"> </span>ruff<span class="w"> </span><span class="k">$(</span>app-root<span class="k">)</span>/
</code></pre></div>
<h3 id="keep-all-paths-in-the-makefile">Keep all paths in the Makefile</h3>
<p>Some of my targets are oftentimes generated via scripts (usually python), which process some input and dump their result to a target file. I find that passing the output file path to the script (instead of hardcoding the file path in the script) allows the <code>Makefile</code> to be more self-contained and makes it easier to rename files without having to update both the <code>Makefile</code> <em>and</em> the script.</p>
<div class="highlight"><pre><span></span><code><span class="nf">$(data-dir)/translations-items-fr.json</span><span class="o">:</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"\n[+] Fetching items french translations"</span>
<span class="w"> </span>@curl<span class="w"> </span>-s<span class="w"> </span><span class="k">$(</span>fr-translations-data-dir<span class="k">)</span>/dnd5e.items.json<span class="w"> </span>><span class="w"> </span><span class="k">$(</span>data-dir<span class="k">)</span>/translations-items-fr.json
<span class="nf">$(data-dir)/items-base.json</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">data-dir</span><span class="k">)</span>/<span class="n">translations</span>-<span class="n">items</span>-<span class="n">fr</span>.<span class="n">json</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"\n[+] Fetching base equipment data"</span>
<span class="w"> </span>@curl<span class="w"> </span>-s<span class="w"> </span><span class="k">$(</span>5etools-data-dir<span class="k">)</span>/items-base.json<span class="w"> </span><span class="p">|</span><span class="w"> </span>./scripts/preprocess_base_item_json.py<span class="w"> </span><span class="k">$(</span>data-dir<span class="k">)</span>/items-base.json
</code></pre></div>
<p>We can then avoid repeating ourselves by leveraging the <code>$@</code> symbol, which expands to the name of the target being generated.</p>
<div class="highlight"><pre><span></span><code><span class="nf">$(data-dir)/translations-items-fr.json</span><span class="o">:</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"\n[+] Fetching items french translations"</span>
<span class="w"> </span>@curl<span class="w"> </span>-s<span class="w"> </span><span class="k">$(</span>fr-translations-data-dir<span class="k">)</span>/dnd5e.items.json<span class="w"> </span>><span class="w"> </span><span class="nv">$@</span>
<span class="nf">$(data-dir)/items-base.json</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">data-dir</span><span class="k">)</span>/<span class="n">translations</span>-<span class="n">items</span>-<span class="n">fr</span>.<span class="n">json</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"\n[+] Fetching base equipment data"</span>
<span class="w"> </span>@curl<span class="w"> </span>-s<span class="w"> </span><span class="k">$(</span>5etools-data-dir<span class="k">)</span>/items-base.json<span class="w"> </span><span class="p">|</span><span class="w"> </span>./scripts/preprocess_base_item_json.py<span class="w"> </span><span class="nv">$@</span>
</code></pre></div>
<h3 id="generate-a-visual-representation-of-the-makefile">Generate a visual representation of the Makefile</h3>
<p>I like having a visual representation of the dependencies of each target. It allows me to debug why some targets are not being rebuilt when they should, or are always being rebuilt when they shouldn't be. I find that it it also helps when getting started with the project for the first time. I leverage the <a href="https://pypi.org/project/makefile2dot/"><code>makefile2dot</code></a> Python package for this:</p>
<div class="highlight"><pre><span></span><code><span class="nf">doc/makefile.png</span><span class="o">:</span><span class="w"> </span><span class="n">Makefile</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"\n[+] Generating a visual graph representation of the Makefile"</span>
<span class="w"> </span>@poetry<span class="w"> </span>run<span class="w"> </span>makefile2dot<span class="w"> </span>-o<span class="w"> </span><span class="nv">$@</span>
</code></pre></div>
<div class="Note">
<p>You'll notice that this target depends on the <code>Makefile</code> itself, as it needs to be re-generated as the <code>Makefile</code> evolves.</p>
</div>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/makefiles/makefile.webp"></p>
<h3 id="keep-things-readable">Keep things readable</h3>
<p>This is probably my most fundamental best practice.</p>
<p>Over the years, I have realized that I'm not smart enough to maintain a cryptic-looking <code>Makefile</code>. I my view, articles such as <a href="https://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/">this one</a> steer the reader into producing "smart" Makefiles that are non obvious to reason about (especially the last example). I need to be able to read a target's logic and understand what it does months after having written it. The same way, I won't hesitate to repeat myself and avoid variables when I think the output looks clearer. I try not to use <a href="https://devhints.io/makefile">"magic variables"</a> too much.</p>
<p>There's a delicate balance to be struck between expressibility and readability, and I think readability should <em>always</em> win. You'll thank yourself later.</p>Pinning your SQLite version across environments2023-08-25T00:00:00+02:002023-08-25T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2023-08-25:/pinning-your-sqlite-version-across-environments<p>This article discusses the challenges of maintaining consistent versions of the SQLite library in different environments for a project that relies heavily on it. Unlike traditional databases, where server versions can be easily pinned, SQLite is embedded in applications, leading to potential feature mismatches due to what version is made available by each environment's system package manager.</p><p>The <a href="https://github.com/brouberol/5esheets">project</a> I'm currrently working on only has a single external dependency: <a href="https://www.sqlite.org/">SQLite</a>, with <a href="https://www.sqlite.org/fts5.html">full text search</a> enabled. As a result, the application is extremely easy to package and run. However, I found out that ensuring that you have the <em>exact same</em> SQLite <a href="https://github.com/brouberol/5esheets/pull/207#issuecomment-1672131123">version and feature</a> set in all your environments (development machines running macOS and linux, CI and production) is trickier than I expected.</p>
<p>When you rely on a traditional database server (PostgreSQL, MySQL, mongoDB, etc), you can achieve this by running the same server version in all your environments.</p>
<div class="Note">
<p>Docker really shines there, as it allows to do just that in a single command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>postgres:15.4
</code></pre></div>
</div>
<p>Things are a bit different with SQLite, as it is <em>not</em> an SQL server. It is a <em>library</em> that you embed in your program (either by compiling it alongside your code, or by relying on a shared library and language bindings). Python does the latter: its <code>sqlite3</code> module is written in C using the CPython API, and <a href="https://github.com/python/cpython/blob/4ae3edf3008b70e20663143553a736d80ff3a501/Modules/_sqlite/connection.h#L32">includes</a> the <code>sqlite3.h</code> header file. Where does this header file come from though?</p>
<h3 id="inspecting-the-sqlite-version-on-linux">Inspecting the sqlite version on linux</h3>
<p>If we have a look at a <code>python3.11</code> installation directory on a random Ubuntu server, we see that it bundles an <code>_sqlite.so</code> shared object, that itself dynamically loads <code>libsqlite3.so.0</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">find</span><span class="w"> </span>/usr/lib/python3.11<span class="w"> </span>-name<span class="w"> </span><span class="s2">"*sqlite3*.so"</span>
/usr/lib/python3.11/lib-dynload/_sqlite3.cpython-311-x86_64-linux-gnu.so
$<span class="w"> </span><span class="nb">ldd</span><span class="w"> </span>/usr/lib/python3.11/lib-dynload/_sqlite3.cpython-311-x86_64-linux-gnu.so
<span class="w"> </span>linux-vdso.so.1<span class="w"> </span><span class="o">(</span>0x00007ffcda976000<span class="o">)</span>
<span class="w"> </span>libsqlite3.so.0<span class="w"> </span><span class="o">=</span><span class="k">></span><span class="w"> </span>/lib/x86_64-linux-gnu/libsqlite3.so.0<span class="w"> </span><span class="o">(</span>0x00007fab44d9c000<span class="o">)</span><span class="w"> </span><span class="c1"># <--</span>
<span class="w"> </span>libc.so.6<span class="w"> </span><span class="o">=</span><span class="k">></span><span class="w"> </span>/lib/x86_64-linux-gnu/libc.so.6<span class="w"> </span><span class="o">(</span>0x00007fab44a00000<span class="o">)</span>
<span class="w"> </span>libm.so.6<span class="w"> </span><span class="o">=</span><span class="k">></span><span class="w"> </span>/lib/x86_64-linux-gnu/libm.so.6<span class="w"> </span><span class="o">(</span>0x00007fab44cb3000<span class="o">)</span>
<span class="w"> </span>/lib64/ld-linux-x86-64.so.2<span class="w"> </span><span class="o">(</span>0x00007fab44f17000<span class="o">)</span>
</code></pre></div>
<p>Same question: where does <code>/lib/x86_64-linux-gnu/libsqlite3.so.0</code> come from then?</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">apt-file</span><span class="w"> </span>search<span class="w"> </span>/lib/x86_64-linux-gnu/libsqlite3.so.0
libsqlite3-0:<span class="w"> </span>/usr/lib/x86_64-linux-gnu/libsqlite3.so.0
libsqlite3-0:<span class="w"> </span>/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
$<span class="w"> </span><span class="nb">apt-cache</span><span class="w"> </span>search<span class="w"> </span>libsqlite3-0
libsqlite3-0<span class="w"> </span>-<span class="w"> </span>SQLite<span class="w"> </span><span class="m">3</span><span class="w"> </span>shared<span class="w"> </span>library
</code></pre></div>
<p>This means that python relies on whatever <code>libsqlite3</code> version is installed by the <em>system package manager</em>. We can double check this by having a look at the <code>python3</code> package recursive dependencies: <a href="https://packages.ubuntu.com/lunar/python3"><code>python3</code></a> -> <a href="https://packages.ubuntu.com/lunar/libpython3-stdlib"><code>libpython3-stdlib</code></a> -> <a href="https://packages.ubuntu.com/lunar/libpython3.11-stdlib"><code>libpython3.11-stdlib</code></a> -> <a href="https://packages.ubuntu.com/lunar/libsqlite3-0"><code>libsqlite3-0</code></a>.</p>
<p>To know what version is installed on that system, we can inspect the version of the <code>libsqlite3-0</code> apt package:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">apt-cache</span><span class="w"> </span>show<span class="w"> </span>libsqlite3-0<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>Version
Version:<span class="w"> </span><span class="m">3</span>.40.1-1
</code></pre></div>
<p>We can check that we're getting this exact version via python:</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">sqlite3</span>
<span class="o">>>></span> <span class="n">conn</span> <span class="o">=</span> <span class="n">sqlite3</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">":memory:"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">"select sqlite_version()"</span><span class="p">)</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
<span class="p">(</span><span class="s1">'3.40.1'</span><span class="p">,)</span>
</code></pre></div>
<h3 id="inspecting-the-sqlite-version-on-macos">Inspecting the sqlite version on macOS</h3>
<p>Assuming you are installing your packages via <code>brew</code> on macOS, you'll find that it does things a bit differently than <code>apt</code>. The <code>python3</code> formula <a href="https://github.com/Homebrew/homebrew-core/blob/1aa36b1d93b4ee968d8d355640735f5ec21e7262/Formula/p/python@3.11.rb#L30">depends on <code>sqlite</code></a>, which itself <a href="https://github.com/Homebrew/homebrew-core/blob/1aa36b1d93b4ee968d8d355640735f5ec21e7262/Formula/s/sqlite.rb#L4">downloads</a> an archive pinned to a given version (<code>3.43.0</code> at the time of writing), and then <a href="https://github.com/Homebrew/homebrew-core/blob/1aa36b1d93b4ee968d8d355640735f5ec21e7262/Formula/s/sqlite.rb#L36-L56">compiles <code>libsqlite3.dylib</code></a>.</p>
<p>Indeed, we see this library when inspecting the content of the <code>sqlite</code> brew package:</p>
<div class="highlight"><pre><span></span><code><span class="k">~</span><span class="w"> </span>❯<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-alh<span class="w"> </span>/opt/homebrew/opt/sqlite/lib/libsqlite3.dylib
lrwxr-xr-x<span class="w"> </span><span class="m">18</span><span class="w"> </span>br<span class="w"> </span><span class="m">16</span><span class="w"> </span>May<span class="w"> </span><span class="m">15</span>:45<span class="w"> </span>/opt/homebrew/opt/sqlite/lib/libsqlite3.dylib<span class="w"> </span>-><span class="w"> </span>libsqlite3.0.dylib
</code></pre></div>
<p>And sure enough, we see that we're running the expected version in python:</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">sqlite3</span>
<span class="o">>>></span> <span class="n">conn</span> <span class="o">=</span> <span class="n">sqlite3</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">":memory:"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">"select sqlite_version()"</span><span class="p">)</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
<span class="p">(</span><span class="s1">'3.43.0'</span><span class="p">,)</span>
</code></pre></div>
<h3 id="pinning-the-sqlite-version-by-vendoring-the-compiled-library">Pinning the sqlite version by vendoring the compiled library</h3>
<p>To pin the <code>sqlite</code> version across all environments and OSes, we can compile these shared/dynamically loaded libraries ourselves for all architectures we plan to support, vendor them in our codebase, and inject them into our application via <code>LD_PRELOAD</code>.</p>
<p>We'd need to cover all the ways we run the app:</p>
<ul>
<li>running <code>make run</code>, which runs the app on the host, against the version of <code>libsqlite3</code> installed by the package manager</li>
<li>running <code>make docker-run</code>, which runs the application in a docker container against the <code>libsqlite3</code> version available through the image OS package manager</li>
<li>running <code>make test</code> in CI (Github Actions), which runs the test against the <code>libsqlite3</code> version made available by the runner OS package manager</li>
</ul>
<p>Compiling the sqlite source code into a shared library was made easy to do as <a href="https://simonwillison.net/">Simon Willison</a> already <a href="https://til.simonwillison.net/sqlite/sqlite-version-macos-python">documented</a> the process.</p>
<h4 id="compiling-libsqlite3-for-linux">Compiling <code>libsqlite3</code> for linux</h4>
<p>The following script compiles <code>libsqlite3</code> for linux, with full text search enabled:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># script/compile-libsqlite-linux.sh</span>
<span class="c1">#!/usr/bin/env bash</span>
<span class="nb">set</span><span class="w"> </span>-e
apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>build-essential<span class="w"> </span>wget<span class="w"> </span>tcl
<span class="c1"># link associated with sqlite 3.42.0, found on https://www.sqlite.org/src/timeline?t=version-3.42.0</span>
<span class="c1"># pointing to https://www.sqlite.org/src/info/831d0fb2836b71c9</span>
<span class="nv">sqlite_ref</span><span class="o">=</span>831d0fb2
wget<span class="w"> </span>https://www.sqlite.org/src/tarball/<span class="si">${</span><span class="nv">sqlite_ref</span><span class="si">}</span>/SQLite-<span class="si">${</span><span class="nv">sqlite_ref</span><span class="si">}</span>.tar.gz
tar<span class="w"> </span>-xzvf<span class="w"> </span>SQLite-<span class="si">${</span><span class="nv">sqlite_ref</span><span class="si">}</span>.tar.gz
<span class="nb">pushd</span><span class="w"> </span>SQLite-<span class="si">${</span><span class="nv">sqlite_ref</span><span class="si">}</span>
<span class="nv">CPPFLAGS</span><span class="o">=</span><span class="s2">"-DSQLITE_ENABLE_FTS5"</span><span class="w"> </span>./configure
make
<span class="nb">popd</span>
<span class="nb">mv</span><span class="w"> </span>SQLite-<span class="si">${</span><span class="nv">sqlite_ref</span><span class="si">}</span>/.libs/libsqlite3.so<span class="w"> </span>./lib/
<span class="nb">rm</span><span class="w"> </span>-r<span class="w"> </span>SQLite-<span class="si">${</span><span class="nv">sqlite_ref</span><span class="si">}</span>.tar.gz<span class="w"> </span>SQLite-<span class="si">${</span><span class="nv">sqlite_ref</span><span class="si">}</span>
</code></pre></div>
<h4 id="compiling-libsqlite3-for-macos">Compiling <code>libsqlite3</code> for macOS</h4>
<p>The following script compiles <code>libsqlite3</code> for macOS, with full text search enabled:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># script/compile-libsqlite-macos.sh</span>
<span class="c1">#!/usr/bin/env bash</span>
<span class="nb">set</span><span class="w"> </span>-eu
<span class="nv">sqlite_version</span><span class="o">=</span><span class="m">3420000</span>
wget<span class="w"> </span>https://www.sqlite.org/2023/sqlite-amalgamation-<span class="si">${</span><span class="nv">sqlite_version</span><span class="si">}</span>.zip
unzip<span class="w"> </span>sqlite-amalgamation-<span class="si">${</span><span class="nv">sqlite_version</span><span class="si">}</span>.zip
<span class="nb">pushd</span><span class="w"> </span>sqlite-amalgamation-<span class="si">${</span><span class="nv">sqlite_version</span><span class="si">}</span>
gcc<span class="w"> </span>-dynamiclib<span class="w"> </span>sqlite3.c<span class="w"> </span>-o<span class="w"> </span>libsqlite3.0.dylib<span class="w"> </span>-lm<span class="w"> </span>-lpthread<span class="w"> </span>-DSQLITE_ENABLE_FTS5
<span class="nb">popd</span>
<span class="nb">mv</span><span class="w"> </span>sqlite-amalgamation-<span class="si">${</span><span class="nv">sqlite_version</span><span class="si">}</span>/libsqlite3.0.dylib<span class="w"> </span>./lib/
<span class="nb">rm</span><span class="w"> </span>-r<span class="w"> </span>sqlite-amalgamation-<span class="si">${</span><span class="nv">sqlite_version</span><span class="si">}</span>.zip<span class="w"> </span>sqlite-amalgamation-<span class="si">${</span><span class="nv">sqlite_version</span><span class="si">}</span>
</code></pre></div>
<h4 id="compiling-the-right-version-on-demand">Compiling the right version on-demand</h4>
<p>We then define a <code>$(libsqlite)</code> <code>make</code> target, either pointing to <code>lib/libsqlite3.so</code> if you run the app on linux, or <code>lib/libsqlite3.0.dylib</code> if you run it on macOS. We finally make sure to override the system shared library by the vendored one when running the app, via <code>LD_PRELOAD</code> on linux and <code>DYLD_LIBRARY_PATH</code> on macOS.</p>
<div class="highlight"><pre><span></span><code><span class="c"># Makefile</span>
<span class="nv">UNAME_S</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>shell<span class="w"> </span>uname<span class="w"> </span>-s<span class="k">)</span>
<span class="nv">PWD</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>shell<span class="w"> </span><span class="nb">pwd</span><span class="k">)</span>
<span class="cp">ifeq ($(UNAME_S),Linux)</span>
<span class="w"> </span><span class="nv">libsqlite</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>lib/libsqlite3.so
<span class="w"> </span><span class="nv">ld_preload</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">LD_PRELOAD</span><span class="o">=</span><span class="k">$(</span>PWD<span class="k">)</span>/<span class="k">$(</span>libsqlite<span class="k">)</span>
<span class="cp">else ifeq ($(UNAME_S),Darwin)</span>
<span class="w"> </span><span class="nv">libsqlite</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>lib/libsqlite3.0.dylib
<span class="w"> </span><span class="nv">ld_preload</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">DYLD_LIBRARY_PATH</span><span class="o">=</span><span class="k">$(</span>PWD<span class="k">)</span>/lib
<span class="cp">endif</span>
<span class="nv">app-root</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>dnd5esheets
<span class="nv">poetry-run</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">$(</span>ld_preload<span class="k">)</span><span class="w"> </span>poetry<span class="w"> </span>run
<span class="nf">lib/libsqlite3.so</span><span class="o">:</span>
<span class="w"> </span>@./scripts/compile-libsqlite-linux.sh
<span class="nf">lib/libsqlite3.0.dylib</span><span class="o">:</span>
<span class="w"> </span>@./scripts/compile-libsqlite-macos.sh
<span class="nf">build</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">libsqlite</span><span class="k">)</span> ...
<span class="nf">test</span><span class="o">:</span>
<span class="w"> </span>@<span class="k">$(</span>poetry-run<span class="k">)</span><span class="w"> </span>pytest
<span class="nf">run</span><span class="o">:</span><span class="w"> </span><span class="n">build</span> ...
<span class="w"> </span>@cd<span class="w"> </span><span class="k">$(</span>app-root<span class="k">)</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="k">$(</span>poetry-run<span class="k">)</span><span class="w"> </span>uvicorn<span class="w"> </span>--factory<span class="w"> </span><span class="k">$(</span>app-root<span class="k">)</span>.app:create_app<span class="w"> </span>--reload
</code></pre></div>
<h4 id="compiling-libsqlite3-in-docker">Compiling <code>libsqlite3</code> in docker</h4>
<p>While the previous steps work, they also prove to be quite brittle, as they only works for a given CPU architecture. For example, the <code>libsqlite3.0.dylib</code> library will not load on an Intel Mac if it was compiled on a M1 or M2.</p>
<p>The most robust way to go remains building <code>libsqlite3</code> in a <a href="https://docs.docker.com/build/building/multi-stage/">build stage</a> of the docker image process. This way, you <em>know</em> that you only need to build it for linux, whatever the host OS is, and you're guaranteed that it will be built for your CPU architecture, thanks to the <a href="https://hub.docker.com/layers/library/python/3.11.4-slim-bullseye/images/sha256-1226f32ad1c1c36e0b6e79706059761c58ada308f4a1ad798e55dab346e10e91?context=explore">multi-arch property</a> of the <code>python:3.11.4-slim</code> base image.</p>
<div class="highlight"><pre><span></span><code><span class="c"># Dockerfile</span>
...
<span class="c"># -- Build the libsqlite3.so shared object for the appropriate architecture</span>
<span class="k">FROM</span><span class="w"> </span><span class="s">python:3.11.4-slim</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">sqlite-build</span>
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/app/src/build</span>
<span class="k">COPY</span><span class="w"> </span>scripts/compile-libsqlite-linux.sh<span class="w"> </span>./
<span class="k">RUN</span><span class="w"> </span>apt-get<span class="w"> </span>update<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>--no-install-recommends<span class="w"> </span>-y<span class="w"> </span>build-essential<span class="w"> </span>wget<span class="w"> </span>tcl<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>./compile-libsqlite-linux.sh<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>apt-get<span class="w"> </span>remove<span class="w"> </span>-y<span class="w"> </span>build-essential<span class="w"> </span>wget<span class="w"> </span>tcl<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>apt-get<span class="w"> </span>auto-clean
<span class="c"># -- Main build combining the FastAPI and compiled frontend apps</span>
<span class="k">FROM</span><span class="w"> </span><span class="s">python:3.11.4-slim</span>
...
<span class="k">COPY</span><span class="w"> </span>--from<span class="o">=</span>sqlite-build<span class="w"> </span>/app/src/build/libsqlite3.so<span class="w"> </span>./lib/libsqlite3.so
<span class="k">CMD</span><span class="w"> </span><span class="p">[</span><span class="s2">"./start-app.sh"</span><span class="p">]</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="c1"># start-app.sh</span>
<span class="c1">#!/bin/bash</span>
<span class="nb">set</span><span class="w"> </span>-e
<span class="nb">exec</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>env<span class="w"> </span><span class="nv">LD_PRELOAD</span><span class="o">=</span>./lib/libsqlite3.so<span class="w"> </span><span class="se">\ </span><span class="c1"># inject the LD_PRELOAD environment variable in the process</span>
<span class="w"> </span>uvicorn<span class="w"> </span>--factory<span class="w"> </span>dnd5esheets.app:create_app<span class="w"> </span>--host<span class="w"> </span><span class="s2">"0.0.0.0"</span><span class="w"> </span>--port<span class="w"> </span><span class="m">8000</span>
</code></pre></div>
<h3 id="unit-testing-the-sqlite-version-and-feature-set">Unit testing the SQLite version and feature set</h3>
<p>With all of that said and done, we can now expose the <code>sqlite</code> version and compilation options through a debug API handler:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># dnd5esheets/api/debug.py</span>
<span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">APIRouter</span><span class="p">,</span> <span class="n">Depends</span>
<span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">text</span>
<span class="kn">from</span> <span class="nn">sqlalchemy.ext.asyncio</span> <span class="kn">import</span> <span class="n">AsyncSession</span>
<span class="kn">from</span> <span class="nn">dnd5esheets.db</span> <span class="kn">import</span> <span class="n">create_scoped_session</span>
<span class="n">debug_api</span> <span class="o">=</span> <span class="n">APIRouter</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s2">"/debug"</span><span class="p">)</span>
<span class="nd">@debug_api</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"/sqlite"</span><span class="p">)</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">sqlite_info</span><span class="p">(</span>
<span class="n">session</span><span class="p">:</span> <span class="n">AsyncSession</span> <span class="o">=</span> <span class="n">Depends</span><span class="p">(</span><span class="n">create_scoped_session</span><span class="p">),</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return debug information about the sqlite database"""</span>
<span class="n">version</span> <span class="o">=</span> <span class="p">(</span><span class="k">await</span> <span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">text</span><span class="p">(</span><span class="s2">"select sqlite_version()"</span><span class="p">)))</span><span class="o">.</span><span class="n">scalar_one</span><span class="p">()</span>
<span class="n">pragma_compile_options</span> <span class="o">=</span> <span class="p">(</span>
<span class="p">(</span><span class="k">await</span> <span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">text</span><span class="p">(</span><span class="s2">"pragma compile_options"</span><span class="p">)))</span><span class="o">.</span><span class="n">scalars</span><span class="p">()</span><span class="o">.</span><span class="n">all</span><span class="p">()</span>
<span class="p">)</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s2">"version"</span><span class="p">:</span> <span class="n">version</span><span class="p">,</span>
<span class="s2">"compile_options"</span><span class="p">:</span> <span class="n">pragma_compile_options</span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div>
<p>We can then query the <code>sqlite</code> version through the API:</p>
<div class="highlight"><pre><span></span><code>❯<span class="w"> </span>curl<span class="w"> </span>-s<span class="w"> </span>localhost:8000/api/debug/sqlite<span class="w"> </span><span class="k">|</span><span class="w"> </span>jq<span class="w"> </span>.version
<span class="s2">"3.42.0"</span>
</code></pre></div>
<p>However, we can go even further! By unit-testing the version and compile options, we ensure that our CI uses the exact required <code>sqlite</code> version and feature set.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># dnd5esheets/tests/test_api_debug.py</span>
<span class="k">def</span> <span class="nf">test_sqlite_version</span><span class="p">(</span><span class="n">client</span><span class="p">):</span>
<span class="n">sqlite_debug_info</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"/api/debug/sqlite"</span><span class="p">)</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
<span class="k">assert</span> <span class="n">sqlite_debug_info</span><span class="p">[</span><span class="s2">"version"</span><span class="p">]</span> <span class="o">==</span> <span class="s2">"3.42.0"</span>
<span class="k">assert</span> <span class="s2">"ENABLE_FTS5"</span> <span class="ow">in</span> <span class="n">sqlite_debug_info</span><span class="p">[</span><span class="s2">"compile_options"</span><span class="p">]</span>
</code></pre></div>
<div class="Note">
<p>See the effect of vendoring the compiled library in CI: <a href="https://github.com/brouberol/5esheets/actions/runs/5956139650/job/16156271800#step:8:49">before</a> / <a href="https://github.com/brouberol/5esheets/actions/runs/5960929753/job/16169158344#step:8:24">after</a>.</p>
</div>
<h3 id="sources">Sources</h3>
<ul>
<li><a href="https://til.simonwillison.net/sqlite/python-sqlite-environment">https://til.simonwillison.net/sqlite/python-sqlite-environment</a></li>
<li><a href="https://til.simonwillison.net/sqlite/sqlite-version-macos-python">https://til.simonwillison.net/sqlite/sqlite-version-macos-python</a></li>
</ul>How to profile a FastAPI asynchronous request2023-08-05T00:00:00+02:002023-08-05T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2023-08-05:/how-to-profile-a-fastapi-asynchronous-request<p>In this article, I share the challenges I faced when trying to profile requests in an asynchronous FastAPI server. The traditional profiler, <code>cProfile</code>, provided inaccurate results due to the nature of asynchronous functions, which resulted in misleading statistics. To overcome this, I explored <code>pyinstrument</code>, a statistical profiler with built-in support for asynchronous Python code.</p><p>I have been experimenting with <a href="https://fastapi.tiangolo.com/">FastAPI</a> recently, a Python API framework self-describing as "high performance, easy to learn, fast to code, ready for production".</p>
<p>One of the features I wanted my <a href="https://github.com/brouberol/5esheets">project</a> to have is to be fully asynchronous, from the app server to the SQL requests. As the API is mostly I/O bound, this would allow it to handle many concurrent requests with a single server process, instead of starting a thread per request, as one commony seen with Flask/Gunicorn.</p>
<p>However, this poses a challenge when it comes to <em>profiling</em> the code and interpreting the results.</p>
<h3 id="the-limitations-of-cprofile-when-profiling-asynchronous-code">The limitations of <code>cProfile</code> when profiling asynchronous code</h3>
<p>For example, the following graph representation was generated from a <code>cProfile</code> profile recording 300 consecutive calls to a single API endpoint, with an associated <code>get_character</code> <a href="https://github.com/brouberol/5esheets/blob/3b3bd1f99159f13e1b0e95b6ce3f825bc65a1e2d/dnd5esheets/api/character.py#L48-L63">handler</a>.</p>
<p><img alt="profile-cprofile" decoding="async" loading="lazy" src="https://user-images.githubusercontent.com/480131/258567029-c3fc4124-4822-49b2-8ce7-1cb79c501227.png"></p>
<p>Zooming in, we notice 2 things about the <code>get_character</code> span:</p>
<ul>
<li>its <code>ncalls</code> value is 9605, when we really called it 300 times</li>
<li>it is free-floating, completely unlinked from any other span</li>
</ul>
<p><img alt="get-character-span" decoding="async" loading="lazy" src="https://github.com/brouberol/5esheets/assets/480131/71ec8ae5-553b-44bc-9613-30b5da9a6240"></p>
<p>As an asynchronous function is "entered" and "exited" by the event loop at each <code>await</code> clause, every time the event-loop re-enters a function, <code>cProfile</code> will see this as an additional call, thus causing seemingly larger-than-normal <code>ncalls</code> numbers. Indeed, we <code>await</code> every-time we perform an SQL request, commit or refresh the SQLAlchemy session, or anything else inducing asynchronous I/O.
Secondly, the reason that the <code>get_character</code> span appears to be free-floating is because it is executed outside of the main thread, by the Python event-loop.</p>
<p>This means that our good old faithful <code>cProfile</code> might not cut it for this inherently asynchronous server, and we need a more modern profiler with builtin asynchronous support if we want to really make sense of where time is spent during a request.</p>
<h3 id="enter-pyinstrument">Enter <a href="https://pyinstrument.readthedocs.io/">pyinstrument</a>!</h3>
<p><code>pyinstrument</code> is a <em>statistical profiler</em>, contrary to <code>cProfile</code>, which is <em>deterministic</em>.</p>
<blockquote>
<p>Deterministic profiling is meant to reflect the fact that all function call, function return, and exception events are monitored, and precise timings are made for the intervals between these events (during which time the user’s code is executing). In contrast, statistical profiling [...] randomly samples the effective instruction pointer, and deduces where time is being spent. The latter technique traditionally involves less overhead (as the code does not need to be instrumented), but provides only relative indications of where time is being spent.</p>
<p><em><a href="https://docs.python.org/3/library/profile.html#what-is-deterministic-profiling">Source</a></em></p>
</blockquote>
<p>Second, it advertises native support for profiling asynchronous python code:</p>
<blockquote>
<p><code>pyinstrument</code> can profile async programs that use <code>async</code> and <code>await</code>. This async support works by tracking the context of execution, as provided by the built-in <a href="https://docs.python.org/3/library/contextvars.html"><code>contextvars</code></a> module.</p>
<p>When you start a <code>Profiler</code> with the <code>async_mode</code> enabled or strict (not disabled), that <code>Profiler</code> is attached to the current async context.</p>
<p>When profiling, <code>pyinstrument</code> keeps an eye on the context. When execution exits the context, it captures the await stack that caused the context to exit. Any time spent outside the context is attributed to the that halted execution of the await.</p>
<p><a href="https://pyinstrument.readthedocs.io/en/latest/how-it-works.html#async-profiling">Source</a></p>
</blockquote>
<p>This should allow us to get a sensible picture of where time is spent during the lifespan of a FastAPI request, while also skipping the spans that are too fast to be profiled.</p>
<h3 id="integrating-pyinstrument-with-fastapi">Integrating pyinstrument with FastAPI</h3>
<p>We rely on the <code>FastAPI.middleware</code> decorator to register a profiling middleware (only enabled if the <code>PROFILING_ENABLED</code> setting it set to <code>True</code>) in charge of profiling a request if the <code>profile=true</code> query argument is passed by the client.</p>
<p>By default, this middleware will generate a JSON report compatible with <a href="https://speedscope.app">Speedscope</a>, an online interactive flamegraph visualizer. However, if the <code>profile_format=html</code> query argument is passed, then a simple HTML report will be dumped to disk instead.</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">Request</span>
<span class="kn">from</span> <span class="nn">pyinstrument</span> <span class="kn">import</span> <span class="n">Profiler</span>
<span class="kn">from</span> <span class="nn">pyinstrument.renderers.html</span> <span class="kn">import</span> <span class="n">HTMLRenderer</span>
<span class="kn">from</span> <span class="nn">pyinstrument.renderers.speedscope</span> <span class="kn">import</span> <span class="n">SpeedscopeRenderer</span>
<span class="k">def</span> <span class="nf">register_middlewares</span><span class="p">(</span><span class="n">app</span><span class="p">:</span> <span class="n">FastAPI</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">if</span> <span class="n">app</span><span class="o">.</span><span class="n">settings</span><span class="o">.</span><span class="n">PROFILING_ENABLED</span> <span class="ow">is</span> <span class="kc">True</span><span class="p">:</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">middleware</span><span class="p">(</span><span class="s2">"http"</span><span class="p">)</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">profile_request</span><span class="p">(</span><span class="n">request</span><span class="p">:</span> <span class="n">Request</span><span class="p">,</span> <span class="n">call_next</span><span class="p">:</span> <span class="n">Callable</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Profile the current request</span>
<span class="sd"> Taken from https://pyinstrument.readthedocs.io/en/latest/guide.html#profile-a-web-request-in-fastapi</span>
<span class="sd"> with small improvements.</span>
<span class="sd"> """</span>
<span class="c1"># we map a profile type to a file extension, as well as a pyinstrument profile renderer</span>
<span class="n">profile_type_to_ext</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"html"</span><span class="p">:</span> <span class="s2">"html"</span><span class="p">,</span> <span class="s2">"speedscope"</span><span class="p">:</span> <span class="s2">"speedscope.json"</span><span class="p">}</span>
<span class="n">profile_type_to_renderer</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"html"</span><span class="p">:</span> <span class="n">HTMLRenderer</span><span class="p">,</span>
<span class="s2">"speedscope"</span><span class="p">:</span> <span class="n">SpeedscopeRenderer</span><span class="p">,</span>
<span class="p">}</span>
<span class="c1"># if the `profile=true` HTTP query argument is passed, we profile the request</span>
<span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">query_params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"profile"</span><span class="p">,</span> <span class="kc">False</span><span class="p">):</span>
<span class="c1"># The default profile format is speedscope</span>
<span class="n">profile_type</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">query_params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"profile_format"</span><span class="p">,</span> <span class="s2">"speedscope"</span><span class="p">)</span>
<span class="c1"># we profile the request along with all additional middlewares, by interrupting</span>
<span class="c1"># the program every 1ms1 and records the entire stack at that point</span>
<span class="k">with</span> <span class="n">Profiler</span><span class="p">(</span><span class="n">interval</span><span class="o">=</span><span class="mf">0.001</span><span class="p">,</span> <span class="n">async_mode</span><span class="o">=</span><span class="s2">"enabled"</span><span class="p">)</span> <span class="k">as</span> <span class="n">profiler</span><span class="p">:</span>
<span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">call_next</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="c1"># we dump the profiling into a file</span>
<span class="n">extension</span> <span class="o">=</span> <span class="n">profile_type_to_ext</span><span class="p">[</span><span class="n">profile_type</span><span class="p">]</span>
<span class="n">renderer</span> <span class="o">=</span> <span class="n">profile_type_to_renderer</span><span class="p">[</span><span class="n">profile_type</span><span class="p">]()</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="sa">f</span><span class="s2">"profile.</span><span class="si">{</span><span class="n">extension</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">out</span><span class="p">:</span>
<span class="n">out</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">profiler</span><span class="o">.</span><span class="n">output</span><span class="p">(</span><span class="n">renderer</span><span class="o">=</span><span class="n">renderer</span><span class="p">))</span>
<span class="k">return</span> <span class="n">response</span>
<span class="c1"># Proceed without profiling</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">call_next</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</code></pre></div>
<div class="Note">
<p>You can browse the project <a href="https://github.com/brouberol/5esheets/blob/main/dnd5esheets/middlewares.py">code</a> to see how the middleware is wired into the application itself</p>
</div>
<h3 id="lets-see-the-results">Let's see the results</h3>
<p><strong>HTML profile</strong>
<img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/fastapi-async-profiling/html-pyinstrument.webp"></p>
<p><strong>Speedscope profile</strong>
<img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/fastapi-async-profiling/speedscope.webp"></p>
<p>We see pretty clearly the different SQL requests being performed (the <code>execute</code> spans), the different <code>await</code> clauses in the code causing the event loop to pause the execution, and that most of the request time is spent in SQL requests.</p>
<p>Finally, using this setup, I was able to <a href="https://github.com/brouberol/5esheets/pull/180">observe the effects</a> of replacing the <code>json</code> stdlib library by <a href="https://github.com/ijl/orjson"><code>orjson</code></a> when deserializing JSON data from database, and speed up this endpoint by a couple of percent very easily.</p>
<h3 id="sources">Sources</h3>
<ul>
<li><a href="https://pyinstrument.readthedocs.io/en/latest/how-it-works.html">https://pyinstrument.readthedocs.io/en/latest/how-it-works.html</a></li>
<li><a href="https://www.roguelynn.com/words/asyncio-profiling/">https://www.roguelynn.com/words/asyncio-profiling</a></li>
</ul>Preventing a pull request from being merged until it's safe2023-07-25T00:00:00+02:002023-07-25T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2023-07-25:/preventing-a-pull-request-from-being-merged-until-its-safe<p>Sometimes, a pull request is ready to go, but shouldn't be merged before some other changes are merged first. While the patch is valid on its own, it might depend on other changes, and could even break the application if merged <em>before</em> the other. I'll demonstrate a simple technique relying on Github Actions and pull request labels to block a pull request from being merged, until deemed safe to do so.</p><p>Sometimes, a pull request is ready to go, but shouldn't be merged before some other changes are merged first. While the patch is valid on its own, it might depend on other changes, and could even break the application if merged <em>before</em> the other.</p>
<p>I'll demonstrate a simple technique relying on Github Actions and pull request labels to fully block a pull request from being merged until deemed safe (at least without some admin privileges on the repository).</p>
<p>First, we introduce a Github Actions <a href="https://github.com/brouberol/5esheets/blob/main/.github/workflows/fail-if-do-not-merge-label.yml">workflow</a> executed when a pull request is opened, labeled or unlabeled. This workflow will fail if labeled with <code>do not merge</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Check do not merge</span>
<span class="nt">on</span><span class="p">:</span>
<span class="w"> </span><span class="c1"># Check label at every push in a feature branch</span>
<span class="w"> </span><span class="nt">push</span><span class="p">:</span>
<span class="w"> </span><span class="nt">branches-ignore</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">main</span>
<span class="w"> </span><span class="c1"># Check label during the lifetime of a pull request</span>
<span class="w"> </span><span class="nt">pull_request</span><span class="p">:</span>
<span class="w"> </span><span class="nt">types</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">opened</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">labeled</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">unlabeled</span>
<span class="nt">jobs</span><span class="p">:</span>
<span class="w"> </span><span class="nt">fail-for-do-not-merge</span><span class="p">:</span>
<span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">contains(github.event.pull_request.labels.*.name, 'do not merge')</span>
<span class="w"> </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ubuntu-latest</span>
<span class="w"> </span><span class="nt">steps</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Fail if PR is labeled with do not merge</span>
<span class="w"> </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span>
<span class="w"> </span><span class="no">echo "This PR can't be merged, due to the 'do not merge' label."</span>
<span class="w"> </span><span class="no">exit 1</span>
</code></pre></div>
<p>We then define a branch protection rule for our <code>main</code> branch, by going to the repository <code>Settings</code>, then <code>Branches</code>. We add a new rule if none exist, tick <code>Require status checks to pass before merging</code>, and add the <code>fail-for-do-not-merge</code> to the list of required checks.</p>
<p>Finally, apply the <code>do not merge</code> label to your pull request.</p>
<div class="row">
<div class="column" style="flex: 70%">
<img src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/do-not-merge/required-checks.webp
">
</div>
<div class="column" style="flex: 30%">
<img src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/do-not-merge/labels.webp
">
</div>
</div>
<p>At that point, the <code>fail-for-do-not-merge</code> check will run and fail, preventing the PR to be merged.</p>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/do-not-merge/merge-blocked.webp"></p>
<p>When the pull request is finally safe to merge, simply remove the <code>do not merge</code> tag, and the checks will automagically pass, thus allowing you to merge.</p>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/do-not-merge/passing-checks.webp"></p>Neapolitan pizza dough recipe2023-06-09T00:00:00+02:002023-06-09T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2023-06-09:/neapolitan-pizza-dough-recipe<p><img alt="pizza" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/neapolitan-pizza-recipe/pizza.jpg"> I have created a small <a href="https://balthazar-rouberol.com/pizza">Neapolitan pizza dough recipe calculator</a> so I don't have to figure out the recipe from scratch everytime I'm making pizza.</p><p>Whenever I'm baking pizzas, the first step is always to remember the dough recipe, and adjust for the number of pies I'm making. I have probably written it on at least 10 pieces of paper, that all ended up in the trash.</p>
<p>I have created a small <a href="https://balthazar-rouberol.com/pizza">pizza dough recipe calculator</a> so I don't have to go through these steps again.</p>Merging multiple mp3 files into an audiobook with chapters2023-05-08T00:00:00+02:002023-05-08T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2023-05-08:/merging-multiple-mp3-files-into-an-audiobook-with-chapters<p>A quick walkthough of how to convert multiple mp3 files into a fully-fledged audiobook m4b file, with chapters and metadata.</p><p>I recently found the 3 Lord of the Rings audiobooks I bought from <a href="https://www.phildragash.com/index.html">Phil Dragash</a> some years back, as I was digging through my NAS. Each book is split into about 20 mp3 files, which makes it a bit unwieldy for me. As I mostly listen to audiobooks when I'm going to sleep, I oftentimes have to find the last part I remember listening to and start again from there the next day.</p>
<p>Luckily, <a href="https://apps.apple.com/fr/app/bookplayer/id1138219998">BookPlayer</a> solves this for me, via its "Stop after this chapter" feature. However, to import these books into the app, I needed to merge the mp3 files into a full-fledge audiobook <a href="https://fileinfo.com/extension/m4b">m4b</a> file, with chapter metadata.</p>
<p>After digging a little bit, this is what I found:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-it<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># to see the output of the containerized process in the terminal</span>
<span class="w"> </span>--rm<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># delete the container once the conversion ends</span>
<span class="w"> </span>-u<span class="w"> </span><span class="k">$(</span>id<span class="w"> </span>-u<span class="k">)</span>:<span class="k">$(</span>id<span class="w"> </span>-g<span class="k">)</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># Use the same UID and GID than in the host to avoid permission issues</span>
<span class="w"> </span>-v<span class="w"> </span><span class="s2">"</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span><span class="s2">/audiobooks"</span>:/mnt<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># mount the ./audiobooks folder into /mnt in the container via a docker volume</span>
<span class="w"> </span>sandreas/m4b-tool:latest<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># cf https://hub.docker.com/r/sandreas/m4b-tool</span>
<span class="w"> </span>merge<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># subcommand in charge of merging the mp3 files into a single m4b file</span>
<span class="w"> </span><span class="s2">"/mnt/The Fellowship of the Ring"</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># directory in which the audio files are located</span>
<span class="w"> </span>--output-file<span class="w"> </span><span class="s2">"/mnt/The Fellowship of the Ring.m4b"</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># name of the generated m4b file</span>
<span class="w"> </span>--series<span class="w"> </span><span class="s2">"The Lord of the Rings"</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># name of the book series</span>
<span class="w"> </span>--name<span class="o">=</span><span class="s2">"The Fellowship of the Ring"</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># title of the book</span>
<span class="w"> </span>--series-part<span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># book number in the series</span>
<span class="w"> </span>--artist<span class="w"> </span><span class="s2">"J.R.R Tolkien"</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># writer's name</span>
<span class="w"> </span>--albumartist<span class="o">=</span><span class="s2">"Phil Dragash"</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># narrator's name</span>
<span class="w"> </span>--use-filenames-as-chapters<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># generate a chapter per mp3 file</span>
<span class="w"> </span>--cover<span class="w"> </span><span class="s2">"/mnt/The Fellowship of the Ring/cover.jpg"</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># path to the cover image</span>
<span class="w"> </span>--jobs<span class="o">=</span><span class="m">8</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># I used number of CPUs - 2</span>
<span class="w"> </span>--audio-channels<span class="o">=</span><span class="m">2</span><span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="c1"># 1=mono, 2=stereo</span>
<span class="w"> </span>--audio-samplerate<span class="o">=</span><span class="m">44100</span><span class="w"> </span><span class="c1"># I used the same as in the input files</span>
</code></pre></div>
<p>Here's the result after I imported the result m4b file into BookPlayer.</p>
<div class="row">
<div class="column">
<img src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/audiobook/IMG_7318.webp
">
</div>
<div class="column">
<img src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/audiobook/IMG_7319.webp
">
</div>
</div>Generating pretty maps ready to be gift-wrapped2023-05-06T00:00:00+02:002023-05-06T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2023-05-06:/generating-pretty-maps-ready-to-be-gift-wrapped<p><img title="Lyon, France" alt="Lyon, France" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/lyonfrance-3000-A3-square-default.jpg" />I have been toying with the idea of generating visually pleasing maps centered on a given address, to have them printed and framed. The way I see it, it would make an original and personalised gift for the person living there. So when Marcelo de Oliveira Rosa Prates' <a href="https://github.com/marceloprates/prettymaps"><code>prettymaps</code></a> blew up on Reddit, I decided to try it.</p><p><img title="Lyon, France" alt="Lyon, France" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/lyonfrance-3000-A3-square-default.jpg" /></p>
<p>I have been toying with the idea of generating visually pleasing maps centered on a given address, to have them printed and framed. The way I see it, it would make an original and personalised gift for the person living there. So when Marcelo de Oliveira Rosa Prates' <a href="https://github.com/marceloprates/prettymaps"><code>prettymaps</code></a> blew up on Reddit, I decided to try it.</p>
<p>The library was great and the visuals looked incredible, yet, I felt it was lacking a couple of features if I were to print the maps.</p>
<ul>
<li>a CLI to make it easy to generate maps on the fly</li>
<li>easily changing the color scheme of buildings (and allowing black and white)</li>
<li>enabling the generation of rectangular maps, on top of circle and square</li>
<li>changing the output format of the figure to make it fit into a standard page (A3, A4, etc)</li>
<li>ensuring a 300dpi output</li>
<li>set the CLI command used to generate the map as the map title, for autodocumentation purposes</li>
</ul>
<p>My good friend <a href="https://etnbrd.com/">Etienne</a> solved the <a href="https://github.com/marceloprates/prettymaps/pull/105">rectangular map generation</a> in a <em>beautifully</em> laid out PR, that has been sadly sitting there for a while without attention. It <em>seems</em> that the repository owner got issues with NFT con "artists", and pretty much abandonned the project, which hasn't seen activity for the last 5 months.</p>
<p>Seeing this, I decided to <a href="https://github.com/brouberol/prettymaps">fork the project</a>, and work on the remaining ideas.</p>
<p>Here are a couple of examples of maps that I've generated and printed for people in my entourage.</p>
<div class="Note">
<p>The command used to generate each map is displayed as the map title</p>
</div>
<p><img
src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/248ruedespyrénées_75020_paris_france-2000-A3-square-RdPu-50.webp"
srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/248ruedespyrénées_75020_paris_france-2000-A3-square-RdPu-30.webp 1448w, https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/248ruedespyrénées_75020_paris_france-2000-A3-square-RdPu-50.webp 2480w"
sizes="(max-width: 1500px) 1448w, 2480w"
alt="Paris 20e, France"
title="Paris 20e, France"
/>
<img
src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/40all.jeanjaurès31000toulouse_france-2000-A3-circle-default-50.webp"
srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/40all.jeanjaurès31000toulouse_france-2000-A3-circle-default-30.webp 1488w, https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/40all.jeanjaurès31000toulouse_france-2000-A3-circle-default-50.webp 2480w"
sizes="(max-width: 1500px) 1488w, 2480w"
alt="Toulouse, France"
title="Toulouse, France"
/>
<img
src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/2impassedel'ancienneposte71100chalonsursaône_france-2000-A3-square-viridis_r-50.webp"
srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/2impassedel'ancienneposte71100chalonsursaône_france-2000-A3-square-viridis_r-30.webp 1488w, https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/2impassedel'ancienneposte71100chalonsursaône_france-2000-A3-square-viridis_r-50.webp 2480w"
sizes="(max-width: 1500px) 1488w, 2480w"
alt="Chalon-sur-Saône, France"
title="Chalon-sur-Saône, France"
/>
<img
src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/19ruegustavebalny_60320_béthisy-saint-martin-3000-A3-square-Oranges-50.webp"
srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/19ruegustavebalny_60320_béthisy-saint-martin-3000-A3-square-Oranges-30.webp 1488w, https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/prettymaps/19ruegustavebalny_60320_béthisy-saint-martin-3000-A3-square-Oranges-50.webp 2480w"
sizes="(max-width: 1500px) 1488w, 2480w"
alt="Béthisy Saint-Martin, France"
title="Béthisy Saint-Martin, France"
/></p>
<p>The color schemes are only applied to buildings, and are automatically generated from <a href="https://matplotlib.org/stable/gallery/color/colormap_reference.html"><code>matplotlib</code> colormaps</a>. This was an quick and easy to generate themes "for free". I also added a couple of Scottish tartan inspired themes, that I used to print a map as a wedding gift for a lovely franco-scottish couple.</p>
<p>My local printer bills me about 1.5€ for each print, which makes for an original and yet remarkably cheap gift. I recommend a thick and matte paper, without any texture, as it might collide with the map dotted background.</p>
<p>If you'd like to give it a try, feel free to have a look at the <a href="https://github.com/brouberol/prettymaps">repository</a>!</p>Monitoring my solar panel power production2023-05-03T00:00:00+02:002023-05-03T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2023-05-03:/monitoring-my-solar-panel-power-production<p>I have recently acquired two solar panels from <a href="https://sunology.eu/products/sunology-play-kit-solaire">Sunology</a> advertising a cumulated instantaneous production of up to 810W. The panels come with a smart plug emitting the data to <a href="https://iot.tuya.com/">Tuya</a>, in order to retain and graph historical data. However, the only available granuarity for that data is <em>daily</em> kWh production. In order to optimize the orientation and placement of the panels, as well as measure the production efficiency (power produced / 810 * 100), I needed a much finer granularity than that. I decided to query the data myself and send it to Datadog.</p><p>I have recently acquired two solar panels from <a href="https://sunology.eu/products/sunology-play-kit-solaire">Sunology</a> advertising a cumulated instantaneous production of up to 810W. The panels come with a smart plug emitting the data to <a href="https://iot.tuya.com/">Tuya</a>, in order to retain and graph historical data. However, the only available granuarity for that data is <em>daily</em> kWh production. In order to optimize the orientation and placement of the panels, as well as measure the production efficiency (power produced / 810 * 100), I needed a much finer granularity than that. I decided to query the data myself and send it to Datadog.</p>
<p><img alt="information flow from plug to Datadog" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/solar-panel/schema.webp"></p>
<p>The first thing I needed to do was to find a working client that would be able to talk to the plug. It seems that <a href="https://github.com/jasonacox/tinytuya"><code>tinytuya</code></a> would do the job. However, it didn't seem like I could simply fetch the data from the plug locally. Instead, I first needed to create a Tuya account, a cloud project, and add the plug to the project devices to get both an API key as well as a key for the plug. That proved out to be quite tedious, as the Tuya IoT interface is very confusing and slow, but I managed thanks to these <a href="https://www.home-assistant.io/integrations/tuya/">Home-Assistant instructions</a>.</p>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/solar-panel/tuya-project.webp"></p>
<hr>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/solar-panel/tuya-device.webp"></p>
<p>With that data now available, I was then able to setup the <code>tinytuya</code> client on a Raspberry Pi with network access to the plug IP.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>python<span class="w"> </span>-m<span class="w"> </span>tinytuya<span class="w"> </span>wizard
TinyTuya<span class="w"> </span>Setup<span class="w"> </span>Wizard<span class="w"> </span><span class="o">[</span><span class="m">1</span>.12.4<span class="o">]</span>
<span class="w"> </span>Enter<span class="w"> </span>API<span class="w"> </span>Key<span class="w"> </span>from<span class="w"> </span>tuya.com:<span class="w"> </span><span class="o">[</span>REDACTED<span class="o">]</span>
<span class="w"> </span>Enter<span class="w"> </span>API<span class="w"> </span>Secret<span class="w"> </span>from<span class="w"> </span>tuya.com:<span class="w"> </span><span class="o">[</span>REDACTED<span class="o">]</span>
<span class="w"> </span>Enter<span class="w"> </span>any<span class="w"> </span>Device<span class="w"> </span>ID<span class="w"> </span>currently<span class="w"> </span>registered<span class="w"> </span><span class="k">in</span><span class="w"> </span>Tuya<span class="w"> </span>App<span class="w"> </span><span class="o">(</span>used<span class="w"> </span>to<span class="w"> </span>pull<span class="w"> </span>full<span class="w"> </span>list<span class="o">)</span><span class="w"> </span>or<span class="w"> </span><span class="s1">'scan'</span><span class="w"> </span>to<span class="w"> </span>scan<span class="w"> </span><span class="k">for</span><span class="w"> </span>one:<span class="w"> </span><span class="o">[</span>REDACTED<span class="o">]</span>
<span class="w"> </span>Enter<span class="w"> </span>Your<span class="w"> </span>Region<span class="w"> </span><span class="o">(</span>Options:<span class="w"> </span>cn,<span class="w"> </span>us,<span class="w"> </span>us-e,<span class="w"> </span>eu,<span class="w"> </span>eu-w,<span class="w"> </span>or<span class="w"> </span><span class="k">in</span><span class="o">)</span>:<span class="w"> </span>eu
>><span class="w"> </span>Configuration<span class="w"> </span>Data<span class="w"> </span>Saved<span class="w"> </span>to<span class="w"> </span>tinytuya.json
>><span class="w"> </span>Device<span class="w"> </span>Listing
>><span class="w"> </span>Saving<span class="w"> </span>list<span class="w"> </span>to<span class="w"> </span>devices.json
<span class="w"> </span><span class="m">1</span><span class="w"> </span>registered<span class="w"> </span>devices<span class="w"> </span>saved
>><span class="w"> </span>Saving<span class="w"> </span>raw<span class="w"> </span>TuyaPlatform<span class="w"> </span>response<span class="w"> </span>to<span class="w"> </span>tuya-raw.json
Poll<span class="w"> </span><span class="nb">local</span><span class="w"> </span>devices?<span class="w"> </span><span class="o">(</span>Y/n<span class="o">)</span>:<span class="w"> </span>y
Scanning<span class="w"> </span><span class="nb">local</span><span class="w"> </span>network<span class="w"> </span><span class="k">for</span><span class="w"> </span>Tuya<span class="w"> </span>devices...
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="nb">local</span><span class="w"> </span>devices<span class="w"> </span>discovered
Polling<span class="w"> </span><span class="nb">local</span><span class="w"> </span>devices...
<span class="w"> </span><span class="o">[</span>Sunology<span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="m">192</span>.168.5.171<span class="w"> </span>-<span class="w"> </span><span class="o">[</span>On<span class="o">]</span><span class="w"> </span>-<span class="w"> </span>DPS:<span class="w"> </span><span class="o">{</span><span class="s1">'1'</span>:<span class="w"> </span>True,<span class="w"> </span><span class="s1">'9'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'17'</span>:<span class="w"> </span><span class="m">109</span>,<span class="w"> </span><span class="s1">'18'</span>:<span class="w"> </span><span class="m">2704</span>,<span class="w"> </span><span class="s1">'19'</span>:<span class="w"> </span><span class="m">6491</span>,<span class="w"> </span><span class="s1">'20'</span>:<span class="w"> </span><span class="m">2379</span>,<span class="w"> </span><span class="s1">'21'</span>:<span class="w"> </span><span class="m">1</span>,<span class="w"> </span><span class="s1">'22'</span>:<span class="w"> </span><span class="m">529</span>,<span class="w"> </span><span class="s1">'23'</span>:<span class="w"> </span><span class="m">26153</span>,<span class="w"> </span><span class="s1">'24'</span>:<span class="w"> </span><span class="m">13705</span>,<span class="w"> </span><span class="s1">'25'</span>:<span class="w"> </span><span class="m">3040</span>,<span class="w"> </span><span class="s1">'26'</span>:<span class="w"> </span><span class="m">0</span><span class="o">}</span>
>><span class="w"> </span>Saving<span class="w"> </span>device<span class="w"> </span>snapshot<span class="w"> </span>data<span class="w"> </span>to<span class="w"> </span>snapshot.json
>><span class="w"> </span>Saving<span class="w"> </span>IP<span class="w"> </span>addresses<span class="w"> </span>to<span class="w"> </span>devices.json
<span class="w"> </span><span class="m">1</span><span class="w"> </span>device<span class="w"> </span>IP<span class="w"> </span>addresses<span class="w"> </span>found
Done.
</code></pre></div>
<p>At that point, the <code>tinytuya</code> wizard script had scanned the networks the Pi was connected to, found the plug, and was able to connect to it via the provided device key.</p>
<p>I then created a dedicated APP/API keypair on Datadog, and scheduled this python script to run every minute via cron.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Run every minute via this crontab</span>
<span class="c1"># * * * * * cd /home/br/tuya && /home/br/tuya/.env/bin/python exporter.py</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">datadog</span>
<span class="kn">import</span> <span class="nn">tinytuya</span>
<span class="n">datadog</span><span class="o">.</span><span class="n">initialize</span><span class="p">(</span>
<span class="n">api_key</span><span class="o">=</span><span class="s2">"[REDACTED]"</span><span class="p">,</span>
<span class="n">app_key</span><span class="o">=</span><span class="s2">"[REDACTED]"</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"devices.json"</span><span class="p">)</span> <span class="k">as</span> <span class="n">device_file</span><span class="p">:</span>
<span class="n">device_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">device_file</span><span class="p">)</span>
<span class="n">plug</span> <span class="o">=</span> <span class="n">tinytuya</span><span class="o">.</span><span class="n">OutletDevice</span><span class="p">(</span>
<span class="n">dev_id</span><span class="o">=</span><span class="n">device_data</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="s2">"id"</span><span class="p">],</span>
<span class="n">address</span><span class="o">=</span><span class="n">device_data</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="s2">"ip"</span><span class="p">],</span>
<span class="n">local_key</span><span class="o">=</span><span class="n">device_data</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="s2">"key"</span><span class="p">],</span>
<span class="n">version</span><span class="o">=</span><span class="mf">3.3</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">plug_status</span> <span class="o">=</span> <span class="n">plug</span><span class="o">.</span><span class="n">updatedps</span><span class="p">()</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">plug_status</span><span class="p">[</span><span class="s2">"dps"</span><span class="p">]</span>
<span class="n">now</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="n">metrics</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">if</span> <span class="s2">"18"</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span>
<span class="n">current</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="s2">"18"</span><span class="p">]</span> <span class="c1"># mA</span>
<span class="n">metrics</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="p">{</span>
<span class="s2">"metric"</span><span class="p">:</span> <span class="s2">"solarpanel.current"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="p">:</span> <span class="s2">"gauge"</span><span class="p">,</span>
<span class="s2">"points"</span><span class="p">:</span> <span class="p">[(</span><span class="n">now</span><span class="p">,</span> <span class="n">current</span><span class="p">)],</span>
<span class="s2">"tags"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"location:terrasse_1"</span><span class="p">],</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="k">if</span> <span class="s2">"19"</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span>
<span class="n">power</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="s2">"19"</span><span class="p">]</span> <span class="o">/</span> <span class="mf">10.0</span> <span class="c1"># W</span>
<span class="n">metrics</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="p">{</span>
<span class="s2">"metric"</span><span class="p">:</span> <span class="s2">"solarpanel.power"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="p">:</span> <span class="s2">"gauge"</span><span class="p">,</span>
<span class="s2">"points"</span><span class="p">:</span> <span class="p">[(</span><span class="n">now</span><span class="p">,</span> <span class="n">power</span><span class="p">)],</span>
<span class="s2">"tags"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"location:terrasse_1"</span><span class="p">],</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="k">if</span> <span class="s2">"20"</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span>
<span class="n">voltage</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="s2">"20"</span><span class="p">]</span> <span class="o">/</span> <span class="mf">10.0</span> <span class="c1"># V</span>
<span class="n">metrics</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="p">{</span>
<span class="s2">"metric"</span><span class="p">:</span> <span class="s2">"solarpanel.voltage"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="p">:</span> <span class="s2">"gauge"</span><span class="p">,</span>
<span class="s2">"points"</span><span class="p">:</span> <span class="p">[(</span><span class="n">now</span><span class="p">,</span> <span class="n">voltage</span><span class="p">)],</span>
<span class="s2">"tags"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"location:terrasse_1"</span><span class="p">],</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="n">datadog</span><span class="o">.</span><span class="n">api</span><span class="o">.</span><span class="n">Metric</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">metrics</span><span class="o">=</span><span class="n">metrics</span><span class="p">)</span>
</code></pre></div>
<p>At that point, the measured current, voltage and power was sent out to Datadog every minute, and I was then able to create the following <a href="https://p.datadoghq.com/sb/bc352bb82-f277a5982d97a0a007ab56fbc05e0ee8">dashboard</a>:</p>
<p><a href="https://p.datadoghq.com/sb/bc352bb82-f277a5982d97a0a007ab56fbc05e0ee8"><img alt="Dashboard detailing electricity production over time" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/solar-panel/dd-dash.webp"></a></p>
<div class="Note">
<p>This dashboard makes it seem like the panel can only hit 75% efficiency, when I have seen them hit 95-99%. This is due to the Datadog point interpolation happening on large time windows. When we focus on a smaller window, we can see these high (albeit brief) peaks. <img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/solar-panel/dd-dash-2.webp"></p>
</div>
<p>With that granularity, I realized that the panels only started to really kick in after midday, and that I should probably move them to a spot with more exposure if I wanted to produce more than 4kWh a day (measured on a hot and sunny day without any clouds). That day, I only hit 85% efficiency though, even though I had hit 99% at some point during the previous weeks. That makes me wonder if I need to wash the panel.</p>
<p><strong>Edit</strong>: it rained that very night and I did hit 95% efficiency the next day.</p>Speeding up a 21h job to 8 minutes: a story of SQLAlchemy optimization2023-01-08T00:00:00+01:002023-01-08T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2023-01-08:/speeding-up-a-21h-job-to-8-minutes-a-story-of-sqlalchemy-optimization<p>In this article published on the <a href="https://medium.com/alan/blog-post-optimizing-our-longest-nightly-job-a-story-of-sessions-complexity-and-toilets-750ef4dfaa51">Alan tech blog</a>, we explain how my team has reduced the runtime of our longest nightly job from 21h to about 8 minutes, by using simple profiling and SQLAlchemy optimizations.</p><p>I have recently published an article on the <a href="https://medium.com/alan/blog-post-optimizing-our-longest-nightly-job-a-story-of-sessions-complexity-and-toilets-750ef4dfaa51">Alan tech blog</a> walking the reader through how we have reduced the runtime of our longest nightly job from 21 hours to about 8 minutes, by using simple profiling and SQLAlchemy optimizations.</p>
<p>Enjoy the reading!</p>My DIY Dungeons and Dragons ambiance mixer2022-09-24T00:00:00+02:002022-09-24T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2022-09-24:/my-diy-dungeons-and-dragons-ambiance-mixer<p><img src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/diy-sound-mixer/keypad.jpg"> I find that an immersive sound ambiance is key to helping tabletop RPG players engage. It can increase their stress and sense of urgency during a fight, galvanize them during a harrowing speech, or break their heart when they realize they've just lost something and there's no getting it back. In that article, I will walk you through the design of my fully DIY sound mixer, inspired by the <a href="https://novationmusic.com/en/launch/launchpad-x">Launchpad</a>, allowing me to create a great sound ambiance at the table.</p><p><img alt="keypad" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/diy-sound-mixer/keypad.jpg"></p>
<div class="toc"><span class="toctitle">Table of Contents</span><ul>
<li><a href="#getting-started">Getting started</a></li>
<li><a href="#reacting-to-key-presses">Reacting to key presses</a></li>
<li><a href="#sending-structured-data-from-the-keypad">Sending structured data from the keypad</a></li>
<li><a href="#playing-sounds-after-a-keypress">Playing sounds after a keypress</a></li>
<li><a href="#lets-rub-some-web-on-it">Let's rub some web on it</a></li>
<li><a href="#the-finishing-touch">The finishing touch</a></li>
<li><a href="#demo-time">Demo time</a></li>
<li><a href="#i-have-the-hardware-how-can-i-run-it">I have the hardware! How can I run it?</a></li>
<li><a href="#closing-words">Closing words</a></li>
</ul>
</div>
<p>I find that an immersive sound ambiance is key to helping tabletop RPG players engage. It can increase their stress and sense of urgency during a fight, galvanize them during a harrowing speech, or break their heart when they realize they've just lost something and there's no getting it back.</p>
<p>I have been thinking about using a <a href="https://novationmusic.com/en/launch/launchpad-x">Launchpad</a> to control and mix the ambiance while we play, but the more I read about its design, the less it seemed to fit. The <a href="https://store.focusrite.com/en-gb/product/launchpad-mini-mk3-/NOVLPD11~NOVLPD11">cheapest Launchpad</a> starts at 110€, and it is a full fledged MIDI controller. What I wanted was something simpler: a way to play different long sound ambiance tracks at the same time, and adjust their respective volume to create an immersive atmosphere.</p>
<p>The project started to take shape when I stumbled upon the <a href="https://shop.pimoroni.com/products/pico-rgb-keypad-base">Pimoroni RGB Keypad</a>, a 4x4 rainbow-illuminated keypad that I could program using a <a href="https://www.raspberrypi.com/products/raspberry-pi-pico/">Raspberry Pi Pico</a>, for a budget of about 30€.</p>
<p><img alt="pimoroni keypad" decoding="async" loading="lazy" src="https://shop.pimoroni.com/cdn/shop/products/pico-addons-2_768x768_crop_center.jpg"></p>
<p>The color and brightness of the LEDs under the keys is <a href="https://github.com/martinohanlon/pico-rgbkeypad">programmable</a>, meaning I could go for the look and feel of a Launchpad, while keeping my budget and the overall complexity in check.</p>
<p>The main idea would be to use 12 of the available 16 keys to start and stop audio tracks, and use the 4 remaining keys as controls (increase/decrease volume, pause all tracks).</p>
<p><img alt="idea" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/diy-sound-mixer/idea.jpg"></p>
<h2 id="getting-started">Getting started</h2>
<p>If you, like myself, want to program a Raspberry Pi Pico in Python, you have two options:</p>
<ul>
<li><a href="https://micropython.org/">MicroPython</a></li>
<li><a href="https://circuitpython.readthedocs.io/">CircuitPython</a></li>
</ul>
<p>It took me a while to figure out that these are more or less <a href="https://core-electronics.com.au/videos/circuitpython-vs-micropython-key-differences">the same</a>. In the end, I went with the CircuitPython <a href="https://learn.adafruit.com/welcome-to-circuitpython">starting-up guide</a>, and was ready to make these keys light up.</p>
<p>A CircuitPython main program lives in a <code>code.py</code> file, that is executed when the board is plugged in. Any dependency can be put under the <code>lib/</code> directory, itself placed at the root of the board filesystem.</p>
<p>I downloaded the <a href="https://github.com/martinohanlon/pico-rgbkeypad/blob/main/rgbkeypad/rgbkeypad.py"><code>rgbkeypad.py</code></a> library, placed it under <code>lib/</code> and wrote the following program in <code>code.py</code></p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">rgbkeypad</span> <span class="kn">import</span> <span class="n">RGBKeypad</span>
<span class="n">keypad</span> <span class="o">=</span> <span class="n">RGBKeypad</span><span class="p">()</span>
<span class="c1"># make all the keys red</span>
<span class="n">keypad</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="p">(</span><span class="mi">255</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="c1"># red</span>
<span class="n">keypad</span><span class="o">.</span><span class="n">brightness</span> <span class="o">=</span> <span class="mf">0.5</span>
<span class="c1"># turn a key blue when pressed</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keypad</span><span class="o">.</span><span class="n">keys</span><span class="p">:</span>
<span class="k">if</span> <span class="n">key</span><span class="o">.</span><span class="n">is_pressed</span><span class="p">():</span>
<span class="n">key</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">255</span><span class="p">)</span> <span class="c1"># blue</span>
</code></pre></div>
<p>I then copied <code>code.py</code> and <code>lib/rgbkeypad.py</code> under the <code>CIRCUITPY</code> volume that is mounted when the keypad gets plugged into the computer, and <em>voilà</em>.</p>
<p><img alt="red-blue" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/diy-sound-mixer/red-blue.webp"></p>
<h2 id="reacting-to-key-presses">Reacting to key presses</h2>
<p>Now that I knew how to program the key colors, brightness as well as knowing what keys were being pressed, I still needed a way to map these key events to starting audio tracks, and I was facing an immediate problem: the Pico has no way to play sound, even less on a Bluetooth-connected speaker. You know what can do all that really well though? My laptop.</p>
<p>So, if I could send messages from the Pico to my laptop (on which the Pico is connected for power anyway) and have a program running on my laptop receive them, I could then start thinking about how to play sounds.</p>
<p>It turns out that this was way easier than I thought, thanks to CircuitPython sending anything <code>print</code>-ed as binary data over the serial port. Using <a href="https://pyserial.readthedocs.io/en/latest/"><code>pyserial</code></a>, I can write a program that connects to the same serial port the Pico is connected to, and receive the data.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># code.py, running on the Raspberry Pi Pico</span>
<span class="kn">from</span> <span class="nn">rgbkeypad</span> <span class="kn">import</span> <span class="n">RGBKeypad</span>
<span class="n">keypad</span> <span class="o">=</span> <span class="n">RGBKeypad</span><span class="p">()</span>
<span class="c1"># make all the keys red</span>
<span class="n">keypad</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="p">(</span><span class="mi">255</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="c1"># red</span>
<span class="n">keypad</span><span class="o">.</span><span class="n">brightness</span> <span class="o">=</span> <span class="mf">0.5</span>
<span class="c1"># turn a key blue when pressed</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keypad</span><span class="o">.</span><span class="n">keys</span><span class="p">:</span>
<span class="k">if</span> <span class="n">key</span><span class="o">.</span><span class="n">is_pressed</span><span class="p">():</span>
<span class="n">key</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">255</span><span class="p">)</span> <span class="c1"># blue</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Key (</span><span class="si">{</span><span class="n">key</span><span class="o">.</span><span class="n">x</span><span class="si">}</span><span class="s2">, </span><span class="si">{</span><span class="n">key</span><span class="o">.</span><span class="n">y</span><span class="si">}</span><span class="s2"> pressed!"</span><span class="p">)</span> <span class="c1"># <-- that message will be sent over USB</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="c1"># usb_listener.py, running on the laptop</span>
<span class="kn">from</span> <span class="nn">serial</span> <span class="kn">import</span> <span class="n">Serial</span>
<span class="c1"># /dev/tty.usbmodem14201 is the name of the serial port the Pico was connected to</span>
<span class="c1"># on my mac. Your mileage may vary.</span>
<span class="n">usb_device</span> <span class="o">=</span> <span class="n">Serial</span><span class="p">(</span><span class="s2">"/dev/tty.usbmodem14201"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">usb_device</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">))</span>
</code></pre></div>
<p>I can now run <code>usb_listener.py</code> and press a key on the keypad to see the following:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>python<span class="w"> </span>usb_listener.py
<span class="go">Key (1, 0) was pressed</span>
<span class="go">Key (1, 0) was pressed</span>
<span class="go">Key (1, 0) was pressed</span>
<span class="go">...</span>
</code></pre></div>
<h2 id="sending-structured-data-from-the-keypad">Sending structured data from the keypad</h2>
<p>Sending text data is fine, but we should probably send data that can be serialized on the keypad size and deserialized on the event listener side, as we will probably send the key ID, a state (<code>pressed</code>, <code>stop</code>, <code>volume_up</code>, etc). JSON is simple enough, and while the <code>json</code> package isn't available in CircuitPython, it's pretty easy to hand-encode JSON data:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># code.py, running on the Raspberry Pi Pico</span>
<span class="kn">from</span> <span class="nn">rgbkeypad</span> <span class="kn">import</span> <span class="n">RGBKeypad</span>
<span class="n">keypad</span> <span class="o">=</span> <span class="n">RGBKeypad</span><span class="p">()</span>
<span class="c1"># make all the keys red</span>
<span class="n">keypad</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="p">(</span><span class="mi">255</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="c1"># red</span>
<span class="n">keypad</span><span class="o">.</span><span class="n">brightness</span> <span class="o">=</span> <span class="mf">0.5</span>
<span class="c1"># turn a key blue when pressed</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keypad</span><span class="o">.</span><span class="n">keys</span><span class="p">:</span>
<span class="k">if</span> <span class="n">key</span><span class="o">.</span><span class="n">is_pressed</span><span class="p">():</span>
<span class="n">key</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">255</span><span class="p">)</span> <span class="c1"># blue</span>
<span class="n">key_id</span> <span class="o">=</span> <span class="mi">4</span> <span class="o">*</span> <span class="n">key</span><span class="o">.</span><span class="n">x</span> <span class="o">+</span> <span class="n">key</span><span class="o">.</span><span class="n">y</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'</span><span class="si">{</span><span class="s2">"key"</span><span class="si">:</span><span class="s1"> %d, "state": "pressed"</span><span class="si">}</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">key_id</span><span class="p">))</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="c1"># usb_listener.py, running on the laptop</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">from</span> <span class="nn">serial</span> <span class="kn">import</span> <span class="n">Serial</span>
<span class="c1"># /dev/tty.usbmodem14201 is the name of the serial port the Pico was connected to</span>
<span class="c1"># on my mac. Your mileage may vary.</span>
<span class="n">usb_device</span> <span class="o">=</span> <span class="n">Serial</span><span class="p">(</span><span class="s2">"/dev/tty.usbmodem14201"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">usb_device</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">()))</span>
</code></pre></div>
<h2 id="playing-sounds-after-a-keypress">Playing sounds after a keypress</h2>
<p>Playing multiple sounds at the same time in Python isn't something many packages allow you to do simply. In the end, I could only make it reliably work with <a href="https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.Sound"><code>pygame</code></a>, which was developped to ease the creation of video games in Python. The package provides us with 2 different APIs to work with sound tracks:</p>
<ul>
<li><a href="https://www.pygame.org/docs/ref/music.html"><code>pygame.mixer.music</code></a>, which allows an audio track to be played while streamed. This was intended to play some background music.</li>
<li><a href="https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.Sound"><code>pygame.mixer.Sound</code></a>, which allows you to play an audio track on a specific audio channel. Mutiple <code>Sound</code>s can be played over different audio Channels.</li>
</ul>
<p>Using <code>pygame.mixer.Sound</code>, we manage to react to a keypress and start the associated audio track</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">pygame</span>
<span class="n">pygame</span><span class="o">.</span><span class="n">init</span><span class="p">()</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">from</span> <span class="nn">serial</span> <span class="kn">import</span> <span class="n">Serial</span>
<span class="kn">from</span> <span class="nn">pygame</span> <span class="kn">import</span> <span class="n">mixer</span>
<span class="kn">from</span> <span class="nn">pygame.mixer</span> <span class="kn">import</span> <span class="n">Sound</span><span class="p">,</span> <span class="n">Channel</span>
<span class="n">key_id_to_audio_tracks</span> <span class="o">=</span> <span class="p">{</span>
<span class="mi">0</span><span class="p">:</span> <span class="n">Sound</span><span class="p">(</span><span class="s2">"example0.ogg"</span><span class="p">),</span>
<span class="mi">1</span><span class="p">:</span> <span class="n">Sound</span><span class="p">(</span><span class="s2">"example1.ogg"</span><span class="p">),</span>
<span class="mi">2</span><span class="p">:</span> <span class="n">Sound</span><span class="p">(</span><span class="s2">"example2.ogg"</span><span class="p">),</span>
<span class="p">}</span>
<span class="n">channels</span> <span class="o">=</span> <span class="p">[</span><span class="n">Channel</span><span class="p">()</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="n">key_id_to_audio_tracks</span><span class="p">]</span>
<span class="n">mixer</span><span class="o">.</span><span class="n">set_num_channels</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">channels</span><span class="p">))</span>
<span class="n">usb_device</span> <span class="o">=</span> <span class="n">Serial</span><span class="p">(</span><span class="s2">"/dev/tty.usbmodem14201"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">usb_device</span><span class="p">:</span>
<span class="n">key_event</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
<span class="n">key_id</span> <span class="o">=</span> <span class="n">key_event</span><span class="p">[</span><span class="s1">'key_id'</span><span class="p">]</span>
<span class="n">sound</span> <span class="o">=</span> <span class="n">key_id_to_audio_tracks</span><span class="p">[</span><span class="n">key_id</span><span class="p">]</span>
<span class="n">channel</span> <span class="o">=</span> <span class="n">channels</span><span class="p">[</span><span class="n">key_id</span><span class="p">]</span>
<span class="n">channel</span><span class="o">.</span><span class="n">play</span><span class="p">(</span><span class="n">sound</span><span class="p">)</span> <span class="c1"># will play in the background</span>
</code></pre></div>
<p>(The actual code can be inspected <a href="https://github.com/brouberol/pico-mixer/blob/main/pico_mixer/mixer.py">here</a>).</p>
<p>While it works rather well, this approach has a fundamental issue. Because <code>mixer.Sound</code> does not support streaming the sound, as <code>mixer.music</code> does, it requires that all sounds be fully loaded in memory at startup. As the ambiance tracks that I'm planning to use all last between 30 minutes and 2h, the actual load time takes a couple of minutes. Using <code>pygame.music</code> would solve that issue, except for the fact that it only supports streaming of a single audio file at the same time.</p>
<p>I'm only left with <code>mixer.Sound</code> and loading hours of audio files in memory at startup, which means that the whole ambiance would take a lot of time to restart in case of a crash, and the energy around the table might deflate like a soufflé.</p>
<p><em>Sigh</em></p>
<p>Back to the whiteboard.</p>
<p>Alright, so, what program do I already have on my laptop that is good at streaming sound? What about an Internet browser? Youtube videos don't have to fully load before they start, and the same goes for audio files, so that might just work! I'd need a way to propagate these key events to a web page, so that it can then start/stop the audio files, change their volume, etc. Enter websockets.</p>
<h2 id="lets-rub-some-web-on-it">Let's rub some web on it</h2>
<p>The mixer would be composed of 3 different systems:</p>
<ul>
<li>the keypad, running the CircuitPython code</li>
<li>a webpage, listening for key events over a websocket, in charge of playing the audio files and adjusting their individual volume</li>
<li>an HTTP server in charge of receving the events over USB and propagating them to the websocket (ergo, to the browser), and serving the local audio files to the webpage. I'll use <code>Flask</code> and <code>Flask-sock</code> for this.</li>
</ul>
<p><img alt="new design" decoding="async" loading="lazy" src="https://user-images.githubusercontent.com/480131/190913995-a27c2385-ea1d-491a-8cc8-84a14d738a49.png"></p>
<p>So what happens now when I press a key:</p>
<ul>
<li>a JSON-formatted message is sent from the pico to the serial port</li>
<li>the message is received by the webserver process, and propagated to the browser on a websocket</li>
<li>the browser deserializes the message, and takes action, depending on the content of the event</li>
</ul>
<p>The browser-side message handler looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="nx">ws</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s1">'message'</span><span class="p">,</span><span class="w"> </span><span class="nx">event</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">keyEvent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">event</span><span class="p">.</span><span class="nx">data</span><span class="p">);</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">usbStatus</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="s2">"usb_status"</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">state</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"usb_disconnected"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">usbStatus</span><span class="p">.</span><span class="nx">textContent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"🔌 🚫"</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">state</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"usb_connected"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">usbStatus</span><span class="p">.</span><span class="nx">textContent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"🔌 ✅"</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">state</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"init"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">colorizeTracksKbdElements</span><span class="p">(</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">colors</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">state</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"pause"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">pauseAllPlayingTracks</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">state</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"unpause"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">unpauseAllPlayingTracks</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">trackProgressBar</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="sb">`progress_track_</span><span class="si">${</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">key</span><span class="si">}</span><span class="sb">`</span><span class="p">);</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">audioElement</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="sb">`audio_track_</span><span class="si">${</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">key</span><span class="si">}</span><span class="sb">`</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">audioElement</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="kc">null</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="p">(</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">state</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s2">"on"</span><span class="o">:</span>
<span class="w"> </span><span class="nx">startTrack</span><span class="p">(</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">audioElement</span><span class="p">,</span><span class="w"> </span><span class="nx">trackProgressBar</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s2">"off"</span><span class="o">:</span>
<span class="w"> </span><span class="nx">stopTrack</span><span class="p">(</span><span class="nx">keyEvent</span><span class="p">.</span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">audioElement</span><span class="p">,</span><span class="w"> </span><span class="nx">trackProgressBar</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s2">"vol_up"</span><span class="o">:</span>
<span class="w"> </span><span class="nx">increaseTrackVolume</span><span class="p">(</span><span class="nx">audioElement</span><span class="p">,</span><span class="w"> </span><span class="nx">trackProgressBar</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s2">"vol_down"</span><span class="o">:</span>
<span class="w"> </span><span class="nx">decreaseTrackVolume</span><span class="p">(</span><span class="nx">audioElement</span><span class="p">,</span><span class="w"> </span><span class="nx">trackProgressBar</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h2 id="the-finishing-touch">The finishing touch</h2>
<p>I have added a couple of features that will help me stay as focused on the storytelling as possible while I'm DMing, instead of thinking about the sound mixing process:</p>
<ul>
<li>a <a href="https://github.com/brouberol/pico-mixer/blob/main/config.json">configuration-based tagging system</a>, allowing me to get reminded of the main features for each individual track (is that an ambiance or combat music? Is it dark, light, opressing, eerie, etc?)</li>
<li>I'm also propagating the key colors to the associated volume bar, allowing me to quickly identify the key that I need to press to start/pause/adjust a given audio track.</li>
</ul>
<p><img alt="webapp" decoding="async" loading="lazy" src="https://user-images.githubusercontent.com/480131/191582090-3d54a629-ce9f-4f26-9178-f8311c55de6d.png"></p>
<div class="Note">
<p>The key colors were generated from <a href="https://medialab.github.io/iwanthue/">iwanthue</a> and are stored in the <code>COLORS</code> list, in <a href="https://github.com/brouberol/pico-mixer/blob/main/pico/code.py"><code>code.py</code></a>. Any changes to the colors will be reflected in the web UI, as they are advertised to the web-server at propagated to the UI when the keypad starts.</p>
</div>
<h2 id="demo-time">Demo time</h2>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/cdB_y9KhCgY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
<h2 id="i-have-the-hardware-how-can-i-run-it">I have the hardware! How can I run it?</h2>
<p>Getting started instructions are available <a href="https://github.com/brouberol/pico-mixer#getting-started-on-windows">here</a> for Windows users, and <a href="https://github.com/brouberol/pico-mixer#getting-started-on-macos-and-linux">here</a> for macOS and Linux users.</p>
<div class="Warning">
<p>Don't hesitate to read the comments if you have any doubt, as a fair share of questions have already be answered there.</p>
</div>
<p>Once you have everything running, you can:</p>
<ul>
<li>press one of the 12 track keys to start/stop each individual sound track</li>
<li>press the volume up/down key and a track key at the same time to increase/decrease the volume of the associated track</li>
<li>press the pause key and a track key at the same time to pause/restart the associated track</li>
<li>press the pauseAll key to pause/restart all tracks that were currently playing</li>
</ul>
<p><img alt="idea" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/diy-sound-mixer/idea.jpg"></p>
<hr>
<h2 id="closing-words">Closing words</h2>
<p>The final iteration of that project is available <a href="https://github.com/brouberol/pico-mixer/tree/main/pico">here</a> (for the keypad code) and <a href="https://github.com/brouberol/pico-mixer/tree/main/pico_mixer_web">here</a> (for the webserver and webapp code). The black casing was 3D-printed using the <code>rgb_keypad_-_bottom.stl</code> file from this <a href="https://www.thingiverse.com/thing:4883873/files">Thingiverse</a> model.</p>
<div class="Note">
<p>I am grateful to <a href="https://www.tomshardware.com/news/raspberry-pi-pico-adds-ambience-to-tabletop-adventures">tom's Hardware</a>, <a href="https://blog.adafruit.com/2022/10/05/a-raspberry-pi-pico-dd-keyboard-circuitpython-raspberrypi-keyboard-tomshardware-raspberry_pi/">Adafruit</a>, <a href="https://all3dp.com/1/best-raspberry-pi-projects/#rpg-ambiance-soundboard">all3dp</a>, <a href="https://www.hackster.io/news/balthazar-rouberol-s-raspberry-pi-pico-powered-ambience-mixer-adds-a-new-element-to-d-d-sessions-620dd9bf5c80">hackster</a>, <a href="https://game-news24.com/2022/10/05/raspberry-pi-pico-delivers-ambience-to-tabletop-adventures/">Game News 24</a>, <a href="https://www.msn.com/en-gb/money/technology/best-raspberry-pi-projects-february-2023/ar-AA1657Me">msn</a> and <a href="https://www.weareteachers.com/raspberry-pi-projects/">weareteachers</a> to have featured and shared this project to their audience.</p>
</div>Can't enough be enough?2022-05-11T00:00:00+02:002022-05-11T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2022-05-11:/cant-enough-be-enough<p>In this article, I will explain the impact the Datadog IPO and the years that followed it had on me, both financially and psychologically, as transparently as I can. The intention is to examine how such an event can change one's life, either positively or not, and give you some return of experience on the choices that I made.</p><p>I left Datadog 2 weeks ago, after 5 intense and incredible years. When I joined, we were about 300 people strong, whereas the current headcount is now approaching 4000. If you never have experienced exponential growth, this is about as close as you can get to it! This means that we were close to doubling in size each year, whether in headcount, infrastructure size, number of teams, and complexity.</p>
<p>About 2.5 years after I was hired, Datadog became a publicly traded company.</p>
<p>In this article, I will explain the impact this had on me, both financially and psychologically, as transparently as I can. The intention is to examine how such an event can change one's life, positively and not, and give you some return of experience on the choices that I made.</p>
<div class="Note">
<p>This is a weird and personal article. It is about the stock market, how stock options work, psychological paralysis, burn-out and life choices. I hope some of it can be useful to you, but really, this is also something I needed to write for my own catharsis.</p>
</div>
<h3 id="how-it-started">How it started</h3>
<p>When I joined, back in 2017, the first thing I had to do was choose between 3 compensation packages:</p>
<ul>
<li>higher salary and lower equity (36000 stock options)</li>
<li>medium salary and medium equity (48000 stock options)</li>
<li>lower salary and higher equity (60000 stock options)</li>
</ul>
<p>I chose the first one, as I didn't <em>really</em> understand what stock options were. I held the financial world pretty much in contempt, and chose what I could understand: actual money in my bank account at the end of the month. I felt that there was about a 0% chance these stock options would be worth anything anyway, so choosing the highest salary was the safest move I could make.</p>
<h3 id="wait-what-is-a-stock-option-anyway">Wait. What is a stock option anyway?</h3>
<p>Finance is fraught with lingo. Yes, possibly even more than technology. So before diving into how the stock market affected my psyche, let's try to define a couple of terms.</p>
<p>A <em>stock</em> is a financial instrument representing the ownership of a fraction of a corporation. These shares are bought and sold on <em>stock exchanges</em> (e.g. Nasdaq, Euronext, etc). For example, should I want to, I could currently buy a share of Amazon.com Inc. (referenced as <code>AMZN</code> on the market) for 2,107.44 USD on the Nasdaq, which would make me a (tiny) shareholder of Amazon. The price of said share varies according to demand and offer, basically.</p>
<p>There are multiple reasons why one should want to hold stocks: either the stocks they own give them voting rights at the company annual meetings, allowing them to influence how the company is managed, or maybe they hope to make a profit by selling at a higher price than the one they bought the stock at.</p>
<p>Now, onto stock options. A stock option is the opportunity to acquire a stock at a guaranteed reduced price. That reduced price is called the <em>strike price</em>, and should be part of your employment contract. In my case, that strike price was $0.85. That meant that should I want to acquire one of these 36000 stocks, I needed to give Datadog 85 US cent.</p>
<div class="Note">
<p>The word <em>option</em> really means that you <em>can</em> decide to purchase these stocks coming with a discount, but you don't <em>have</em> to. You have the option to do it.</p>
</div>
<p>Obviously, companies don't grant <em>all</em> stock options to their employees immediately after hiring, because new employees could decide to stay for a couple of days, pocket all their stocks and then move on. What happened in my case (which I hear is pretty common, really) is that I unlocked (the real term is <em>vest</em>) stock options according to a <em>vesting calendar</em>. I didn't get anything for a whole year, and then I unlocked (<em>vested</em>) 25% of my stocks in one go. It's called a <em>one year cliff</em>. After that, I vested 6.25% at the end of every quarter for the remaining 3 years.</p>
<p>Once a stock option was <em>vested</em>, I could then wire money to Datadog and acquire the stock at the reduced price. This is called <em>exercising</em> the stock option. At that point, the stock <em>options</em> were really converted into a stock, of which I was the owner.</p>
<p>Phew. Let's recap.</p>
<p>By staying at Datadog, I had the opportunity to regularly wire my employer money in order to acquire stocks (i.e. to become a shareholder) at a reduced price, according to a 4 year calendar, in the hope of making a profit later.</p>
<h3 id="strike-price-frisk">strike price = f(risk)</h3>
<p>The central notion here is risk. If you join a startup in its infancy, the probability of you turning a profit on your stock options is infinitesimal. To counteract the odds, you will probably get a very low strike price and many stocks, whereas if you join a company on the verge of going through its <abbr title='Initial Public Offering. Understand "woohoo I am on the stock market now".'>IPO</abbr>, you probably will be given less stocks at a much higher strike price. The reason is simple: companies want to reward employees who took the risk of buying in early.</p>
<p>In my case, I joined Datadog when the probability of an IPO was still very low, which was reflected in my strike price.</p>
<h3 id="what-will-buy-you-bread-vs-what-might-buy-you-a-house">What will buy you bread vs what might buy you a house</h3>
<p>Fast forward 2 years. There are now more and more internal rumors about a potential upcoming IPO. These rumors culminate into the subject being publicly discussed in an all-hands. We are told that we are indeed going through the IPO filing process, which could take many more months before it comes through, <em>if it does</em>. One point is hammered in: nothing is sure at that point. Everything could still fail.</p>
<p>Immediately after the announcement, a seemingly never-ending stream of questions are being raised by employees. What strikes me is every question asked by one of our American colleagues seem well-informed. Many of them seem to have gone through an IPO before, and even those who have not seem to understand how these things work.
The same cannot be said for my French colleagues and myself. We are collectively clueless. At that point, I hadn't even exercised a single stock option, as I was still fearful of committing thousands of euros in what could be a pipe dream.</p>
<p>I decide to ask one of my American teammates for advice. When I tell him that I still haven't exercised anything, he pauses for a second, and then proceeds to tell me the following.</p>
<blockquote>
<p>Look. I'm not going to tell you what to do, but here's what <strong>I</strong> do. Every time I vest, I exercise immediately after. Every time. My salary is what buys me bread. My stocks are what <strong>might</strong> buy me a house.</p>
</blockquote>
<p>After that conversation, I started to dig into the relationship between the exercise date and taxes, and proceeded to exercise everything that I had vested until then once things became clearer.</p>
<h3 id="hey-mr-taxman">Hey Mr Taxman</h3>
<div class="Warning">
<p>Everything I say here applies to my understanding of the French tax code. I am not a lawyer. Do not take this as financial advice.</p>
</div>
<p>To understand why my colleague would always exercise right after his vesting date, you first need to understand how stocks are taxed. The way this works in France is pretty similar to the way the IRS does it in the US. If you live in Cyprus, Paraguay or any other tax haven, you don't pay any tax on stocks, which is good for you and sad for your hospitals and roads.</p>
<p>There are 2 things to consider:</p>
<ul>
<li>if you exercise a stock option, you acquire a stock at a reduced price. You virtually made money there, because you should have paid more for the stock, meaning you will pay taxes on this virtual gain. This is call the <em>acquisition tax</em> ("gain d'acquisition" in French).</li>
<li>if you make a profit selling your stock, you will pay taxes on said profit. This is called <em>profit tax</em> ("gain de cession" in French).</li>
</ul>
<p>To understand how this works, let's take an example. Say my strike price is set to $1. I exercise a stock option when the value of the associated stock is $10, and I then sell it later on the market for $40.</p>
<ul>
<li>I will pay acquisition taxes on the $9 difference between the regular market price and the strike price</li>
<li>I will pay profit taxes on the $30 difference between the sell price and the regular market price at the time of the purchase</li>
</ul>
<p>This means that the sooner I exercise, the smaller the difference between the exercise and strike price should be, meaning the smaller my acquisition tax will be in the end (following the hypothesis that the stock price does nothing but grow, which was true for us for a while).</p>
<div class="Note">
<p>In the case of a stock option related to a stock that is <em>not</em> publicly traded yet (pre-IPO), the "regular market price" considered when calculating the acquisition tax is set to the stock <abbr title='Fair Market Value'>FMV</abbr>. The FMV is a theoretical price the stock <em>would</em> have, according to some independent third party appraiser, that is regularly updated.</p>
</div>
<p>In our case, the FMV was updated every quarter and did nothing but go up until the actual IPO. The initial reasoning stood: the earlier my coworker exercised, the less acquisition tax he ended up paying.</p>
<p>At that point, the FMV was at about $9 and I decided to follow his advice.</p>
<h3 id="liftoff">Liftoff</h3>
<p>This is the point in the article where I stop boring you with financial minutiae and start getting into how the IPO process affected me psychologically.</p>
<p>The IPO went really well. <code>DDOG</code> went from $27 to about $42 in a single day, and everyone celebrated. The trouble started the next day, when I configured my mac to display the stock current value in a sidebar widget. If you're at all familiar with addictology, this is where you start wincing hard.</p>
<p>I can't overstate how much of a bad idea this is. Having the feeling of "winning" or "losing" multiple times a day is addictive. The whole thing felt like a game, and I started to check my "Potential Benefit Value" in etrade several times a month. The numbers were in the 7 digits, and felt unreal.</p>
<p>Let's pause for a second, and imagine yourself sitting at a casino table. You're on a strong start, the odds are in your favor, and your chip pile grows pretty fast. Now, until you cash out, these chips are monkey money. They are worth <em>nothing</em>, and are only worth something if you take the decision to take them out of the table. You've won most of your games, so every time you lose one, you convince yourself to stay at the table to try and wait until to at least get back your losses. But then you lose some more, but hey, you should not back out now when you were winning so high not too long ago. On and on, in a loop. And so you stay at the table.</p>
<p>And that, dear reader, is why I think the casino pretty much always wins.</p>
<p>Here are a couple of things I learned in the last years, that were paramount in fighting off that psychological paralysis:</p>
<ul>
<li>The money you have invested on the market is not real money. It's worth nothing until you sell.</li>
<li>Never put money in the market you can't afford to lose.</li>
<li>Know when to check out. This means knowing what you would like to use that money for, how much your plan would require and selling when you reach it.</li>
</ul>
<h3 id="the-money-that-could-buy-bought-me-a-house">The money that <strike>could buy</strike> bought me a house</h3>
<p>In 2020, we were collectively struck by The Great Plague, and everybody was stuck inside. At that point, I realized that I had golden opportunity of being able to buy a house in the area that my fiancee and I dreamt of living in, instead of being boxed in a small flat.</p>
<p>And what happened then was... nothing. I was looking at other tech-company stocks that were benefiting from the lockdown, such as Zoom, Docusign, Shopify, etc, and they were miles ahead of where Datadog was. All I had to do was wait! (<em>rubs hands</em>).</p>
<p>This is when my fiancee kind lost it with my shenanigans, and told me that we could be living our dream today instead of waiting for.. what exactly? More money? To do what?</p>
<blockquote>
<p>Can't enough be enough?</p>
</blockquote>
<p>she told me.</p>
<p>At that point, I knew that however high the stock price was, I was going to be too paralyzed to do anything else than looking at monkey money numbers anyway. And so I estimated how much cash I'd need to cover the house as well as the acquisition and profit taxes, kept a healthy margin in case my tax estimates were wrong (remember the thing about me not being a fiscal advisor?) and for the unescapable renovation work that would need to be done, and sold about 60% of my total stocks at $66.6 (hell yeah).</p>
<p>And just like that, I had enough to afford our dream, pay the taxes on it, as well as supporting my close family.</p>
<p><img alt="the house of our dreams" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/cant-enough-be-enough/house.webp" title="The house of our dreams"></p>
<p>"Now, what about the remaining 40%?" an astute reader might ask? Well, <em>that</em> I could afford to lose, and didn't have any specific plan in mind for. They are still in the market, and are worth a pretty hefty sum of money. I didn't feel like I needed to convert them into cash for anything. If their value rises, good, if not, it might rise again, who knows?</p>
<p>And with that, I was done. Or so I thought.</p>
<h3 id="just-when-i-thought-i-was-out-they-pull-me-back-in">Just when I thought I was out, they pull me back in.</h3>
<p>Do you know what happened at the end of my 4 year vesting period? Here's what I thought was going to happen: nothing. However, what really happened was an impromptu conversation with my Director, telling me that Datadog was giving me a <em>refresher</em>, in the form of a 4 year vesting with a one-year cliff calendar for about 2000 <abbr title="Restricted Stock Unit">RSU</abbr>.</p>
<p>Aaand, back to financial minutiae just for a bit. RSU are "free stocks" the company gives you. You don't have to buy them (contrary to stock options). So you get new stocks just by staying around and doing your job. As they are way less risky than stock options, because the company has already IPO-ed, you also get less of them.</p>
<p>Where things started to get really psychologically tricky for me, is when 2 events coincided:</p>
<ul>
<li>I started to feel the symptoms of a burn-out, as I was working pretty hard, and had to deal with renovation work in the house, organizing our wedding (which was lovely, thank you very much), and various other fun things</li>
<li>For various reasons that I won't go into, I was given 2 more RSU grants, on overlapping vesting calendars</li>
</ul>
<p>After 5 years, I was at a crossroads. The more I went on, the more I felt I needed to slow down. Years of exponential growth and on-call were taxing on my mental health. I was constantly stressed out and on edge. I did not sleep well, was taking on weight and was overall losing interest in my work.</p>
<p>As I saw it, my two options were:</p>
<ul>
<li>I could stay and get more stocks, make more money, and continue working (with great and talented people!) in an ever-exponentially growing company that was promising me a promotion to Engineering Manager (which itself probably meant more stocks, less personal time and more stress), or</li>
<li>I could decide to quit, rest, slow down and do something else.</li>
</ul>
<div class="Note">
<p>I just want to be clear there. There were other options, such as going back to an IC role, that I discussed with my manager. I don't want to come across as passively dissing him. He truly was an incredible manager. But in the end, these were the 2 extreme options.</p>
</div>
<p>As I was slowly coming to the realization that option 2 was the one for me, came an extremely toxic thought. Was there a point in the near future where I'd vest a substantial amount of RSUs, after which I could then quit? The answer was yes, about 10 months from then. And thus I tried to stick around, feeling more and more depressed and disengaged, all that in the prospect of vesting stocks amounting to about $150,000.</p>
<p>Don't get me wrong, this is a <em>substantial</em> amount of money, that most people aren't privileged enough to dream about. Except that I didn't need it really. I was already living where I wanted, with my wife that I loved with all my heart. This <em>was</em> the endgame. I quickly realized that I was putting my mental health in harm's way just because I didn't want to feel like I was checking out of the table and leaving money on it. Money that I didn't really need in the first place, thanks to my remaining 40%.</p>
<p>Realizing how unhealthy that line of thinking was, I settled on option #2, negotiated a 2-month leaving period (the legal one in France is 3 months), after which I said good-bye to all the wonderful people I had been lucky to work with for years.</p>
<p>On my last day, my "Potential Benefit Value" in etrade was at about $1.2M. I left it all on the table.</p>
<p>And you know what? I'm happy. Enough was enough.</p>Measuring the coverage of a rust program in Github Actions2022-04-26T00:00:00+02:002022-04-26T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2022-04-26:/measuring-the-coverage-of-a-rust-program-in-github-actions<p>In this article, I will go through how I set up code coverage measurement for <code>bo</code>, my text editor written in Rust, and publicly hosted the coverage report on S3.</p><p>After having faced a couple of of regressions in <a href="https://github.com/brouberol/bo"><code>bo</code></a> (my personal text editor <a href="/metaprocrastinating-on-writing-a-book-by-writing-a-text-editor">written in Rust</a>) in the past couple of days, I have tried to increase the number of unit tests related to the codebase sections handling navigation. I already had some unit tests, but I needed to know what lines of code were <em>not</em> tested, to know what area of the codebase I needed to focus on.</p>
<p>To do this, I used Mozilla's excellent <a href="https://github.com/mozilla/grcov"><code>grcov</code></a> project. I followed their instructions and ran the following commands locally, in my work directory.</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span><span class="nb">export</span><span class="w"> </span><span class="nv">RUSTFLAGS</span><span class="o">=</span><span class="s2">"-Cinstrument-coverage"</span>
<span class="gp">$ </span>cargo<span class="w"> </span>build
<span class="gp">$ </span><span class="nb">export</span><span class="w"> </span><span class="nv">LLVM_PROFILE_FILE</span><span class="o">=</span><span class="s2">"bo-%p-%m.profraw"</span>
<span class="gp">$ </span>cargo<span class="w"> </span><span class="nb">test</span>
<span class="gp">$ </span>grcov<span class="w"> </span>.<span class="w"> </span>-s<span class="w"> </span>.<span class="w"> </span>--binary-path<span class="w"> </span>./target/debug/<span class="w"> </span>-t<span class="w"> </span>html<span class="w"> </span>--branch<span class="w"> </span>--ignore-not-existing<span class="w"> </span>-o<span class="w"> </span>./target/debug/coverage/
<span class="gp">$ </span>open<span class="w"> </span>./target/debug/coverage/index.html
</code></pre></div>
<p>This way, I got a beautiful HTML report in which I could see my code coverage, either global, file by file,</p>
<p><img alt="Coverage report" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/rust-coverage/cov.webp"></p>
<p>or line by line.</p>
<p><img alt="Coverage report" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/rust-coverage/cov2.webp"></p>
<p><code>grcov</code> even generates nice SVG badges displaying the coverage score, that I could display on the project homepage!</p>
<p>What I ultimately wanted though, was to have every commit touching my <code>main</code> branch to trigger a new coverage generation report, that I could host somewhere public and read at leisure when I needed to.</p>
<p>To do so, I set-up a publicly accessible s3 bucket, configured to host a static website, which turns out to be remarkably easy to do <a href="https://github.com/brouberol/infrastructure/commit/75192443319f36cfbdfbcee0086322c958e3cc82#diff-abe63f10056054dcb55782e4be3ccb2ec28b47e6192b3ee1b45e46ff1884738aR62-R74">in terraform</a>:</p>
<div class="highlight"><pre><span></span><code><span class="kr">resource</span><span class="w"> </span><span class="nc">"aws_s3_bucket"</span><span class="w"> </span><span class="nv">"github-brouberol-coverage"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">bucket</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"my-bucket-name"</span>
<span class="w"> </span><span class="na">provider</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">aws.euwest</span>
<span class="w"> </span><span class="na">acl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"public-read"</span>
<span class="w"> </span><span class="na">force_destroy</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="no">false</span>
<span class="w"> </span><span class="nb">versioning</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">enabled</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="no">false</span>
<span class="w"> </span><span class="na">mfa_delete</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="no">false</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="nb">website</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">index_document</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"index.html"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<div class="Note">
<p>There are other ways to host the HTML files than S3 (such as <a href="https://pages.github.com/">Github Pages</a>), and you do <em>not</em> have you terraform to do it, but I so happen to have a <a href="https://github.com/brouberol/infrastructure/tree/master/terraform">terraform codebase</a> for my personal infrastructure, which made it a no-brainer. If you decide do host the files another way, feel free to jump <a href="#github-secrets">ahead</a>.</p>
</div>
<p>I then created an AWS user, associated with an AWS access_key/secret_key pair and the following IAM policy, granting that user read/write permissions on that S3 bucket, and nothing else.</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"Version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2012-10-17"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Statement"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"Sid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"VisualEditor0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Effect"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Allow"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Action"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"s3:PutObject"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"s3:GetObjectAcl"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"s3:GetObject"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"s3:ListBucket"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"s3:DeleteObject"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"s3:PutObjectAcl"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"Resource"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"arn:aws:s3:::<my-bucket-name>"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"arn:aws:s3:::<my-bucket-name>/*"</span>
<span class="w"> </span><span class="p">]</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<div id="github-secrets"></div>
<p>I then had to store the bucket name, keypair and AWS region name as encrypted secrets in the <code>bo</code> <a href="https://github.com/brouberol/bo">repository</a>, by going to <code>Settings > Secrets > Actions > New repository secret</code>.</p>
<p><img alt="Secrets" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/rust-coverage/secrets.webp"></p>
<p>Once that was all set up, the project CI (Github Actions) needed to perform the <a href="https://github.com/brouberol/bo/blob/main/.github/workflows/tests.yml#L28-L77">following actions</a>:</p>
<ul>
<li>Checking out the project and setting up a nightly rust toolchain</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">actions/checkout@v2</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Setup toolchain</span>
<span class="w"> </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">actions-rs/toolchain@v1</span>
<span class="w"> </span><span class="nt">with</span><span class="p">:</span>
<span class="w"> </span><span class="nt">toolchain</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nightly</span>
<span class="w"> </span><span class="nt">override</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">profile</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minimal</span>
</code></pre></div>
<ul>
<li>running the unit tests with profiling and coverage collection enabled</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Run tests</span>
<span class="w"> </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">actions-rs/cargo@v1</span>
<span class="w"> </span><span class="nt">with</span><span class="p">:</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test</span>
<span class="w"> </span><span class="nt">args</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">--all-features --no-fail-fast</span><span class="w"> </span><span class="c1"># Customize args for your own needs</span>
<span class="w"> </span><span class="nt">env</span><span class="p">:</span>
<span class="w"> </span><span class="nt">CARGO_INCREMENTAL</span><span class="p">:</span><span class="w"> </span><span class="s">'0'</span>
<span class="w"> </span><span class="nt">RUSTFLAGS</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span>
<span class="w"> </span><span class="no">-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code</span>
<span class="w"> </span><span class="no">-Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests -Cinstrument-coverage</span>
<span class="w"> </span><span class="nt">RUSTDOCFLAGS</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span>
<span class="w"> </span><span class="no">-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code</span>
<span class="w"> </span><span class="no">-Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests -Cinstrument-coverage'</span>
</code></pre></div>
<ul>
<li>generating the coverage report using <code>grcov</code>, using the <a href="https://github.com/actions-rs/grcov/"><code>actions-rs/grcov</code></a> action.</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Gather coverage data</span>
<span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">coverage</span>
<span class="w"> </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">actions-rs/grcov@v0.1</span>
</code></pre></div>
<ul>
<li>measuring the total coverage score, and report it in a check, if the job is associated to a pull request</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Report coverage in PR status for the current commit</span>
<span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">github.ref_name != 'main'</span>
<span class="w"> </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span>
<span class="w"> </span><span class="no">set -eu</span>
<span class="w"> </span><span class="no">total=$(cat ${COV_REPORT_DIR}/badges/flat.svg | egrep '<title>coverage: ' | cut -d: -f 2 | cut -d% -f 1 | sed 's/ //g')</span>
<span class="w"> </span><span class="no">curl -s "https://brouberol:${GITHUB_TOKEN}@api.github.com/repos/brouberol/bo/statuses/${COMMIT_SHA}" -d "{\"state\": \"success\",\"target_url\": \"https://github.com/brouberol/bo/pull/${PULL_NUMBER}/checks?check_run_id=${RUN_ID}\",\"description\": \"${total}%\",\"context\": \"Measured coverage\"}"</span>
<span class="w"> </span><span class="nt">env</span><span class="p">:</span>
<span class="w"> </span><span class="nt">GITHUB_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ secrets.GITHUB_TOKEN }}</span>
<span class="w"> </span><span class="nt">COMMIT_SHA</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ github.event.pull_request.head.sha }}</span>
<span class="w"> </span><span class="nt">RUN_ID</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ github.run_id }}</span>
<span class="w"> </span><span class="nt">PULL_NUMBER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ github.event.pull_request.number }}</span>
<span class="w"> </span><span class="nt">COV_REPORT_DIR</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ steps.coverage.outputs.report }}</span>
</code></pre></div>
<p><img alt="Secrets" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/rust-coverage/cov3.webp"></p>
<ul>
<li>uploading the whole HTML coverage report to S3, using the <a href="https://github.com/jakejarvis/s3-sync-action">jakejarvis/s3-sync-action</a> action. We only do this for commits belonging the <code>main</code> branch (<em>i.e.</em> direct pushes or after a pull request was merged).</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s">"Upload</span><span class="nv"> </span><span class="s">the</span><span class="nv"> </span><span class="s">HTML</span><span class="nv"> </span><span class="s">coverage</span><span class="nv"> </span><span class="s">report</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">S3"</span>
<span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">github.ref_name == 'main'</span>
<span class="w"> </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jakejarvis/s3-sync-action@master</span>
<span class="w"> </span><span class="nt">with</span><span class="p">:</span>
<span class="w"> </span><span class="nt">args</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">--acl public-read --follow-symlinks --delete</span>
<span class="w"> </span><span class="nt">env</span><span class="p">:</span>
<span class="w"> </span><span class="nt">AWS_S3_BUCKET</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ secrets.AWS_BUCKET }}</span>
<span class="w"> </span><span class="nt">AWS_ACCESS_KEY_ID</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ secrets.AWS_ACCESS_KEY_ID }}</span>
<span class="w"> </span><span class="nt">AWS_SECRET_ACCESS_KEY</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ secrets.AWS_SECRET_ACCESS_KEY }}</span>
<span class="w"> </span><span class="nt">AWS_REGION</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ secrets.AWS_REGION }}</span>
<span class="w"> </span><span class="nt">SOURCE_DIR</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${{ steps.coverage.outputs.report }}</span>
<span class="w"> </span><span class="nt">DEST_DIR</span><span class="p">:</span><span class="w"> </span><span class="s">'bo'</span>
</code></pre></div>
<p>With all of that set up, the coverage report is now <a href="http://github-brouberol-coverage.s3-website.eu-west-3.amazonaws.com/bo/main">publicly available</a>, refreshed every time a new commit hits <code>main</code>, and I even get a coverage shield for free! <img alt="coverage shield" decoding="async" loading="lazy" src="https://github-brouberol-coverage.s3.eu-west-3.amazonaws.com/bo/main/badges/flat.svg"></p>Tools I'm thankful for2022-02-22T00:00:00+01:002022-02-22T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2022-02-22:/tools-im-thankful-for<p>Software engineers sometimes have a reputation for being overly critical when it comes to tools and programming languages. The web is full of rants, heated debates and articles about what technology is "better" and which is "crap". It was thus refreshing to read an post titled <a href="https://www.jowanza.com/blog/2022/2/21/software-im-thankful-for"><em>Software I'm thankful for</em></a>, that shone a light on some pieces of software in a positive light. In honor of this article, I've decided to go through the same exercise.</p><p>Software engineers sometimes have a reputation of being overly critical when it comes to tools and programming languages. The web is full of rants, heated debates and articles about what technology is "better" and which is "crap". It was thus refreshing to read an post titled <a href="https://www.jowanza.com/blog/2022/2/21/software-im-thankful-for"><em>Software I'm thankful for</em></a>, that shone a light on some pieces of software in a positive light. In honor of this article, I've decided to go through the same exercise.</p>
<h2 id="python">Python</h2>
<p><a href="https://python.org">Python</a> was my gateway to becoming a software engineer. It was the first programming language I <em>loved</em>, and I still do to this day.
I wrote Python code professionally for a an AI startup, an e-ticketing startup, the Scottish government, a global hosting provider, a huge observability SaaS. I've written large Python webapps and quick Python scripts. I've written large asynchronous task workflows processing payments, trained machine learning models, written self-documented REST APIs, found my house listing by scraping the web, <a href="/river-monitoring-with-datadog">monitor the level of the river close by</a>, all of that in Python.</p>
<p>I also write Python code to maintain my own <a href="https://github.com/brouberol/infrastructure">infrastructure</a>, that I deploy via <a href="https://docs.ansible.com/">ansible</a>, itself written in Python. This blog is generated via <a href="https://pelican.readthedocs.org">Pelican</a>, which is written in Python. I've started to play with a Raspberry Pi Pico, that I program in ... <a href="http://docs.circuitpython.org/en/latest/README.html">CircuitPython</a>. It's ubiquitous, and I've heard it be called "The second best tool for every job", meaning that it probably won't be the most performant tool for what you're working on, but you'll make progress really fast.</p>
<p>Learning and programming Python has taught me many programming concepts, such as object-oriented programming, functional programming, unit testing, dataclasses, metaprogramming, REST APIs, HTTP, JSON, etc.</p>
<p>I however now realize that it also allowed me to get introduced to <em>lower-level</em> concepts, such as ioctl, sockets, system calls, file descriptors, etc, through the reassuring lens of the Python standard library, instead of having to interact with these concepts in C, which was much more intimidating (and still is today).</p>
<h2 id="docker">Docker</h2>
<p>The first time I was introduced to <a href="https://docs.docker.com/">Docker</a> was at a Python meetup in Lyon, circa 2013. After the 30 minute long presentation, I still had no clue as to what any of it meant and why I'd ever need it and pretty much shrugged it off. As the Docker ecosystem flourished and the dust settled, I started to understand the appeal.</p>
<p>Do you need to run redis to prototype against? Just run <code>docker run redis</code> and <em>voila</em>. Do you want to run <code>calibre-web</code> on your local VPS without having to install its dependencies in your system libraries? <a href="https://github.com/brouberol/infrastructure/blob/0e2ece50b45bc998cfc09dff1dc002c96f91cdee/playbooks/roles/gallifrey/calibre/tasks/main.yml#L10-L26">Sure</a>.</p>
<p>Docker allowed me to self-host a collection of tools that I use every day, package and run applications in extremely large production environments, spin up development environments without having to pollute my system libraries. It boosted my productivity and became part of my day-to-day workflow. None of these are the <em>real</em> reason why I'm thankful for Docker.</p>
<p>I've seen many companies break down their monolith into dockerized microservices. The commonly invoked reasons are allowing teams to chose their own language for each project, and helping the horizontal scaling of some load-critical apps. As useful Docker was to start a single container, it didn't solve the issue of starting several containers that could communicate with each other on a single host. Enter <a href="https://docs.docker.com/compose/">docker-compose</a>, which in turn didn't solve the issue of orchestrating containers on a fleet of nodes. Enter <a href="https://mesosphere.github.io/marathon/">Mesos/Marathon</a>, <a href="https://docs.docker.com/engine/swarm/">Docker Swarm</a>, <a href="https://kubernetes.io">Kubernetes</a>, <a href="https://aws.amazon.com/fr/ecs/">Amazon ECS</a>, etc.</p>
<p>The beef I have with Docker is that the hype around its <em>ecosystem</em> caused small companies to onboard immense amount of complexity from the absolute get-go, to help with recruiting. Because engineers want to build experience with Kubernetes, these companies find themselves dividing their attention between grappling with its inherent complexity, distributed tracing, image recycling policies, RBAC, etc, and building their actual core value. </p>
<p>This is why I'm thankful for Docker and its ecosystem. I believe I've seen situations in which it truly was critically useful, and I'll now be able to differentiate between situations in which we need it, and situations in which we only wished we did. </p>
<h2 id="raspberry-pi">Raspberry Pi</h2>
<p>Before I joined OVH, the <em>only</em> sysadmin experience I had was tinkering with my <a href="https://www.raspberrypi.com/products/raspberry-pi-4-model-b/">Raspberry Pi</a>. Thanks to that 35$ matchbox-sized computer, I got to learn iptables and systemd, port forwarding, ssh hardening, file system checks and repairs. But really, the crucial point is that I was able to learn all that by making mistakes. I'd rather learn about why you need to be careful with <code>iptables -j DROP</code> in the comfort of my own home than in a production, high pressure, environment. I can't stress the impact that learning without the fear of public failure had on me. </p>
<p>I'm now getting into electronics through the <a href="https://www.raspberrypi.com/products/raspberry-pi-pico/">Raspberry Pi Pico</a>, which opens a whole new exploration and tinkering domain for me!</p>
<h2 id="the-terminal">The terminal</h2>
<p>The terminal is a truly important part of my day as a software engineer. It's really what allows me to feel in control. Like Python, it became a familiar tool in which I could discover entirely new domains, interact with new systems and concepts. I learned so much from it that I decided to help people out <a href="/category/essential-tools-and-practices-for-the-aspiring-software-developer">getting familiarized with the terminal and the shell</a>. </p>Sending a webhook from Synology DSM to Discord2022-01-17T00:00:00+01:002022-01-17T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2022-01-17:/sending-a-webhook-from-synology-dsm-to-discord<p>Given the fact that running a Datadog agent on a Synology Play NAS is not obvious, I wanted to enable Discord webhooks push notifications (as this is where my Datadog alerts are already being sent). This way, I'd get plenty of alerts "for free" without having to configure new Datadog …</p><p>Given the fact that running a Datadog agent on a Synology Play NAS is not obvious, I wanted to enable Discord webhooks push notifications (as this is where my Datadog alerts are already being sent). This way, I'd get plenty of alerts "for free" without having to configure new Datadog monitors.</p>
<p>While sending webhooks notifications from a Synology NAS to Discord is technically possible, the DSM UI somehow seems to prevent us from doing so, as documented in this <a href="https://www.synoforum.com/threads/webhooks-to-post-alerts-messages-on-to-discord.6725/#post-32618">forum thread</a>. Somehow, we <em>have</em> to include a <code>hello world</code> message in the notification, as part of the message content, without which, the UI won't allow us to save the webhook configuration.</p>
<p>You can however circumvent the issue by <code>ssh</code>-ing into the NAS and edit the <code>/usr/syno/etc/synowebhook.conf</code> into this:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"Discord"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"needssl"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"port"</span><span class="p">:</span><span class="w"> </span><span class="mi">8090</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"prefix"</span><span class="p">:</span><span class="w"> </span><span class="s2">"A new system event occurred on your %HOSTNAME%"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"req_header"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"req_method"</span><span class="p">:</span><span class="w"> </span><span class="s2">"post"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"req_param"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{\"username\":\"Synology\", \"avatar_url\": \"https://play-lh.googleusercontent.com/HjbYWbXJ-6e6Cia-mBbHDSdontW1yE6MHMaXqlHW80CQegDOEPQ1HGACxvEpnqCUHgo\", \"embeds\": [{\"description\": \"@@TEXT@@\", \"title\": \"@@PREFIX@@\"}]}"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"sepchar"</span><span class="p">:</span><span class="w"> </span><span class="s2">" "</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"template"</span><span class="p">:</span><span class="w"> </span><span class="s2">"$webhook_url"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"custom"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"$webhook_url"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<div class="Note">
<p><b>Note</b>: replace <code>$webhook_url</code> by your Discord webhook URL.</p>
</div>
<p>When this is done, you should see a <code>Discord</code> webhook in your Webhook Push Services, and you should now be able to send a test message to Discord!</p>
<p><picture>
<source srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/syno-discord/dark/discord-notif.webp" media="(prefers-color-scheme: dark)">
<img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/syno-discord/light/discord-notif.webp">
</picture></p>
<p>Now, any warning or alert generated from DSM will automatically be sent to Discord as well!</p>River monitoring with Datadog2021-11-02T00:00:00+01:002021-11-02T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2021-11-02:/river-monitoring-with-datadog<p>Last month, Ardèche experienced <em>very</em> heavy precipitations in the span of couple of hours. As a result, the dam located upriver from me opened the floodgates (literally), which caused the Chassezac level to raise by about 6.5m in about 1.5h. I've setup some monitoring using Datadog and Pagerduty to make sure I know about it as soon as possible.</p><p style="text-align:center;font-style:italic;font-size:0.8em;">water level (m) over time</p>
<p>Last month, Ardèche experienced <em>very</em> heavy precipitations in the span of couple of hours. As a result, the dam located upriver from me opened the floodgates (literally), which caused the Chassezac level to raise by about 6.5m in about 1.5h. My basement was completely flooded, and the water level stabilized just about a 1m from the house ground floor. We had just enough time to move our belongings to the first floor. The riverside was unrecognizable, to the point where we found fish in the trees.</p>
<p>3 weeks later, the same thing happened, but this time the dam managers did their job. They only let enough water to make the dam wasn't overrun, while keeping everyone safe downriver.</p>
<p>What really bothered me though, is that at no point were we alerted of anything by EDF (the company managing the grid). No text, to alert, nothing.</p>
<p>Datadog to the rescue.</p>
<p>Using custom scripts, I now measure the <a href="https://github.com/brouberol/infrastructure/blob/master/playbooks/roles/gallifrey/monitoring/templates/monitor_rivers">river level</a> at the station before and after my house. I also keep tabs on the <a href="https://github.com/brouberol/infrastructure/blob/master/playbooks/roles/gallifrey/monitoring/templates/monitor_rain">amount of rain</a> measured at these stations, as well as the general alert level.</p>
<p><a target="blank" href="https://p.datadoghq.com/sb/bc352bb82-c122f0855899cdbcc73f2ca478d6d7b6"><picture>
<source srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/river-monitoring/river-monitoring-dark.webp"
media="(prefers-color-scheme: dark)">
<img class=dark src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/river-monitoring/river-monitoring-light.webp" />
</picture></a></p>
<p>By "chance", the first flood stopped right before the house level, and the second one stopped right before the basement. By extrapolating just a bit, I'm now able to have a good idea of the impact of a flood by looking at the river level at the station upriver.</p>
<p>I thus created Datadog monitors over the <a href="https://github.com/brouberol/infrastructure/blob/0d4fa0ca629b852e2e3cbff2d7e2ea0701135371/terraform/datadog/monitors.tf#L217-L239">river level</a> and the <a href="https://github.com/brouberol/infrastructure/blob/0d4fa0ca629b852e2e3cbff2d7e2ea0701135371/terraform/datadog/monitors.tf#L193-L215">alert level</a>, and I hooked them to a personal <a href="https://pagerduty.com">Pagerduty</a> account, using their free tier.</p>
<p>I made sure to enable <code>Critical Alerts for High Urgency</code> in the app settings, which enables Pagerduty to override my phone volume preference, to wake me up even if is is in silent mode.</p>
<p>Now, if the dam managers decide to open the gates during the night (it has happened), I'll know.</p>
<p><img alt="pagerduty" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/river-monitoring/pagerduty.webp"></p>To the Underdark and back2021-10-21T00:00:00+02:002021-10-21T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2021-10-21:/to-the-underdark-and-back<p>I've recently designed a 2 session long (6h) detour into the Underdark, that would feed into one of my player's character's backstory. The goal was to allow him to meet his long-disappeared father, while introducing both the players and the characters to the strange and dangerous land that is the Underdark.</p><p>I've recently designed a 2 session long (6h) detour into the Underdark, that would feed into one of my player's character's backstory. The goal was to allow her to meet her long-disappeared father, while introducing both the players and the characters to the strange and dangerous land that is the Underdark.</p>
<p>The way I prepared these sessions was an interesting process. I wanted these sessions to be mostly focused on exploration and roleplay, with a single (intense) fight, as well as a puzzle. I tried to design a sandboxed environement with enough lore and backstory to make sure the players enjoy themselves and have a reason to interact with the NPCs. I wanted them to care and have the necessary space and freedom to express themselves.</p>
<p>Following are my session design notes, that lasted me 2 whole sessions. These were as much a way to create the world as reminders about key elements or creature capabilities that I should remember mid-fight. They ended up being quite short, because I tried really hard to paint a picture, and prepare some colorful moments, but not to anticipate my player's reactions. <em>They</em> mostly filled the gaps and brought life to that setting.</p>
<p><em><a id=lang-switcher>Click here to switch to the <span id=lang-switcher-flag>🇫🇷</span> version.</a></em></p>
<p><picture>
<source srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/dark/1.png" media="(prefers-color-scheme: dark)">
<img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/light/1.png">
</picture>
<picture>
<source srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/dark/2.png" media="(prefers-color-scheme: dark)">
<img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/light/2.png">
</picture>
<picture>
<source srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/dark/3.png" media="(prefers-color-scheme: dark)">
<img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/light/3.png">
</picture>
<picture>
<source srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/dark/4.png" media="(prefers-color-scheme: dark)">
<img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/light/4.png">
</picture>
<picture>
<source srcset="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/dark/5.png" media="(prefers-color-scheme: dark)">
<img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/underdark/en/light/5.png">
</picture></p>
<script>
const langSwitcher = document.querySelector('#lang-switcher');
const flag = document.querySelector('#lang-switcher-flag');
const pictures = document.getElementsByTagName("picture");
var currentLang = "en";
function toggleLangInUrl(url, currentLang) {
if (currentLang == "fr") {
return url.replace("/fr/", "/en/");
} else {
return url.replace("/en/", "/fr/");
}
}
function toggleCurrentLang(currentLang) {
if (currentLang == "fr") {
return "en";
}
return "fr";
}
langSwitcher.addEventListener('click', event => {
for (i=0; i<pictures.length; i++) {
pic = pictures[i];
source = pic.getElementsByTagName("source")[0];
img = pic.getElementsByTagName("img")[0];
source.srcset = toggleLangInUrl(source.srcset, currentLang);
img.src = toggleLangInUrl(img.src, currentLang)
}
if (currentLang == "en") {
flag.textContent = "🇬🇧";
} else {
flag.textContent = "🇫🇷";
}
currentLang = toggleCurrentLang(currentLang);
});
</script>Metaprocrastinating on writing a book by writing a text editor2021-09-04T00:00:00+02:002021-09-04T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2021-09-04:/metaprocrastinating-on-writing-a-book-by-writing-a-text-editor<p>If you have been following my <a href="https://blog.balthazar-rouberol.com/category/essential-tools-and-practices-for-the-aspiring-software-developer">Essential Tools and Practices for the Aspiring Software Developer</a> posts and were anxious to read more, you might have noticed that they stopped coming after a while. I have a draft for the last chapter, and I regularly think about getting back to it, at least to get some closure. Alas, procrastination being what it is, I never did. My procrastination level became really interesting when I convinced myself that one of the reasons that I didn't want to write that final chapter was that my text editor was standing in the way. I was either using a full-fledged code editor (Sublime Text/VSCode) riddled with complex features I didn't need (autocompletion, linting, etc) or getting lost in configuring <code>vim</code> into the perfect markdown editor. Either way, these were the wrong tools for the job, and my only way to get back to writing was to.. write my own?</p><p>If you have been following my <a href="https://blog.balthazar-rouberol.com/category/essential-tools-and-practices-for-the-aspiring-software-developer">Essential Tools and Practices for the Aspiring Software Developer</a> posts and were anxious to read more, you might have noticed that they stopped coming after a while. I have a draft for the last chapter, and I regularly think about getting back to it, at least to get some closure. Alas, procrastination being what it is, I never did.</p>
<p>My procrastination level became really interesting when I convinced myself that one of the reasons that I didn't want to write that final chapter was that my text editor was standing in the way. I was either using a full-fledged code editor (Sublime Text/VSCode) riddled with complex features I didn't need (autocompletion, linting, etc) or getting lost in configuring <code>vim</code> into the perfect markdown editor. Either way, these were the wrong tools for the job, and my only way to get back to writing was to.. write my own?</p>
<p>And thus, <a href="https://github.com/brouberol/bo"><code>bo</code></a> was born.</p>
<video controls>
<source src="https://user-images.githubusercontent.com/480131/131999617-61acc5a2-4055-4cd1-9da1-134ee9e075b4.mp4" type="video/mp4">
</video>
<p>The idea was to create a simple text editor, with powerful <code>vim</code>-like navigation. It should allow me to write in a very simple interface,
while being able to navigate through the text in a couple of keystrokes, leveraging the muscle memory I built over the years using <code>vim</code> (or the vim mode in various editors). </p>
<p>I wanted it to be written in Rust, as it would be a good opportunity for me to write non-trivial code in a safe language, and also because, well, it just sounded fun.</p>
<p>I've been working on it on and off in the last month, and I've implemented enough features so that it's starting to feel comfortable.</p>
<p><img alt="bo-help" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/metaprocrastination-bo/bo-help.webp"></p>
<p>There's still <a href="https://github.com/brouberol/bo/issues">a lot to do</a>! I'd be delighted if you wanted to test it and give it a go!</p>
<p><em>Written with <code>bo</code>.</em></p>Cleaning up the Dungeondraft tag list2021-08-31T00:00:00+02:002021-08-31T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2021-08-31:/cleaning-up-the-dungeondraft-tag-list<p>I have spent quite a lot of time using <a href="https://dungeondraft.net">Dungeondraft</a> recently, as I've designed many homebrewed places and encounters. The more maps I created, the more assets pack I bought from <a href="https://cartographyassets.com">CartographyAssets</a>, to further enrich and improve them. I quickly started to realize that some of these asset packs caused the tag list to be filled with entries that weren't linked to any assets at all. This made the asset discovery process quite frustrating.</p><p>I have spent quite a lot of time using <a href="https://dungeondraft.net">Dungeondraft</a> recently, as I've designed many homebrewed places and encounters.
The more maps I created, the more assets pack I bought from <a href="https://cartographyassets.com">CartographyAssets</a>, to further enrich and improve them.
I quickly started to realize that some of these asset packs caused the tag list to be filled with entries that weren't linked to any assets at all. This made the asset discovery process quite frustrating.</p>
<p><img alt="board" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/cleaning-up-dungeondraft-tag-list/empty-assets.webp"></p>
<p>Luckily, I found out about <a href="https://github.com/Ryex/Dungeondraft-GoPackager"><code>Dungeondraft-GoPackager</code></a>, a tool that allows anyone to unpack a Dungeondraft asset pack, and inspect its metadata. I discovered that some asset packs would ship with tag entries linked to an empty asset list:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>dungeondraft-unpack<span class="w"> </span>2M<span class="se">\ </span>Forest<span class="se">\ </span>Floor<span class="se">\ </span>Assets.dungeondraft_pack<span class="w"> </span>.
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>2M<span class="se">\ </span>Forest<span class="se">\ </span>Floor<span class="se">\ </span>Assets
$<span class="w"> </span>cat<span class="w"> </span>data/default.dungeondraft_tags<span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.
<span class="c1"># ... snip</span>
<span class="w"> </span><span class="s2">"Magic"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Mattresses"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Mill"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Mine"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Mirror"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Molds and Stains"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Mushroom"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Obstacle"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="s2">"textures/objects/forestfloor_cliff_1.webp"</span>,
<span class="w"> </span><span class="s2">"textures/objects/forestfloor_cliff_2.webp"</span>,
<span class="w"> </span><span class="s2">"textures/objects/forestfloor_cliff_3.webp"</span>
<span class="w"> </span><span class="o">]</span>,
<span class="w"> </span><span class="s2">"Ocean"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Ottomans"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Paddles"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Paper and Books"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Pillar"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Pillows"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Pine Trees"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"Planks and Debris"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="c1"># ...</span>
</code></pre></div>
<p>I suspect the author does that because they export the same list of tag for each asset pack they release. However, having only bought a couple, that caused my tag list to be pretty spotty.</p>
<p>I decided to create a script that would automate the process of unpacking asset packs, removing these empty metadata entries, and then repacking everything up. While the process is pretty simple conceptually, it can become tedious when the number of packs grows.</p>
<p>The result of that work is <a href="https://github.com/brouberol/cleanup-dungeondraft-asset-packs"><code>cleanup-dungeondraft-asset-packs</code></a>, that you can install by running the following command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pip3<span class="w"> </span>install<span class="w"> </span>--user<span class="w"> </span>cleanup-dungeondraft-asset-packs
Collecting<span class="w"> </span>cleanup-dungeondraft-asset-packs
<span class="w"> </span>Downloading<span class="w"> </span>cleanup_dungeondraft_asset_packs-0.1.0-py3-none-any.whl<span class="w"> </span><span class="o">(</span><span class="m">4</span>.2<span class="w"> </span>kB<span class="o">)</span>
Installing<span class="w"> </span>collected<span class="w"> </span>packages:<span class="w"> </span>cleanup-dungeondraft-asset-packs
Successfully<span class="w"> </span>installed<span class="w"> </span>cleanup-dungeondraft-asset-packs-0.1.0
</code></pre></div>
<p>Once installed, you just have to point it to your assets directory, and <em>voilà</em>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cleanup-dungeondraft-asset-packs<span class="w"> </span>--assets-dir<span class="w"> </span>~/Documents/DnD/DungeonDraft/Assets
INFO:root:Unpacking<span class="w"> </span>/Users/br/Documents/DnD/DungeonDraft/Assets/CH-Forest-Demo.dungeondraft_pack
INFO:root:Repacking<span class="w"> </span>tmp/CH-Forest-Demo
WARN<span class="o">[</span><span class="m">0000</span><span class="o">]</span><span class="w"> </span>overwriting<span class="w"> </span>file<span class="w"> </span><span class="nv">id</span><span class="o">=</span>kt201FMq<span class="w"> </span><span class="nv">name</span><span class="o">=</span><span class="s2">"CH - Forest Demo"</span><span class="w"> </span><span class="nv">outPackagePath</span><span class="o">=</span><span class="s2">"/Users/br/Documents/DnD/DungeonDraft/Assets/cleaned/CH - Forest Demo.dungeondraft_pack"</span><span class="w"> </span><span class="nv">path</span><span class="o">=</span>/Users/br/Documents/DnD/DungeonDraft/Assets/tmp/CH-Forest-Demo
INFO:root:Unpacking<span class="w"> </span>/Users/br/Documents/DnD/DungeonDraft/Assets/AS-Forest-apmh1i.dungeondraft_pack
INFO:root:Skipping,<span class="w"> </span>as<span class="w"> </span>no<span class="w"> </span>dungeondraft_tags<span class="w"> </span>file<span class="w"> </span>is<span class="w"> </span>found
INFO:root:Repacking<span class="w"> </span>tmp/AS-Forest-apmh1i
WARN<span class="o">[</span><span class="m">0000</span><span class="o">]</span><span class="w"> </span>overwriting<span class="w"> </span>file<span class="w"> </span><span class="nv">id</span><span class="o">=</span>qxhzAkxg<span class="w"> </span><span class="nv">name</span><span class="o">=</span><span class="s2">"AS Forest"</span><span class="w"> </span><span class="nv">outPackagePath</span><span class="o">=</span><span class="s2">"/Users/br/Documents/DnD/DungeonDraft/Assets/cleaned/AS Forest.dungeondraft_pack"</span><span class="w"> </span><span class="nv">path</span><span class="o">=</span>/Users/br/Documents/DnD/DungeonDraft/Assets/tmp/AS-Forest-apmh1i
INFO:root:Unpacking<span class="w"> </span>/Users/br/Documents/DnD/DungeonDraft/Assets/2M<span class="w"> </span>Forest<span class="w"> </span>Floor<span class="w"> </span>Assets.dungeondraft_pack
INFO:root:Skipping<span class="w"> </span>empty<span class="w"> </span>tag<span class="w"> </span>Administration
INFO:root:Skipping<span class="w"> </span>empty<span class="w"> </span>tag<span class="w"> </span>Animals
INFO:root:Skipping<span class="w"> </span>empty<span class="w"> </span>tag<span class="w"> </span>Armchairs
INFO:root:Skipping<span class="w"> </span>empty<span class="w"> </span>tag<span class="w"> </span>Armor
...
</code></pre></div>
<p>Now, re-open Dungeondraft, and point it to the <code>cleaned</code> directory that <code>cleanup-dungeondraft-asset-packs</code> created, in which it placed all cleaned assets.</p>
<p><img alt="cleaned-assets" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/cleaning-up-dungeondraft-tag-list/dungeondraft-assets-cleaned.webp"></p>
<p>At that point, your tag list should only contain entries linked to <em>actual</em> assets!</p>
<p><img alt="cleaned-up-taglist" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/cleaning-up-dungeondraft-tag-list/cleaned-up-taglist.webp"></p>
<p>There you go, I hope that helps! Happy Dungeondrafting!</p>Running the Port Nyanzaru Dinosaur Race2021-04-10T00:00:00+02:002021-04-10T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2021-04-10:/running-the-port-nyanzaru-dinosaur-race<p>When I was preparing for Port Nyanzaru, in <a href="https://dnd.wizards.com/products/tabletop-games/rpg-products/tomb-annihilation">Tomb of Annihilation</a>, I started reading what other Dungeons Masters had to say about the city. A lot of them would mention that the dinosaur race was a must-do, and that if done properly, it could really be a high point in the start of the adventure. The problem was, I felt that the official rules regarding this race were, well, underwhelming, to say the least. Each player rolls a dice, gets some points or not, repeatedly until the end of the race. If that race was going to be something to remember, I felt that I needed to spice it up a bit.</p><p>When I was preparing for Port Nyanzaru, in <a href="https://dnd.wizards.com/products/tabletop-games/rpg-products/tomb-annihilation">Tomb of Annihilation</a>, I started reading what other Dungeons Masters had to say about the city. A lot of them would mention that the dinosaur race was a must-do, and that if done properly, it could really be a high point in the start of the adventure. The problem was, I felt that the official rules regarding this race were, well, underwhelming, to say the least. Each player rolls a dice, gets some points or not, repeatedly until the end of the race. If that race was going to be something to remember, I felt that I needed to spice it up a bit.</p>
<p>The way I designed the race was as a mix between the official rules, the Game of the Goose and Mario Kart. You win if you are the first to complete 2 full laps around the city. Each lap is made of 48 squares, and starts/finishes at the Coliseum, marked with an X.</p>
<p><img alt="board" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/nyanzaru-race/nyanzaru-race.webp"></p>
<p>The players roll initiative to determine the order in which they'll play. We however consider that they all move at the same time, meaning that if 2 dinosaurs cross the finish line during the same round, they'll be considered ex aequo.</p>
<p>A player's turn goes as follows:</p>
<ul>
<li>the jockey rolls an animal handling check against the dinosaur's DC to see if they can control it<ul>
<li>if successful, the dinosaur moves using its high speed dice</li>
<li>if unsuccessful, the dinosaur moves its low speed dice. The jockey could however choose to hit its mount with its whip to coerce it into running faster (using its high speed dice).<ul>
<li>The dinosaur needs to make a successful CON DD10 check for that.</li>
<li>If unsuccessful, it moves at half speed for the rest of the turn.</li>
<li>For the more aggressive dinosaurs (the ones marked with an asterisk), if the CON check failed by more than 5 points, it stops moving for 2 rounds, in protest.</li>
</ul>
</li>
</ul>
</li>
<li>If a dinosaur moves through or stops on a red square (on a bridge), it could attempt to trip another dinosaur located on the same square.<ul>
<li>If the other dinosaur fails a DD 10 DEX check, it's considered prone for a turn.</li>
</ul>
</li>
<li>If a dinosaur stops (or <em>chooses</em> to stop) on a blue square, the jockey can decide to pick up some loot box. These boxes can have positive or negative effects, either instantaneous or to possibly be used later, anytime during the player's turn (think Mario Kart loot boxes).</li>
</ul>
<p>I've kept the same dinosaur stats as given in the book, and used the official <a href="https://5e.tools/bestiary.html#velociraptor_vgm">stats block</a> for the Velociraptor.</p>
<table>
<thead>
<tr>
<th>Dinosaur</th>
<th>Race</th>
<th>Jockey <abbr title="Wisdom">WIS</abbr></th>
<th>Low speed dice</th>
<th>High speed dice</th>
<th>Animal Handling DC</th>
<th><abbr title="Constitution">CON</abbr></th>
<th><abbr title="Dexterity">DEX</abbr></th>
</tr>
</thead>
<tbody>
<tr>
<td>Un Tej et l'Addition</td>
<td>Triceratops</td>
<td>14(+2)</td>
<td>1d6</td>
<td>1d4+6</td>
<td>14</td>
<td>15(+2)</td>
<td>9(-1)</td>
</tr>
<tr>
<td>Aubrion du Gers</td>
<td>Hadrosaurus</td>
<td>12(+1)</td>
<td>1d6</td>
<td>1d2+6</td>
<td>10</td>
<td>13(+1)</td>
<td>10(0)</td>
</tr>
<tr>
<td>Mambo Mambo King of Tango</td>
<td>Tyrannosaurus</td>
<td>17(+3)</td>
<td>1d6</td>
<td>1d6+6</td>
<td>18*</td>
<td>17(+3)</td>
<td>10(0)</td>
</tr>
<tr>
<td>Brigadier Gérard</td>
<td>Dimetrodon</td>
<td>13(+1)</td>
<td>1d4</td>
<td>1d4+4</td>
<td>8</td>
<td>15(+2)</td>
<td>12(+1)</td>
</tr>
<tr>
<td>Fanfreluche</td>
<td>Allosaurus</td>
<td>16(+2)</td>
<td>1d6</td>
<td>1d4+6</td>
<td>16*</td>
<td>15(+2)</td>
<td>13(+1)</td>
</tr>
<tr>
<td>Pourquoi il pleure?</td>
<td>Deinonychus</td>
<td>17(+3)</td>
<td>1d6</td>
<td>1d2+6</td>
<td>12*</td>
<td>14(+2)</td>
<td>15(+2)</td>
</tr>
<tr>
<td>Excelsior VII</td>
<td>Ankylosaurus</td>
<td>16(+2)</td>
<td>1d4</td>
<td>1d6+4</td>
<td>13</td>
<td>16(+2)</td>
<td>11(+0)</td>
</tr>
<tr>
<td>Irène</td>
<td>Velociraptor</td>
<td>15(+2)</td>
<td>1d8</td>
<td>1d2+8</td>
<td>12</td>
<td>13(+1)</td>
<td>14(+2)</td>
</tr>
</tbody>
</table>
<p>Here are the loot boxes that I came up with.</p>
<table>
<thead>
<tr>
<th>Effect</th>
<th>Instantaneous?</th>
</tr>
</thead>
<tbody>
<tr>
<td>An insect swarm scares off your mount. Your next move will use your low speed dice.</td>
<td>Yes</td>
</tr>
<tr>
<td>You find a juicy spider. When given to your mount, it will run using its high speed dice.</td>
<td>No</td>
</tr>
<tr>
<td>You injure yourself on an hallucinogenic vine. Your next animal handling check will be performed at a disadvantage.</td>
<td>Yes</td>
</tr>
<tr>
<td>A reflex potion, when consumed, will give you an advantage at the next DEX check.</td>
<td>No</td>
</tr>
<tr>
<td>This net will allow you to immobilize an adversary located on the same square than you during a whole turn if they fail a DEX check DC 12.</td>
<td>No</td>
</tr>
<tr>
<td>This blessing potion will allow you to add 1d4 to your next skill check or saving throw.</td>
<td>No</td>
</tr>
<tr>
<td>These beads allow you to trip all dinosaurs located on the same square than you or the square before you. A dinosaur trips if it fails a DEX check DC 13. In case of failure, its speed is divided by 2 during its next turn.</td>
<td>No</td>
</tr>
<tr>
<td>A blinding bomb explodes in your face. If you fail a WIS saving throw DC 13, your next 2 animal handling checks will be performed at a disadvantage.</td>
<td>Yes</td>
</tr>
<tr>
<td>An appetizing chicken heart will allow you to relaunch your speed dice, after consumption.</td>
<td>No</td>
</tr>
<tr>
<td>You get teleported on the same square than the penultimate dinosaur.</td>
<td>Yes</td>
</tr>
</tbody>
</table>
<p>Each player had to pay 20 gold to enter the race. The first finisher gets 100 gold, the second one gets 50 gold and the third one gets 20 gold. The players are obviously free to bet on anything they like, and the DM is responsible for giving them appropriate odds.</p>
<p>I hope these rules will help you run a fun race, or at least give you ideas to create your own set of rules! Feel free to tell me what worked and what didn't if you ran with these!</p>
<div class="Note">
<p>For those of you using Foundry, Steve Vlaminck has created a <a href="https://gitlab.com/mikedave/foundryvtt-macros/-/tree/master/dino-racing">plugin</a> implementing those very rules!</p>
</div>Shell productivity tips and tricks2020-04-24T00:00:00+02:002020-04-24T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2020-04-24:/shell-productivity-tips-and-tricks<p>This chapter will walk you through different features of your shell allowing you to do more while typing less, such as autocompletion, keyboard shortcuts, history navigation and shell expansions. Even mastering <em>some</em> of these should make you immensely more productive in your shell day-to-day!</p><header>
<p>
This article is part of a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>
If you are interested in the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</header>
<h2 id="table-of-contents">Table of Contents</h2>
<!-- MarkdownTOC autolink="true" levels="2" autoanchor="true" -->
<ul>
<li><a href="#tab-completion">Tab completion</a></li>
<li><a href="#keyboard-shortcuts">Keyboard shortcuts</a></li>
<li><a href="#navigating-through-history">Navigating through history</a></li>
<li><a href="#shell-expansions">Shell expansions</a></li>
<li><a href="#real-life-examples">Real-life examples</a></li>
<li><a href="#summary">Summary</a></li>
<li><a href="#going-further">Going further</a></li>
</ul>
<!-- /MarkdownTOC -->
<h1 id="shell-productivity-tips">Shell productivity tips</h1>
<p>I estimate that I spend around 50% of my day working in my text editor
and my terminal. Any way I can get more productive in these environments
has a direct and measurable impact on my daily productivity as a whole.</p>
<p>If you spend a good chunk of your day repeatedly hitting the left and
right arrow keys to navigate in long commands or correct typos, or
hitting the up or down arrow keys to navigate your command history, this
chapter should help you get more done quicker. We will cover some shell
features you can leverage to make your shell do more of the work for
you.</p>
<p>On a personal level, I probably use some of these up to 30 times a day,
sometimes even without thinking about it, and it gives me a real sense
of ownership of my tool.</p>
<p>In the immortal words of Kimberly “Sweet Brown” Wilkins:</p>
<blockquote>
<p>Ain't nobody got time for that.</p>
</blockquote>
<p><a id="tab-completion"></a></p>
<h2 id="tab-completion">Tab completion</h2>
<p>When you are typing in your shell, I suggest you treat the
<kbd>Tab</kbd> key as a superpower. Indeed, the same way your phone
keyboard can autocomplete words for you, so can your shell. It can
suggest completions of command names and even command arguments or
options! This works by pressing <kbd>Tab</kbd> (twice for <code>bash</code> and
once for <code>zsh</code>).</p>
<div class="Note">
<p>One of the reasons <code>zsh</code> might be favored over <code>bash</code> is its more
powerful auto-completion system, giving more results out-of-the-box and
allowing you to navigate through the auto-completion options.</p>
</div>
<p>Here is an example of <code>bash</code> auto-completing a command name:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkd<span class="k"><</span>Tab>
mkdep<span class="w"> </span><span class="nb">mkdir</span>
</code></pre></div>
<p>Here is an example of <code>bash</code> auto-completing a command argument:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">man</span><span class="w"> </span>mkd<span class="k"><</span>Tab>
<span class="nb">mkdir</span><span class="w"> </span>mkdirat<span class="w"> </span>mkdtemp<span class="w"> </span>mkdtempat_np
</code></pre></div>
<p>And finally, an example of <code>bash</code> auto-completing a command option:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">python</span><span class="w"> </span>-<span class="k"><</span>Tab>
-<span class="w"> </span>-3<span class="w"> </span>-B<span class="w"> </span>-E<span class="w"> </span>-O<span class="w"> </span>-OO<span class="w"> </span>-Q<span class="w"> </span>-R<span class="w"> </span>-S<span class="w"> </span>-V<span class="w"> </span>-W
-b<span class="w"> </span>-c<span class="w"> </span>-d<span class="w"> </span>-h<span class="w"> </span>-i<span class="w"> </span>-m<span class="w"> </span>-s<span class="w"> </span>-t<span class="w"> </span>-u<span class="w"> </span>-v<span class="w"> </span>-x
</code></pre></div>
<p>I suggest you get used to using auto-completion as much as possible. It
can save you keystrokes, as well as make you discover command options
you didn't know about.</p>
<p>Pro-tip: if you are using bash, you can get install the
<code>bash-completion</code><sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup> package (using your system package-manager) in
order to enable auto-completion for a wide variety of commands that do
not support it out-of-the-box.</p>
<p><a id="keyboard-shortcuts"></a></p>
<h2 id="keyboard-shortcuts">Keyboard shortcuts</h2>
<p>The shell uses a library called <code>readline</code><sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup> to provide you with many
keyboard shortcuts to navigate, edit, cut, paste, search, etc, in the
command line. Mastering these will help to dramatically increase your
efficiency, instead of copying and pasting with your mouse, and
navigating the command with the <kbd>↑</kbd> and <kbd>↓</kbd> arrow
keys.</p>
<p>The default shortcuts are inspired by the <code>emacs</code><sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup> terminal-based
text editor. If you are already familiar with it, a lot of the default
<code>readline</code> shortcuts might feel familiar. <code>emacs</code> isn't the only famous
text editor in the history of computers though: another one, dating back
from 1976, is <code>vi</code>.<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup> <code>vi</code> and <code>emacs</code> are designed in two very
different ways, and have two very different logics. It is possible that
one might “click” more than the other for you. If you happen to be
familiar with the <code>vi</code> editor and are accustomed to its navigation
system, you can replicate it in your shell as well by adding <code>set -o vi</code>
in your shell configuration file. If you are using <code>zsh</code> with the Oh My
Zsh framework that we introduced in the previous chapter, you can also
use the <code>vi-mode</code> plugin to do this.</p>
<p>The advantage of using the same navigation logic and shortcuts in your
text editor and your terminal is that is blurs the line between both,
and brings consistency to your terminal environment. If you have no clue
how <code>emacs</code> or <code>vi</code> work though, I would probably suggest you don't
worry about all this for now and experiment with the default terminal
shortcuts.</p>
<h3 id="navigating-the-current-line">Navigating the current line</h3>
<p>The following navigation shortcuts allow you to move quickly your cursor
in the current command saving you from relying solely on the
<kbd>→</kbd> and <kbd>←</kbd> arrows.</p>
<table>
<thead>
<tr>
<th>Navigation</th>
<th style="text-align: right;">Shortcut</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go to beginning of line</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>A</kbd></td>
</tr>
<tr>
<td>Go to end of line</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>E</kbd></td>
</tr>
<tr>
<td>Go to next word</td>
<td style="text-align: right;"><kbd>Alt</kbd> - <kbd>F</kbd></td>
</tr>
<tr>
<td>Go to previous word</td>
<td style="text-align: right;"><kbd>Alt</kbd> - <kbd>B</kbd></td>
</tr>
<tr>
<td>Toggle your cursor between its current position and the beginning of line</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>X</kbd> - <kbd>X</kbd></td>
</tr>
</tbody>
</table>
<p>If you however prefer using the <code>vi</code> navigation system, you will first
need to type <kbd>Esc</kbd> to switch from the <em>Insertion</em> mode to an
emulation of <code>vi</code>'s <em>normal</em> mode, in which you can navigate in your
text using the following shortcuts:</p>
<table>
<thead>
<tr>
<th>Navigation</th>
<th style="text-align: right;">Shortcut</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go to beginning of line</td>
<td style="text-align: right;"><kbd>^</kbd></td>
</tr>
<tr>
<td>Go to end of line</td>
<td style="text-align: right;"><kbd>$</kbd></td>
</tr>
<tr>
<td>Go to next word</td>
<td style="text-align: right;"><kbd>w</kbd></td>
</tr>
<tr>
<td>Go to previous word</td>
<td style="text-align: right;"><kbd>b</kbd></td>
</tr>
<tr>
<td>Move to the end of the previous word</td>
<td style="text-align: right;"><kbd>e</kbd></td>
</tr>
</tbody>
</table>
<p>You can go back to editing your command line by hitting the <code>i</code> key.</p>
<h3 id="deleting-and-editing-text">Deleting and editing text</h3>
<p>These shortcuts allow you to quickly edit the current command more
efficiently than by just using the <kbd>Delete</kbd> key.</p>
<table>
<thead>
<tr>
<th>Edition</th>
<th style="text-align: right;">Shortcut</th>
</tr>
</thead>
<tbody>
<tr>
<td>Delete current character</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>D</kbd></td>
</tr>
<tr>
<td>Delete previous word</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>W</kbd></td>
</tr>
<tr>
<td>Delete next word</td>
<td style="text-align: right;"><kbd>Alt</kbd> - <kbd>D</kbd></td>
</tr>
<tr>
<td>Edit the current command in your text editor</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>X</kbd> <kbd>Ctrl</kbd> - <kbd>E</kbd></td>
</tr>
<tr>
<td>Undo previous action(s)</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>-</kbd></td>
</tr>
</tbody>
</table>
<p>The equivalent <code>vi</code>-style shortcuts are:</p>
<table>
<thead>
<tr>
<th>Edition</th>
<th style="text-align: right;">Shortcut</th>
</tr>
</thead>
<tbody>
<tr>
<td>Replace current character by another (ex: <em>e</em>)</td>
<td style="text-align: right;"><kbd>r</kbd> - <kbd>e</kbd></td>
</tr>
<tr>
<td>Delete current character</td>
<td style="text-align: right;"><kbd>x</kbd></td>
</tr>
<tr>
<td>Delete previous word</td>
<td style="text-align: right;"><kbd>d</kbd> - <kbd>b</kbd></td>
</tr>
<tr>
<td>Delete next word</td>
<td style="text-align: right;"><kbd>d</kbd> - <kbd>w</kbd></td>
</tr>
<tr>
<td>Edit the current command in your text editor</td>
<td style="text-align: right;"><kbd>v</kbd></td>
</tr>
<tr>
<td>Undo previous action(s)</td>
<td style="text-align: right;"><kbd>u</kbd></td>
</tr>
</tbody>
</table>
<h3 id="cutting-and-pasting">Cutting and pasting</h3>
<p>The shell provides you with shortcuts to cut and paste commands quickly
without using your mouse.</p>
<table>
<thead>
<tr>
<th>Action</th>
<th style="text-align: right;">Shortcut</th>
</tr>
</thead>
<tbody>
<tr>
<td>Cut current word before the cursor</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>W</kbd></td>
</tr>
<tr>
<td>Cut from cursor to end of line</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>K</kbd></td>
</tr>
<tr>
<td>Cut from cursor to start of line</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>U</kbd></td>
</tr>
<tr>
<td>Paste the cut buffer at current position</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>Y</kbd></td>
</tr>
</tbody>
</table>
<p>The equivalent <code>vi</code>-style shortcuts are:</p>
<table>
<thead>
<tr>
<th>Action</th>
<th style="text-align: right;">Shortcut</th>
</tr>
</thead>
<tbody>
<tr>
<td>Cut current word before the cursor</td>
<td style="text-align: right;"><kbd>d</kbd> - <kbd>w</kbd></td>
</tr>
<tr>
<td>Cut from cursor to end of line</td>
<td style="text-align: right;"><kbd>d</kbd> - <kbd>$</kbd></td>
</tr>
<tr>
<td>Cut from cursor to start of line</td>
<td style="text-align: right;"><kbd>d</kbd> - <kbd>^</kbd></td>
</tr>
<tr>
<td>Paste the cut buffer at current position</td>
<td style="text-align: right;"><kbd>p</kbd></td>
</tr>
</tbody>
</table>
<h3 id="controlling-the-terminal">Controlling the terminal</h3>
<p>Finally, these shortcuts will let you interact with the terminal itself.</p>
<table>
<thead>
<tr>
<th>Action</th>
<th style="text-align: right;">Shortcut</th>
<th style="text-align: right;">Equivalent command</th>
</tr>
</thead>
<tbody>
<tr>
<td>Clear the terminal screen</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>L</kbd></td>
<td style="text-align: right;"><code>clear</code></td>
</tr>
<tr>
<td>Close the terminal screen</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>D</kbd></td>
<td style="text-align: right;"><code>exit</code></td>
</tr>
<tr>
<td>Send current command to the background.</td>
<td style="text-align: right;"><kbd>Ctrl</kbd> - <kbd>Z</kbd></td>
<td style="text-align: right;"></td>
</tr>
</tbody>
</table>
<p>Even mastering <em>some</em> of these shortcuts should make you immensely more
productive at typing commands and navigating command-line interfaces. I
suggest you take time to experiment until you feel more accustomed with
them. I can guarantee that you will feel the productivity boost!</p>
<h3 id="a-unified-command-line-editing-experience">A unified command-line editing experience</h3>
<p>These shortcuts do not just work in your shell, but in any application
using the <code>readline</code> library to allow the user to type and edit
commands. Learning these shortcuts will thus make you productive in all
types of command lines that you might encounter in your career, such as
<code>python</code>, <code>irb</code>, <code>sqlite3</code>, etc.</p>
<p>To make sure you get a smooth and homogeneous editing experience in all
command lines you use in your system, you can set your preferred mode in
the <code>readline</code> configuration file itself.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>~/.inputrc
<span class="nb">set</span><span class="w"> </span>editing-mode<span class="w"> </span>vi<span class="w"> </span><span class="c1"># or emacs</span>
</code></pre></div>
<p><a id="navigating-through-history"></a></p>
<h2 id="navigating-through-history">Navigating through history</h2>
<p>If you find yourself typing a certain command times and times again, you
should probably be aware of how to navigate and search your shell
history, in order to save time and keystrokes.</p>
<p>While the obvious way to re-execute a previous command might seem to
just bash on the <kbd>↑</kbd> key until you find the command you want,
there are faster and smarter ways to accomplish this.</p>
<h3 id="searching-the-history">Searching the history</h3>
<p>A very useful and time-saving trick is searching for a command into your
shell history instead of re-typing it from scratch. You can search your
command history by typing <kbd>Ctrl</kbd> - <kbd>R</kbd> which opens a
<code>reverse-i-search</code> (backwards search) prompt, in which you can search
for previously executed command containing a given search pattern.</p>
<p>Type <kbd>Ctrl</kbd> - <kbd>R</kbd> to navigate through the results,
until you find the one you were looking for and type the
<kbd>Enter</kbd> key to execute it.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="k"><</span>Ctrl-R>
<span class="o">(</span>reverse-i-search<span class="o">)</span>:<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="k"><</span>Ctrl-R><span class="w"> </span><span class="k"><</span>Enter>
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"hello world"</span>
hello<span class="w"> </span>world
</code></pre></div>
<p>If you want to stop the search, either hit
<kbd>Ctrl</kbd> - <kbd>C</kbd> or <kbd>Ctrl</kbd> - <kbd>G</kbd> to be
sent back into the regular shell prompt.</p>
<p>History search works by looking into the shell history file
(<code>~/.bash_history</code> for <code>bash</code> and <code>~/.zsh_history</code> for <code>zsh</code> by
default). Every time you execute a command, it will be added to your
shell history file (with a maximum number of retained commands defined
by the <code>HISTSIZE</code> environment variable).</p>
<div class="Note">
<p>The location of your shell history file can be configured by setting the
<code>HISTFILE</code> environment variable.</p>
</div>
<h3 id="rewriting-history">Rewriting history</h3>
<p>If you want to remove a sensitive command from your history, you can
simply edit your <code>$HISTFILE</code> history file and remove it.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>secret-command<span class="w"> </span>--password<span class="w"> </span>1234qwerty<span class="w"> </span><span class="c1"># oh no! that should not be in my history!</span>
$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>secret-command<span class="w"> </span><span class="nv">$HISTFILE</span>
secret-command<span class="w"> </span>--password<span class="w"> </span>1234qwerty
$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'/secret-command/d'</span><span class="w"> </span><span class="nv">$HISTFILE</span><span class="w"> </span><span class="c1"># deletion of history line containing 'secret-command'</span>
$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>secret-command<span class="w"> </span><span class="nv">$HISTFILE</span>
$<span class="w"> </span><span class="c1"># it's not in history anymore</span>
</code></pre></div>
<p>You can also use the <code>history</code> built-in command to display your whole
history</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">5</span>
<span class="w"> </span><span class="m">496</span><span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span><span class="nb">test</span>
<span class="w"> </span><span class="m">497</span><span class="w"> </span>secret-command<span class="w"> </span>--password<span class="w"> </span>1234qwerty
<span class="w"> </span><span class="m">498</span><span class="w"> </span><span class="nb">cd</span>
<span class="w"> </span><span class="m">499</span><span class="w"> </span><span class="nb">man</span><span class="w"> </span><span class="nb">history</span>
<span class="w"> </span><span class="m">500</span><span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">5</span>
</code></pre></div>
<p>Each history line is prefixed by its index in the history. You can then
use <code>history -d <index></code> to remove the associated line from history.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">history</span><span class="w"> </span>-d<span class="w"> </span><span class="m">497</span>
$<span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">7</span>
<span class="w"> </span><span class="m">496</span><span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span><span class="nb">test</span>
<span class="w"> </span><span class="m">497</span><span class="w"> </span><span class="nb">cd</span>
<span class="w"> </span><span class="m">498</span><span class="w"> </span><span class="nb">man</span><span class="w"> </span><span class="nb">history</span>
<span class="w"> </span><span class="m">499</span><span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">5</span>
<span class="w"> </span><span class="m">500</span><span class="w"> </span><span class="nb">history</span><span class="w"> </span>-d<span class="w"> </span><span class="m">497</span>
<span class="w"> </span><span class="m">501</span><span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">7</span>
</code></pre></div>
<div class="Note">
<p>This only works with <code>bash</code>, not <code>zsh</code>.</p>
</div>
<h3 id="avoiding-history">Avoiding history</h3>
<p>There is a trick you can use if you want to fly under the radar and
never have a command recorded in history in the first place. Simply
prefix your command by a space.</p>
<div class="Note">
<p>If you are using <code>zsh</code>, you need to add <code>setopt HIST_IGNORE_SPACE</code> in
your <code>~/.zshrc</code> to make sure that behavior is enabled.</p>
</div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>secret-command<span class="w"> </span>--password<span class="w"> </span>1234qwerty<span class="w"> </span><span class="c1"># notice the space at the start of the command!</span>
$<span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">2</span>
<span class="w"> </span><span class="m">502</span><span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">7</span>
<span class="w"> </span><span class="m">503</span><span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">2</span>
</code></pre></div>
<p><a id="shell-expansions"></a></p>
<h2 id="shell-expansions">Shell expansions</h2>
<p>The shell can perform expansions, meaning it can replace portions of the
command before executing it. Relying on expansions allows you to type
less and rely on the shell itself to do the heavy lifting. While there
are multiple types of expansions, we will only cover 5:</p>
<ul>
<li>history expansion: quickly access previous commands and arguments
from history</li>
<li>tilde expansion: replace the <code>~</code> path prefix</li>
<li>pathname expansion: expand a path pattern into a list of files</li>
<li>braces expansion: expand a pattern between braces into a longer
sequence</li>
<li>command expansion: replace a sub-command by its output</li>
</ul>
<p>Expansions are extremely powerful. When used right, an expansion can
literally save you from writing a script.</p>
<div class="Note">
<p>As we only over what we think are the most useful expansions and
shortcuts, feel free to refer to the <code>bash</code> manual, section <code>EXPANSION</code>
if you want to see the full list.</p>
</div>
<h3 id="history-expansion">History expansion</h3>
<p>Your shell has multiple tricks up its sleeve to allow you to quickly
reference previous commands or arguments in history with a minimum of
keystrokes. While this section only provides you with what we feel are
the most useful of them, feel free to go to the <code>HISTORY EXPANSION</code>
section of the <code>bash</code> manual.</p>
<h4 id="event-designators">Event designators</h4>
<p>An <em>Event designator</em> is a reference to a command line entry in the
history list. It allows you to quickly refer to a previous command
without having to re-type it.</p>
<h5 id="-n"><code>!-n</code></h5>
<p><code>!-n</code> refers to the nth latest command: <code>!-1</code> refers to the latest
command, <code>!-2</code> to the command before that, etc.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"hello world!"</span>
hello<span class="w"> </span>world!
$<span class="w"> </span><span class="nb">cd</span>
$<span class="w"> </span>!-2<span class="w"> </span><span class="c1"># !-1 is "cd" and !-2 is 'echo "hello world!"'</span>
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"hello world"</span>
hello<span class="w"> </span>world
</code></pre></div>
<p><code>!!</code> is a shortcut for <code>!-1</code>, aka the latest command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"hello world!"</span>
hello<span class="w"> </span>world!
$<span class="w"> </span><span class="k">!!</span>
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"hello world"</span>
hello<span class="w"> </span>world
</code></pre></div>
<div class="Note">
<p><code>!!</code> is oftentimes used in conjunction with <code>sudo</code>, to re-execute the
previous command with superuser privileges when it failed, due to a lack
of permission.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">vim</span><span class="w"> </span>/etc/myfile
vim:<span class="w"> </span>/etc/myfile:<span class="w"> </span>Permission<span class="w"> </span>denied
$<span class="w"> </span><span class="nb">sudo</span><span class="w"> </span><span class="k">!!</span>
$<span class="w"> </span><span class="nb">sudo</span><span class="w"> </span><span class="nb">vim</span><span class="w"> </span>/etc/myfile
</code></pre></div>
</div>
<h5 id="string1string2"><code>^string1^string2</code></h5>
<p><code>^string1^string2</code> is used to repeat the previous command in which
<code>string1</code> is replaced by <code>string2</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>./myfile
Just<span class="w"> </span>a<span class="w"> </span>file<span class="w"> </span>full<span class="w"> </span>of<span class="w"> </span>junk
$<span class="w"> </span>^cat^rm
$<span class="w"> </span><span class="nb">rm</span><span class="w"> </span>./myfile
</code></pre></div>
<p>I personally use and abuse of this technique when I'm about to
irremediably delete some resources (files, folders, containers, etc),
and I want to make sure I'm about to delete the <em>right</em> things by
listing these resources first. If you are familiar with SQL queries, it
is the equivalent of executing a <code>SELECT</code> query before changing the
<code>SELECT</code> to <code>DELETE</code> to make sure you're not going to delete more than
you wanted to.</p>
<h4 id="word-designators">Word designators</h4>
<p>Word designators are used to select desired words from a previous
command (by default, the latest). They can be very useful when you want
to type a new command that uses arguments previously typed in a previous
command.</p>
<h5 id="_1"><code>!^</code></h5>
<p><code>!^</code> maps to the first argument of your latest command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">touch</span><span class="w"> </span>first.txt<span class="w"> </span>second.txt<span class="w"> </span>last.txt
$<span class="w"> </span><span class="nb">vim</span><span class="w"> </span><span class="k">!^</span>
$<span class="w"> </span><span class="nb">vim</span><span class="w"> </span>first.txt
</code></pre></div>
<h5 id="_2"><code>!$</code></h5>
<p><code>!$</code> maps to the last argument of your latest command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">touch</span><span class="w"> </span>first.txt<span class="w"> </span>second.txt<span class="w"> </span>last.txt
$<span class="w"> </span><span class="nb">vim</span><span class="w"> </span>!$
$<span class="w"> </span><span class="nb">vim</span><span class="w"> </span>last.txt
</code></pre></div>
<h5 id="combining-event-and-word-designators">Combining event and word designators</h5>
<p>You can even combine event and word designators in more complex shapes
by using the following syntax</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="n">EVENT</span><span class="w"> </span><span class="n">DESIGNATOR</span><span class="p">]</span><span class="o">:</span><span class="p">[</span><span class="n">WORD</span><span class="w"> </span><span class="n">DESIGNATOR</span><span class="p">]</span>
</code></pre></div>
<p>For example, you could use the <code>!!</code> event designator to select the last
command, and the <code>2</code> word designator to select the second argument.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">touch</span><span class="w"> </span>first.txt<span class="w"> </span>second.txt<span class="w"> </span>last.txt
$<span class="w"> </span><span class="nb">vim</span><span class="w"> </span>!!:2
$<span class="w"> </span><span class="nb">vim</span><span class="w"> </span>second.txt
</code></pre></div>
<h3 id="tilde-expansion">Tilde expansion</h3>
<p>For each unquoted word starting with <code>~</code> in the command, all characters
preceding a forward slash (<code>/</code>) will be considered a <em>tilde prefix</em>.
Depending on its actual value, the tilde prefix can be expanded several
ways, although the simple <code>~</code> is probably its most common use.</p>
<table>
<thead>
<tr>
<th>Tilde prefix</th>
<th style="text-align: right;">Expansion</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>~</code></td>
<td style="text-align: right;">Your home directory</td>
</tr>
<tr>
<td><code>~+</code></td>
<td style="text-align: right;">Your current working directory</td>
</tr>
<tr>
<td><code>~-</code></td>
<td style="text-align: right;">Your previous working directory</td>
</tr>
</tbody>
</table>
<p><strong>Example</strong></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span><span class="k">~</span>
Android<span class="w"> </span>code<span class="w"> </span>Downloads<span class="w"> </span>Music
AndroidStudioProjects<span class="w"> </span>Desktop<span class="w"> </span>Dropbox<span class="w"> </span>Pictures
bin<span class="w"> </span>Documents<span class="w"> </span>Firefox_wallpaper.png<span class="w"> </span>Videos
</code></pre></div>
<p>This lists the content of your home directory, and is the equivalent to
<code>ls $HOME</code>. You can combine the tilde with a suffix to compose an
absolute path to some file or folder in your home directory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>~/code
$<span class="w"> </span><span class="nb">pwd</span>
/home/br/code
</code></pre></div>
<h3 id="pathname-expansion">Pathname expansion</h3>
<p>Pathname expansions allow you to write an short path pattern and have it
expanded in a list of files and directories, saving you from tedious
copy-pastes or a possibly long (and error-prone) command writing.</p>
<h4 id="_3"><code>*</code></h4>
<p>The <em>glob</em>, or <em>wildcard</em> <code>*</code> character matches any string. It allows
you to give a <em>pattern</em> to the shell, that it will then expand to all
files and directories matching the pattern. The wildcard can be prefixed
or suffixed, which will further specify our pattern. For example,
<code>*.jpg</code> matches all files ending with the <code>.jpg</code> extension, and
<code>README.*</code> matches all files named <code>README</code> whatever their extension.</p>
<p>Let us consider the following file and directory structure.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">tree</span>
.
<span class="k">|</span>--<span class="w"> </span>pic1.jpg
<span class="k">|</span>--<span class="w"> </span>pic2.jpg
<span class="k">|</span>--<span class="w"> </span>pic3.jpg
<span class="k">|</span>--<span class="w"> </span>pic4.jpg
<span class="se">\_</span>_<span class="w"> </span>pics
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic5.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic6.jpg
<span class="k">|</span><span class="w"> </span><span class="se">\_</span>_<span class="w"> </span>pic7.jpg
<span class="se">\_</span>_<span class="w"> </span>sounds
<span class="w"> </span><span class="se">\_</span>_sound1.mp3
<span class="m">2</span><span class="w"> </span>directory,<span class="w"> </span><span class="m">8</span><span class="w"> </span>files
</code></pre></div>
<p>We want to move all <code>jpg</code> files into our <code>pics</code> directory. Instead of
running 4 different <code>mv</code> commands or manually typing a long <code>mv</code>
command, we can run just one using a pathname expansion.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mv</span><span class="w"> </span>*.jpg<span class="w"> </span>pics
$<span class="w"> </span><span class="nb">tree</span>
.
<span class="se">\_</span>_<span class="w"> </span>pics
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic1.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic2.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic3.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic4.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic5.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic6.jpg
<span class="k">|</span><span class="w"> </span><span class="se">\_</span>_<span class="w"> </span>pic7.jpg
<span class="se">\_</span>_<span class="w"> </span>sounds
<span class="w"> </span><span class="se">\_</span>_sound1.mp3
<span class="m">2</span><span class="w"> </span>directory,<span class="w"> </span><span class="m">8</span><span class="w"> </span>files
</code></pre></div>
<p><code>*.jpg</code> was expanded to all files ending with <code>.jpg</code>, causing the shell
to actually run <code>mv pic1.jpg pic2.jpg pic3.jpg pic4.jpg pics</code>, causing
all 4 <code>jpg</code> files to be moved to the <code>pics</code> directory in a single
command.</p>
<div class="Note">
<p>We could have executed the following command for the same result: <code>mv pic*.jpg pics</code>. This would have moved all files with name starting by <code>pic</code> and ending with <code>.jpg</code> to the <code>pics</code> directory</p>
</div>
<p>You can use <code>*</code> several times within the same pattern. For example
<code>ls */*</code> will list all files and directories located in a subdirectory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>*/*
sounds/sound1.mp3<span class="w"> </span>pics/pic2.jpg<span class="w"> </span>pics/pic4.jpg<span class="w"> </span>pics/pic6.jpg
pics/pic1.jpg<span class="w"> </span>pics/pic3.jpg<span class="w"> </span>pics/pic5.jpg<span class="w"> </span>pics/pic7.jpg
</code></pre></div>
<p>Like in our second example, we can also use <code>*/*.jpg</code> to list all <code>jpg</code>
files located in a subdirectory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>*/*.jpg
pics/pic1.jpg<span class="w"> </span>pics/pic3.jpg<span class="w"> </span>pics/pic5.jpg<span class="w"> </span>pics/pic7.jpg
pics/pic2.jpg<span class="w"> </span>pics/pic4.jpg<span class="w"> </span>pics/pic6.jpg
</code></pre></div>
<h4 id="_4"><code>**</code></h4>
<p><code>**</code> is expanded to all files and directories in the children
directories, with a depth limit of 1.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">touch</span><span class="w"> </span>README.txt
$<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>sounds/lyrics
$<span class="w"> </span><span class="nb">touch</span><span class="w"> </span>sounds/lyrics/sound1.txt
$<span class="w"> </span><span class="nb">tree</span>
.
<span class="k">|</span>--<span class="w"> </span>README.txt
<span class="se">\_</span>_<span class="w"> </span>pics
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic1.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic2.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic3.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic4.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic5.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic6.jpg
<span class="k">|</span><span class="w"> </span><span class="se">\_</span>_<span class="w"> </span>pic7.jpg
<span class="se">\_</span>_<span class="w"> </span>sounds
<span class="w"> </span><span class="se">\_</span>_<span class="w"> </span>lyrics
<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\_</span>_sound1.txt
<span class="w"> </span><span class="se">\_</span>_sound1.mp3
<span class="m">3</span><span class="w"> </span>directories,<span class="w"> </span><span class="m">10</span><span class="w"> </span>files
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>**
README.txt
pics:
pic1.jpg<span class="w"> </span>pic2.jpg<span class="w"> </span>pic3.jpg<span class="w"> </span>pic4.jpg<span class="w"> </span>pic5.jpg<span class="w"> </span>pic6.jpg<span class="w"> </span>pic7.jpg
sounds:
lyrics<span class="w"> </span>sounds.mp3
</code></pre></div>
<p><code>ls **</code> was expanded into <code>ls README.txt pics/ sounds/</code>, which does not
include the content of <code>sounds/lyrics</code> because of the depth limit of 1.</p>
<h4 id="_5"><code>**/</code></h4>
<p><code>**/</code> is expanded into all directories and subdirectories with a depth
limit of 1 starting from our first directory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">tree</span>
.
<span class="k">|</span>--<span class="w"> </span>README.txt
<span class="se">\_</span>_<span class="w"> </span>pics
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic1.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic2.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic3.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic4.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic5.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic6.jpg
<span class="k">|</span><span class="w"> </span><span class="se">\_</span>_<span class="w"> </span>pic7.jpg
<span class="se">\_</span>_<span class="w"> </span>sounds
<span class="w"> </span><span class="se">\_</span>_<span class="w"> </span>lyrics
<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\_</span>_sound1.txt
<span class="w"> </span><span class="se">\_</span>_sound1.mp3
<span class="m">3</span><span class="w"> </span>directories,<span class="w"> </span><span class="m">10</span><span class="w"> </span>files
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>**/
pics/:
pic1.jpg<span class="w"> </span>pic2.jpg<span class="w"> </span>pic3.jpg<span class="w"> </span>pic4.jpg<span class="w"> </span>pic5.jpg<span class="w"> </span>pic6.jpg<span class="w"> </span>pic7.jpg
sounds/:
lyrics<span class="w"> </span>sounds.mp3
sounds/lyrics/:
sound1.txt
</code></pre></div>
<p><code>ls **/</code> was expanded into <code>ls sounds/ sounds/lyrics pics/</code>. It thus
listed all files located in our subdirectories.</p>
<h3 id="brace-expansion">Brace expansion</h3>
<p>A brace expansion is a mechanism by which the shell can generate
multiple strings based on a sequence of tokens defined within curly
braces. The brace expansion pattern can be preceded by an optional
<em>preamble</em> and followed by an optional <em>postscript</em>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>~/test/<span class="o">{</span>pics,sounds,sprites<span class="o">}</span>
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>~/test
pics<span class="w"> </span>sounds<span class="w"> </span>sprites
</code></pre></div>
<p><code>~/test/{pics,sounds,sprites}</code> was expanded into
<code>~/test/pics ~/test/sounds ~/test/sprites</code> causing the shell to execute
<code>mkdir ~/test/pics ~/test/sounds ~/test/sprites</code> (which will be expanded
further into
<code>mkdir /home/br/test/pics /home/br/test/sounds /home/br/test/sprites</code> by
a tilde expansion).</p>
<p>We could have done the same thing by factoring the final <code>s</code> of each
token into a postscript.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>~/test/<span class="o">{</span>pic,sound,sprite<span class="o">}</span>s
</code></pre></div>
<p>A brace expansion can also have a sequence pattern <code>{x..y[..incr]}</code>
where <code>x</code> and <code>y</code> are either an integer or a single character, and
<code>incr</code> is an optional increment value.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">touch</span><span class="w"> </span>~/test/sounds/noise-<span class="o">{</span><span class="m">1</span>..5<span class="o">}</span>.mp3
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>~/test/sounds
noise-1.mp3<span class="w"> </span>noise-2.mp3<span class="w"> </span>noise-3.mp3<span class="w"> </span>noise-4.mp3<span class="w"> </span>noise-5.mp3
</code></pre></div>
<p>The default increment is 1 if the sequence end is greater than its
start, and -1 otherwise. However, we could specify a custom increment
value if we want.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">touch</span><span class="w"> </span>~/test/pics/pic<span class="o">{</span><span class="m">1</span>..10..2<span class="o">}</span>.jpg
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>~/test/pics
pic1.jpg<span class="w"> </span>pic3.jpg<span class="w"> </span>pic5.jpg<span class="w"> </span>pic7.jpg<span class="w"> </span>pic9.jpg
</code></pre></div>
<h3 id="command-expansion">Command expansion</h3>
<p>Your shell can replace a command surrounded by <code>$()</code> with its output.</p>
<p>I personally like use to commands expansions to iterate over a
command's result, or by combining it with a heredoc redirection:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span><span class="s"><<EOF > aboutme</span>
<span class="s">My name is $(whoami)</span>
<span class="s">and I live in $HOME</span>
<span class="s">EOF</span>
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>aboutme
My<span class="w"> </span>name<span class="w"> </span>is<span class="w"> </span>br
and<span class="w"> </span>I<span class="w"> </span>live<span class="w"> </span><span class="k">in</span><span class="w"> </span>/home/br
</code></pre></div>
<p><a id="real-life-examples"></a></p>
<h2 id="real-life-examples">Real-life examples</h2>
<h3 id="moving-a-pattern-of-files-contained-in-directories-and-subdirectories">Moving a pattern of files contained in directories and subdirectories</h3>
<p>What is really powerful with these expansions is that, like almost
everything in the shell, they can be combined. The following example
combines a pathname expansion, a brace expansion and a tilde expansion.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">tree</span>
.
<span class="k">|</span>--<span class="w"> </span>README.txt
<span class="se">\_</span>_<span class="w"> </span>pics
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic1.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic2.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic3.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic4.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic5.jpg
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>pic6.jpg
<span class="k">|</span><span class="w"> </span><span class="se">\_</span>_<span class="w"> </span>pic7.jpg
<span class="se">\_</span>_<span class="w"> </span>sounds
<span class="w"> </span><span class="se">\_</span>_<span class="w"> </span>lyrics
<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\_</span>_sound1.txt
<span class="w"> </span><span class="se">\_</span>_sound1.mp3
$<span class="w"> </span><span class="nb">mv</span><span class="w"> </span>**/*.<span class="o">{</span>jpg,mp3<span class="o">}</span><span class="w"> </span>~/assets/
$<span class="w"> </span><span class="nb">tree</span>
<span class="k">|</span>--<span class="w"> </span>README.txt
<span class="se">\_</span>_<span class="w"> </span>pics
<span class="se">\_</span>_<span class="w"> </span>sounds
<span class="w"> </span><span class="se">\_</span>_<span class="w"> </span>lyrics
<span class="w"> </span><span class="se">\_</span>_sound1.txt
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>~/assets
pic1.jpg<span class="w"> </span>pic2.jpg<span class="w"> </span>pic3.jpg<span class="w"> </span>pic4.jpg<span class="w"> </span>pic5.jpg<span class="w"> </span>pic6.jpg<span class="w"> </span>pic7.jpg<span class="w"> </span>sound1.mp3
</code></pre></div>
<p>Using these expansions, we were able to move all <code>jpg</code> and <code>mp3</code> files
located in directories and subdirectories to the <code>assets</code> directory
located in your home directory, in exactly 27 characters!</p>
<h3 id="renaming-multiple-directories">Renaming multiple directories</h3>
<p>We could use a <code>for</code> loop, pathname expansion and a command expansion to
rename all directories contained in the current directory to their
uppercase equivalent.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="k">for</span><span class="w"> </span>dir<span class="w"> </span><span class="k">in</span><span class="w"> </span>*/<span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nb">mv</span><span class="w"> </span><span class="s2">"</span><span class="nv">$dir</span><span class="s2">"</span><span class="w"> </span><span class="s2">"</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$dir</span><span class="s2">"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span><span class="s1">'[:lower:]'</span><span class="w"> </span><span class="s1">'[:upper:]'</span><span class="k">)</span><span class="s2">"</span>
<span class="w"> </span><span class="k">done</span>
</code></pre></div>
<p>Let's decompose that command into its different steps:</p>
<ul>
<li>the <code>*/</code> glob pattern is expanded over the list of
directories, on which we iterate via a <code>for</code> loop</li>
<li>we execute <code>echo $dir | tr '[:lower:]' '[:upper:]'</code>, which will
convert the current directory name to uppercase</li>
<li>the <code>$(echo $dir | tr '[:lower:]' '[:upper:]')</code> command is expanded
into the uppercase directory name</li>
<li>the directory is renamed into an uppercase name</li>
<li>the <code>for</code> loop iterates over the next directory name</li>
<li>we move on to the next directory and repeat the previous steps for
each of them</li>
</ul>
<div class="Note">
<p>Iterating over paths with a <code>for</code> loop is brittle as it breaks if a path
contains a space. We will later see how to properly do it using the
<code>find</code> command.</p>
</div>
<p><a id="summary"></a></p>
<h2 id="summary">Summary</h2>
<p>Your shell has so many productivity tricks and shortcuts up its sleeve
it can be a little bit daunting. I suggest you don't try to learn them
all at once, but really just experiment with them and see what feels
natural. Even mastering some of them will make you more productive!</p>
<p>What if there is an action you find useful but you just don't like the
keyboard shortcut? Luckily for you, the next chapter will dive into how
to personalize and customize your shell.</p>
<p><a id="going-further"></a></p>
<h2 id="going-further">Going further</h2>
<p><strong>5.1</strong>: Create a directory. Use a bash expansion to move into that
directory without typing its name a second time.</p>
<p><strong>5.2</strong>: Print your 4th last command typed into your terminal without
re-typing it.</p>
<p><strong>5.3</strong>: Create the following empty files <code>README.txt</code>,
<code>requirements.txt</code> and <code>TODO.txt</code> in a single command, without typing
<code>.txt</code> more than once.</p>
<p><strong>5.4</strong>: Delete all the files created in the last question without
typing <code>.txt</code> more than once.</p>
<p><strong>5.5</strong>: Create the following directory tree in a single command.</p>
<div class="highlight"><pre><span></span><code>files
<span class="k">|</span>--<span class="w"> </span><span class="m">1</span>
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>1a
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>1b
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>1c
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>2a
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>2b
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>2c
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>3a
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>3b
<span class="k">|</span><span class="w"> </span><span class="se">\-</span>-<span class="w"> </span>3c
<span class="k">|</span>--<span class="w"> </span><span class="m">2</span>
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>1a
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>1b
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>1c
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>2a
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>2b
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>2c
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>3a
<span class="k">|</span><span class="w"> </span><span class="k">|</span>--<span class="w"> </span>3b
<span class="k">|</span><span class="w"> </span><span class="se">\-</span>-<span class="w"> </span>3c
<span class="se">\-</span>-<span class="w"> </span><span class="m">3</span>
<span class="w"> </span><span class="k">|</span>--<span class="w"> </span>1a
<span class="w"> </span><span class="k">|</span>--<span class="w"> </span>1b
<span class="w"> </span><span class="k">|</span>--<span class="w"> </span>1c
<span class="w"> </span><span class="k">|</span>--<span class="w"> </span>2a
<span class="w"> </span><span class="k">|</span>--<span class="w"> </span>2b
<span class="w"> </span><span class="k">|</span>--<span class="w"> </span>2c
<span class="w"> </span><span class="k">|</span>--<span class="w"> </span>3a
<span class="w"> </span><span class="k">|</span>--<span class="w"> </span>3b
<span class="w"> </span><span class="se">\-</span>-<span class="w"> </span>3c
</code></pre></div>
<p><strong>5.6</strong>: Remove all subdirectories starting with <code>3</code> created in the
previous command, while keeping the top <code>3</code> directory.</p>
<p><strong>5.7</strong>: Re-execute the command from exercise 5.3 by looking backwards
into your shell history.</p>
<footer>
<p>
<em>Essential Tools and Practices for the Aspiring Software Developer</em> is a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>The book will help you set up a productive development environment and get acquainted with tools and practices that, along with your programming languages of choice, will go a long way in helping you grow as a software developer.
It will cover subjects such as mastering the terminal, configuring and getting productive in a shell, the basics of code versioning with <code>git</code>, SQL basics, tools such as <code>Make</code>, <code>jq</code> and regular expressions, networking basics as well as software engineering and collaboration best practices.
</p>
<p>
If you are interested in the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</footer>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p><a href="https://github.com/scop/bash-completion">https://github.com/scop/bash-completion</a> <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p><a href="https://tiswww.case.edu/php/chet/readline/rltop.html">https://tiswww.case.edu/php/chet/readline/rltop.html</a> <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p><a href="https://www.gnu.org/software/emacs/">https://www.gnu.org/software/emacs/</a> <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p><a href="https://en.wikipedia.org/wiki/Vi">https://en.wikipedia.org/wiki/Vi</a> <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>Customizing your shell2020-04-17T00:00:00+02:002020-04-17T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2020-04-17:/customizing-your-shell<p>It is very common for programmers to tweak and customize their terminal and shell for hours, add or write new plug-ins, all in pursuit of the “perfect environment” and an increase of productivity. In that spirit, this chapter will cover different recommendations of terminal configurations, as well as a deep dive into how to customize your prompt, add colors, experiment with color palettes, for both <code>bash</code> and <code>zsh</code>. We will finally introduce the Oh My Zsh configuration framework.</p><header>
<p>
This article is part of a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>
If you are interested in the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</header>
<h2 id="table-of-contents">Table of Contents</h2>
<!-- MarkdownTOC autolink="true" levels="2" autoanchor="true" -->
<ul>
<li><a href="#which-terminal-should-i-use">Which terminal should I use?</a></li>
<li><a href="#what-font-should-i-use">What font should I use?</a></li>
<li><a href="#what-shell-should-i-use">What shell should I use?</a></li>
<li><a href="#configuring-your-shell">Configuring your shell</a></li>
<li><a href="#configuring-your-prompt">Configuring your prompt</a></li>
<li><a href="#shell-configuration-frameworks">Shell configuration frameworks</a></li>
<li><a href="#summary">Summary</a></li>
<li><a href="#going-further">Going further</a></li>
</ul>
<!-- /MarkdownTOC -->
<h1 id="customizing-your-shell">Customizing your shell</h1>
<p>It is very common for programmers to tweak and customize their terminal
and shell for hours, add or write new plug-ins, all in pursuit of the
“perfect environment” and an increase of productivity. Others, on the
contrary, avoid tweaking their shell altogether in order to always get
the same experience on every machine.</p>
<p>On a personal note, I tend to favor having a personalized shell as much
as possible. I feel that sharing files between different computers is
now a solved issue, and the benefits I get from having personalized my
work environments are so great that I gladly pay the small price of
synchronizing that configuration between my computers.</p>
<p>In that chapter, we will learn more about the shell and how to configure
your terminal environment to make it work <em>for</em> you. Please note that
some of the recommendations come from personal taste, and might not work
for you nor suit you. We encourage you to explore and find what feels
right, but we hope to at least nudge you in the right direction.</p>
<p><a id="which-terminal-should-i-use"></a></p>
<h2 id="which-terminal-should-i-use">Which terminal should I use?</h2>
<p>First off, if you are new to using the terminal, you might not have
realized that it exists <em>multiple</em> terminal applications. MacOS comes
with Terminal pre-installed, and most Linux distributions come with
either xterm, Gnome-terminal or Konsole pre-installed, and there is a
vast number of available alternatives.</p>
<p>I don't think there is a good, absolute and definitive answer when it
comes to picking the “right” terminal application. You might get various
answers depending who you ask. That being said, I can at least mention
my own personal recommendations and preferences.</p>
<p>Whatever terminal you end up using, I think that it is really important
you configure it to your liking and preferences. As a programmer, you
will probably spend a great deal of time in your terminal, and for you
to feel productive and empowered, it needs to work <em>for</em> you.</p>
<h3 id="terminator">Terminator</h3>
<p>If you are running Linux, I personally favor Terminator<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup> over the
default choices. It has several features I find useful:</p>
<ul>
<li>a tab system, allowing you to have multiple tab of terminal(s)
within the same window</li>
<li>a grid system, allowing you to have multiple terminals in the same
tab</li>
</ul>
<p><img alt="I can work in multiple panes within the same tab, and have one tab per project" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/terminator.png">
<span class=imgcaption>I can work in multiple panes within the same tab, and have one tab per project</span></p>
<p>Here are the terminator keyboard shortcuts I find the most useful:</p>
<table>
<thead>
<tr>
<th>Shortcut</th>
<th style="text-align: right;">Action</th>
</tr>
</thead>
<tbody>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>Shift</kbd> - <kbd>E</kbd></td>
<td style="text-align: right;">split the screen vertically</td>
</tr>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>Shift</kbd> - <kbd>O</kbd></td>
<td style="text-align: right;">split the screen horizontally</td>
</tr>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>Shift</kbd> - <kbd>T</kbd></td>
<td style="text-align: right;">open a new tab</td>
</tr>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>PageUp</kbd></td>
<td style="text-align: right;">switch to the next tab</td>
</tr>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>PageDown</kbd></td>
<td style="text-align: right;">switch to the previous tab</td>
</tr>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>N</kbd></td>
<td style="text-align: right;">open a new window</td>
</tr>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>Shift</kbd> - <kbd>+</kbd></td>
<td style="text-align: right;">zoom in</td>
</tr>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>Shift</kbd> - <kbd>-</kbd></td>
<td style="text-align: right;">zoom out</td>
</tr>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>D</kbd></td>
<td style="text-align: right;">close the current terminal</td>
</tr>
</tbody>
</table>
<h3 id="iterm2">iTerm2</h3>
<p>As far as macOS is concerned, I find the default terminal (plainly named
Terminal) to be hard to use. The terminal that seems to be widely
accepted by the macOS programming community is iTerm2<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>. It has all of
the features cited above, and many (many) more!</p>
<p><img alt="iTerm2 looks similar to Terminator but can do much, much more" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/iterm2.png">
<span class=imgcaption>iTerm2 looks similar to Terminator but can do much, much more</span></p>
<p>The iTerm2 keyboard shortcuts I find the most useful are:</p>
<table>
<thead>
<tr>
<th>Shortcut</th>
<th style="text-align: right;">Action</th>
</tr>
</thead>
<tbody>
<tr>
<td><kbd>Cmd</kbd> - <kbd>D</kbd></td>
<td style="text-align: right;">split the screen vertically</td>
</tr>
<tr>
<td><kbd>Cmd</kbd> - <kbd>Shift</kbd> - <kbd>D</kbd></td>
<td style="text-align: right;">split the screen horizontally</td>
</tr>
<tr>
<td><kbd>Cmd</kbd> - <kbd>T</kbd></td>
<td style="text-align: right;">open a new tab</td>
</tr>
<tr>
<td><kbd>Cmd</kbd> - <kbd>Shift</kbd> - <kbd>+</kbd></td>
<td style="text-align: right;">zoom in</td>
</tr>
<tr>
<td><kbd>Cmd</kbd> - <kbd>Shift</kbd> - <kbd>-</kbd></td>
<td style="text-align: right;">zoom out</td>
</tr>
<tr>
<td><kbd>Cmd</kbd> - <kbd>N</kbd></td>
<td style="text-align: right;">open a new window</td>
</tr>
<tr>
<td><kbd>Ctrl</kbd> - <kbd>D</kbd></td>
<td style="text-align: right;">close the current terminal</td>
</tr>
</tbody>
</table>
<p>The following sections go over some non-default iTerm2 settings that I
find convenient. Again, these are my preference and are in no way
prescriptive. Feel free to discard them if you want.</p>
<h4 id="open-file-shortcut">Open file shortcut</h4>
<p>One of the iTerm2 features I enjoy is the ability of using
<kbd>Cmd</kbd> + mouse click on a file path or an URL, to open the
resource with the default associated program. For example, it will open
an URL in your browser, a path to a local PDF file with Preview, a text
file with your preferred text editor, etc.</p>
<p><img alt="By enabling this feature, you will be able to open a file using a graphical application from your terminal" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/iterm2-pointer-pref.png">
<span class=imgcaption>By enabling this feature, you will be able to open a file using a graphical application from your terminal</span></p>
<h4 id="intuitive-location-for-new-terminals">Intuitive location for new terminals</h4>
<p>Another tweak I've done to iTerm2 was changing the working directory new
terminals will open into by default. What I wanted was</p>
<ul>
<li>open a new terminal window in my home directory</li>
<li>open a new terminal tab in my home directory</li>
<li>open a new terminal split pane in the previous session's directory</li>
</ul>
<p>I did this because I oftentimes found myself splitting the current tab
when I want to run multiple commands within the same project, and I had
to <code>cd</code> into the project directory every time I did a pane split.</p>
<p><img alt="I reduced the time I spent cd-ing into project directories with these settings. Preferences > Profiles > General > Working Directory > Advanced Configuration > Edit" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/iterm2-advanced-pref.png">
<span class=imgcaption>I reduced the time I spent <code>cd</code>-ing into project directories with these settings. Preferences > Profiles > General > Working Directory > Advanced Configuration > Edit</span></p>
<p><a id="what-font-should-i-use"></a></p>
<h2 id="what-font-should-i-use">What font should I use?</h2>
<p>Using a font you enjoy is paramount. If you spend a lot of time reading
and writing in your terminal, you might as well do it using a font that
feels right to you.</p>
<p>I personally really enjoy the Fira Code<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup> font, both in my text editor
and my terminal. Not only does it look really nice on the eye, but it
also contains a set of <em>ligatures</em> for multi character combinations,
such as <code>!</code> and <code>=</code> rendered in a single character, allowing you to read
code and decode symbols more easily.</p>
<p><img alt="Example of rendered character ligatures" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/ligature-code-example.png">
<span class=imgcaption>Example of rendered character ligatures</span></p>
<p>Note that not all terminals support fonts with ligatures. For example,
iTerm2 does but Terminator does not.</p>
<div class="Note">
<p>While Fira Code has my preference, there are other well-designed fonts
including ligatures, such as JetBrains Mono.<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup></p>
</div>
<p><a id="what-shell-should-i-use"></a></p>
<h2 id="what-shell-should-i-use">What shell should I use?</h2>
<p>We have hinted at it until now: <code>bash</code> is not the only shell out there.
You are free to use other shells if you want, such as <code>zsh</code>, <code>fish</code>,
<code>nushell</code>, … As it was the case with terminals, the “good” terminal
really depends on your definition of “good”. If you deeply care about
using the same shell on every machine you work on, then <code>bash</code> is
possibly for you. It has been around since 1989, is stable, mature and
is the default shell on almost<sup id="fnref:5"><a class="footnote-ref" href="#fn:5">5</a></sup> every UNIX system out there.</p>
<p>When researching this book, I was surprised to learn that <code>zsh</code> (or the
Z-shell) wasn't really the last “kid on the block” either, as it was
first released in 1990, just a year after the first stable bash release!
You can expect the same level of stability, maturity and even syntax (to
a large extent, except when it comes to configuration) than bash.</p>
<p>I personally think <code>zsh</code> really shines by providing a powerful default
auto-completion experience, as well as more configuration options. As
<code>zsh</code> is compatible with <code>bash</code>'s own syntax, I encourage you to try
them until you feel comfortable with one or the other.</p>
<p>The <code>fish</code><sup id="fnref:6"><a class="footnote-ref" href="#fn:6">6</a></sup> shell takes a radical turn from <code>bash</code> or <code>zsh</code> by
providing an incompatible but “simple and clean” syntax, an extremely
powerful command suggestion system, and an interactive configuration
wizard.</p>
<p>If you are getting started with using the shell, my personal
recommendation is to stick to <code>bash</code> or <code>zsh</code> and experiment with other
shells to see what value they bring once you feel more confident.</p>
<h3 id="changing-your-default-shell">Changing your default shell</h3>
<p>The <code>chsh</code> (standing for <em>change shell</em>) command allows you to change
your default shell.</p>
<p><strong>Examples</strong>:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Switching to bash by default</span>
$<span class="w"> </span><span class="nb">chsh</span><span class="w"> </span>-s<span class="w"> </span>/bin/bash
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="c1"># Switching to zsh by default</span>
$<span class="w"> </span><span class="nb">chsh</span><span class="w"> </span>-s<span class="w"> </span>/bin/zsh
</code></pre></div>
<p>Once you have run <code>chsh</code>, any new terminal window you open will run your
new default shell.</p>
<p><a id="configuring-your-shell"></a></p>
<h2 id="configuring-your-shell">Configuring your shell</h2>
<p>Up until now, every example we have seen have defined environment
variables, aliases and functions directly in the shell. However, if we
closed that shell, all of these changes would be undone and we would
have to start again the next time we open a new one. Fortunately, all of
these settings can be persisted in a <em>configuration file</em>. Adding
aliases, environment variables and functions to that file will make sure
they get imported every time you open a new shell.</p>
<p>These files usually reside in your home directory, and are named
<code>.bashrc</code> for bash, and <code>.zshrc</code> for zsh.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>~/.zshrc
<span class="nb">export</span><span class="w"> </span><span class="nv">EDITOR</span><span class="o">=</span><span class="nb">vim</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">PATH</span><span class="o">=</span><span class="nv">$HOME</span>/bin:<span class="nv">$PATH</span>
<span class="nb">alias</span><span class="w"> </span><span class="nb">ls</span><span class="o">=</span><span class="s1">'ls -G'</span>
<span class="nb">alias</span><span class="w"> </span>..<span class="o">=</span><span class="s1">'cd ..'</span>
<span class="nb">alias</span><span class="w"> </span>...<span class="o">=</span><span class="s1">'cd ../..'</span>
<span class="k">function</span><span class="w"> </span>mkcd<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">target</span><span class="o">=</span><span class="nv">$1</span>
<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>-p<span class="w"> </span><span class="s2">"</span><span class="nv">$target</span><span class="s2">"</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="nv">$target</span>
<span class="o">}</span>
</code></pre></div>
<p>After adding anything to your shell configuration file, you need to run
<code>source ~/.zshrc</code> (or <code>source ~/.bashrc</code>, depending on your shell). The
<code>source</code> built-in command reads and executes commands from the argument
file name in the current shell environment. Said in another way, running
<code>source ~/.<file></code> will cause the shell to reload its configuration.</p>
<div class="Note">
<p><code>rc</code> stands for <em>run commands</em>. Indeed, when you <code>source</code> your
configuration file, you will run the commands it contains. The subtlety
with <code>source</code> is that it executes the argument script within your
<em>current</em> shell, meaning any sourced commands will have a side-effect on
your running shell.</p>
</div>
<p>If you can never remember a given command's options, or if you always
find yourself typing a group of commands, I encourage you to define
aliases and functions in your shell configuration file. They will allow
you to feel more productive day after day, especially so if the alias
and tools are abstracting complex commands.</p>
<div class="Note">
<p>The previous chapter ended with some real-life examples of alias and
functions. Feel free to add them to your shell configuration file.</p>
</div>
<p><a id="configuring-your-prompt"></a></p>
<h2 id="configuring-your-prompt">Configuring your prompt</h2>
<p>Configuring your <span class="gls" key="prompt">prompt</span> is a very
good way to make the shell work for you as much as possible, by
providing you with useful context, such as the time of day, whether the
last command was successful, your current working directory… While they
can provide context and information to you, they will carry that context
to anyone you copy and paste a command and associated output to.</p>
<p>Configuring your prompt is done by changing the value of the <code>PS1</code>
environment variable.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PS1</span><span class="o">=</span><span class="s2">"MY COOL PROMPT </span>$<span class="s2">"</span>
MY<span class="w"> </span>COOL<span class="w"> </span>PROMPT<span class="w"> </span>$
</code></pre></div>
<p>I think we can agree that <code>MY COOL PROMPT</code> is not as informative as it
could, so let's change it to put our prompt to work. As the prompt
configuration work slightly different between <code>bash</code> and <code>zsh</code>, we will
address both cases in two different sections.</p>
<h3 id="configuring-your-bash-prompt">Configuring your bash prompt</h3>
<p>The <code>PS1</code> environment variable can be defined by using a mix and match
of both regular and special characters. The regular characters are just
displayed as-is, whereas the backslash-escaped special characters are
interpreted by bash at the time <code>PS1</code> is displayed and replaced by the
associated value. The most useful special characters are defined as
follows.</p>
<table>
<thead>
<tr>
<th>Character</th>
<th>Meaning</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>\h</code></td>
<td>The hostname up to the first dot</td>
</tr>
<tr>
<td><code>\t</code></td>
<td>The current time, in 24-hour HH:MM:SS format</td>
</tr>
<tr>
<td><code>\u</code></td>
<td>The current user's username</td>
</tr>
<tr>
<td><code>\w</code></td>
<td>The full current working directory (<code>$HOME</code> rendered as <code>~</code>)</td>
</tr>
<tr>
<td><code>\W</code></td>
<td>The basename of the current working directory (<code>$HOME</code> rendered as <code>~</code>)</td>
</tr>
<tr>
<td><code>\n</code></td>
<td>A new line</td>
</tr>
</tbody>
</table>
<p>These special characters are evaluated every-time the prompt is displayed to make sure you always get the most up-to-date context.</p>
<div class="Note">
<p>The <code>PROMPTING</code> section of the <code>bash</code> manual contains the full list of
backslash-escaped special characters.</p>
</div>
<p><strong>Examples</strong></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PS1</span><span class="o">=</span><span class="s1">'\u@\h \W $'</span>
br@morenika<span class="w"> </span><span class="k">~</span><span class="w"> </span>$
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PS1</span><span class="o">=</span><span class="s1">'[\t] \u@\h \W $'</span>
<span class="o">[</span><span class="m">13</span>:33:55<span class="o">]</span><span class="w"> </span>br@morenika<span class="w"> </span><span class="k">~</span><span class="w"> </span>$
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PS1</span><span class="o">=</span><span class="s1">'[\t \u@\h:\w]\n>>> '</span>
<span class="o">[</span><span class="m">13</span>:57:55<span class="w"> </span>br@morenika:~/code<span class="o">]</span>
>>>
</code></pre></div>
<div class="Note">
<p>You can use online tools such as ezprompt<sup id="fnref:7"><a class="footnote-ref" href="#fn:7">7</a></sup> to try different
configurations until you find something you like.</p>
<p>Whatever <code>PS1</code> value you settle with should be persisted and exported in
your <code>.bashrc</code> configuration file.</p>
</div>
<h3 id="configuring-your-zsh-prompt">Configuring your zsh prompt</h3>
<p><code>zsh</code> exposes a bit more options than <code>bash</code> when it comes to prompt
configuration. Both <code>PS1</code> and <code>PROMPT</code> environment variable can be set
to the same effect, if you find <code>PROMPT</code> more explicit.</p>
<p>Instead of being backslash-escaped, zsh's special characters are
prefixed by <code>%</code>, and are called <em>prompt sequences</em>. The most useful are
detailed here.</p>
<table>
<thead>
<tr>
<th>Sequence</th>
<th>Meaning</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>%m</code></td>
<td>The hostname up to the first dot</td>
</tr>
<tr>
<td><code>%*</code></td>
<td>The current time, in 24-hour HH:MM:SS format</td>
</tr>
<tr>
<td><code>%n</code></td>
<td>The current user's username</td>
</tr>
<tr>
<td><code>%~</code></td>
<td>The full current working directory (<code>$HOME</code> rendered as <code>~</code>)</td>
</tr>
<tr>
<td><code>%1~</code></td>
<td>The basename of the current working directory (<code>$HOME</code> rendered as <code>~</code>)</td>
</tr>
<tr>
<td><code>%?</code></td>
<td>The exit status of the last command executed</td>
</tr>
<tr>
<td><code>%%</code></td>
<td>A %</td>
</tr>
<tr>
<td><code>$'\n'</code></td>
<td>A new line</td>
</tr>
<tr>
<td><code>%B (%b)</code></td>
<td>Start (stop) bold font mode</td>
</tr>
<tr>
<td><code>%F (%f)</code></td>
<td>Start (stop) using a given foreground color, if supported by the terminal</td>
</tr>
</tbody>
</table>
<div class="Note">
<p>You will find the full list of prompt sequences in the zsh
documentation<sup id="fnref:8"><a class="footnote-ref" href="#fn:8">8</a></sup>.</p>
</div>
<p><strong>Examples</strong></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PROMPT</span><span class="o">=</span><span class="s1">'%n@%m %~ $ '</span>
br@morenika<span class="w"> </span>~/code<span class="w"> </span>$
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PROMPT</span><span class="o">=</span><span class="s1">'[%*] %n@%m %~ $ '</span>
<span class="o">[</span><span class="m">23</span>:38<span class="o">]</span><span class="w"> </span>br@morenika<span class="w"> </span>~/code<span class="w"> </span>$
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PROMPT</span><span class="o">=</span><span class="s2">"[%* %n@%m %~]"</span><span class="s1">$'\n'</span><span class="s2">">>> "</span>
<span class="o">[</span><span class="m">23</span>:41<span class="w"> </span>br@morenika<span class="w"> </span>~/code/izk<span class="o">]</span>
>>>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PROMPT</span><span class="o">=</span><span class="s2">"[%* %n@%m %1~]"</span><span class="s1">$'\n'</span><span class="s2">"%% "</span>
<span class="o">[</span><span class="m">23</span>:41<span class="w"> </span>br@morenika<span class="w"> </span>izk<span class="o">]</span>
%
</code></pre></div>
<p><code>zsh</code> goes even further by letting you define the content of a
right-sided prompt, through the <code>RPROMPT</code> environment variable, which
uses the same syntax as <code>PROMPT</code>.</p>
<p><strong>Example</strong></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PROMPT</span><span class="o">=</span><span class="s1">'%~ $ '</span><span class="p">;</span><span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">RPROMPT</span><span class="o">=</span><span class="s1">'%*'</span>
~/code<span class="w"> </span>$<span class="w"> </span><span class="m">21</span>:04:00
</code></pre></div>
<div class="Note">
<p>To make sure your changes are persisted, <code>PROMPT</code> and <code>RPROMPT</code> should
be exported in your <code>.zshrc</code> configuration file.</p>
</div>
<h3 id="adding-colors">Adding Colors</h3>
<p>Adding color is a good way to spice up your prompt as well as providing
some visual context. You can use color to indicate whether you are
running with super-user privileges, if the last command succeeded or
failed, or simply colorized each individual part of your prompt
(username, hostname, etc) in a different way to make it even simpler to
parse.</p>
<h4 id="adding-color-to-your-bash-prompt">Adding color to your bash prompt</h4>
<p>Bash allows you to style elements of your prompt by using 3-bit ANSI<sup id="fnref:9"><a class="footnote-ref" href="#fn:9">9</a></sup>
codes defining a zone associated with a potential effect, foreground
color and background color.</p>
<p>Each effect, background or foreground color has an associated code,
described in the following tables. The combination of these parameters
is called <span class="gls" key="sgr">Select Graphic Rendition</span>,
which is defined as a semicolon (;) separated list of codes.</p>
<div class="fullwidth">
<table>
<thead>
<tr>
<th>Effect</th>
<th style="text-align: right;">ANSI Code</th>
</tr>
</thead>
<tbody>
<tr>
<td>Normal</td>
<td style="text-align: right;"><code>0</code></td>
</tr>
<tr>
<td>Bold</td>
<td style="text-align: right;"><code>1</code></td>
</tr>
<tr>
<td>Faint</td>
<td style="text-align: right;"><code>2</code></td>
</tr>
<tr>
<td>Italic</td>
<td style="text-align: right;"><code>3</code></td>
</tr>
<tr>
<td>Underline</td>
<td style="text-align: right;"><code>4</code></td>
</tr>
<tr>
<td>Strike through</td>
<td style="text-align: right;"><code>9</code></td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Background Color</th>
<th style="text-align: right;">ANSI Code</th>
</tr>
</thead>
<tbody>
<tr>
<td>Red</td>
<td style="text-align: right;"><code>41</code></td>
</tr>
<tr>
<td>Green</td>
<td style="text-align: right;"><code>42</code></td>
</tr>
<tr>
<td>Brown</td>
<td style="text-align: right;"><code>43</code></td>
</tr>
<tr>
<td>Blue</td>
<td style="text-align: right;"><code>44</code></td>
</tr>
<tr>
<td>Purple</td>
<td style="text-align: right;"><code>45</code></td>
</tr>
<tr>
<td>Cyan</td>
<td style="text-align: right;"><code>46</code></td>
</tr>
<tr>
<td>White</td>
<td style="text-align: right;"><code>47</code></td>
</tr>
<tr>
<td>Bright black</td>
<td style="text-align: right;"><code>100</code></td>
</tr>
<tr>
<td>Bright red</td>
<td style="text-align: right;"><code>101</code></td>
</tr>
<tr>
<td>Bright green</td>
<td style="text-align: right;"><code>102</code></td>
</tr>
<tr>
<td>Bright brown</td>
<td style="text-align: right;"><code>103</code></td>
</tr>
<tr>
<td>Bright blue</td>
<td style="text-align: right;"><code>104</code></td>
</tr>
<tr>
<td>Bright purple</td>
<td style="text-align: right;"><code>105</code></td>
</tr>
<tr>
<td>Bright cyan</td>
<td style="text-align: right;"><code>106</code></td>
</tr>
<tr>
<td>Bright white</td>
<td style="text-align: right;"><code>107</code></td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Foreground Color</th>
<th style="text-align: right;">ANSI Code</th>
</tr>
</thead>
<tbody>
<tr>
<td>Black</td>
<td style="text-align: right;"><code>30</code></td>
</tr>
<tr>
<td>Red</td>
<td style="text-align: right;"><code>31</code></td>
</tr>
<tr>
<td>Green</td>
<td style="text-align: right;"><code>32</code></td>
</tr>
<tr>
<td>Brown</td>
<td style="text-align: right;"><code>33</code></td>
</tr>
<tr>
<td>Blue</td>
<td style="text-align: right;"><code>34</code></td>
</tr>
<tr>
<td>Purple</td>
<td style="text-align: right;"><code>35</code></td>
</tr>
<tr>
<td>Cyan</td>
<td style="text-align: right;"><code>36</code></td>
</tr>
<tr>
<td>White</td>
<td style="text-align: right;"><code>37</code></td>
</tr>
<tr>
<td>Bright black</td>
<td style="text-align: right;"><code>90</code></td>
</tr>
<tr>
<td>Bright red</td>
<td style="text-align: right;"><code>91</code></td>
</tr>
<tr>
<td>Bright green</td>
<td style="text-align: right;"><code>92</code></td>
</tr>
<tr>
<td>Bright brown</td>
<td style="text-align: right;"><code>93</code></td>
</tr>
<tr>
<td>Bright blue</td>
<td style="text-align: right;"><code>94</code></td>
</tr>
<tr>
<td>Bright purple</td>
<td style="text-align: right;"><code>95</code></td>
</tr>
<tr>
<td>Bright cyan</td>
<td style="text-align: right;"><code>96</code></td>
</tr>
<tr>
<td>Bright white</td>
<td style="text-align: right;"><code>97</code></td>
</tr>
</tbody>
</table>
</div>
<p><strong>Examples of SGRs</strong></p>
<ul>
<li>blue text: <code>34</code></li>
<li>bold green text: <code>1;32</code></li>
<li>purple text on a white background: <code>35;47</code></li>
<li>bold red text on a bright cyan background: <code>1;31;106</code></li>
<li>bold and striked-through brown text on a green background
<code>1;9;33;42</code></li>
</ul>
<p>To define colorized zones in your bash prompt, use the following
(granted, ugly) syntax:</p>
<div class="highlight"><pre><span></span><code>\e[<SGR>mTEXT\e[m
</code></pre></div>
<p><strong>Examples</strong></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PS1</span><span class="o">=</span><span class="s1">'[\t] \u@\h \W \e[32m$\e[m '</span>
</code></pre></div>
<p><img alt="The $ sign is now displayed in green" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/green-prompt.png">
<span class=imgcaption>The <code>$</code> sign is now displayed in green</span></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PS1</span><span class="o">=</span><span class="s1">'\e[31m\u\e[m@\e[32m\h\e[m \e[36m\W\e[m $ '</span>
</code></pre></div>
<p><img alt="The username is in red, the hostname in green and the path is in cyan. " decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/bash-colors-prompt.png">
<span class=imgcaption>The username is in red, the hostname in green and the path is in cyan. </span></p>
<h5 id="color-palettes">Color palettes</h5>
<p>Notice how an ANSI code only maps to a color name? That's because it is
up to your <em>terminal</em> to interpret and render that color name into an
actual color, meaning that the same prompt configuration could be
rendered differently on two different terminals.</p>
<p>Mapping ANSI color names to actual <span class="gls"
key="rgb">RGB</span> colors is done through what is called <em>color
palettes</em>.</p>
<p>Following are two different color schemes, as well as the associated
rendered prompt, both using the same <code>PS1</code> value, used in the previous
example.</p>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/color-scheme-solarized-dark-ansi.png"></p>
<p><img alt="The popular Solarized Dark color scheme" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/color-scheme-solarized-dark.png">
<span class=imgcaption>The popular Solarized Dark color scheme</span></p>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/color-scheme-pastel-ansi.png"></p>
<p><img alt="The Pastel (Dark Background) color scheme" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/color-scheme-pastel-dark-shell.png">
<span class=imgcaption>The Pastel (Dark Background) color scheme</span></p>
<p>As you can see, these both look quite different from the prompt
displayed in <span class="ref" key="fig:basic-prompt-colors">the
previous screenshot</span>, even though the underlying prompt
configuration is exactly the same. This means that, even if using 16
colors can feel limiting, you actually can map these colors to any color
you like. The ANSI color system just prevents you from having more than
16 <em>different</em> colors in your prompt.</p>
<div class="Note">
<p>I recommend you to have a look at the
<code>mbadolato/iTerm2-Color-Schemes</code><sup id="fnref:10"><a class="footnote-ref" href="#fn:10">10</a></sup> project, showcasing popular color
palettes and providing you with the configuration files allowing you to
used them in many terminal applications (and not just iTerm2 contrary to
what its name suggests).</p>
</div>
<h5 id="up-to-256-colors">Up to 256 colors</h5>
<p>As computers eventually started to have 256 colors graphics card, a 8
bit ANSI code scheme was introduced, allowing the user to render 256
colors in their terminal, instead of 16.</p>
<p>The 8-bit ANSI code syntax is <code>\e[38;5;n</code> where the colors associated
with each value of <code>n</code> between 0 and 255 are represented in the
following table<sup id="fnref:11"><a class="footnote-ref" href="#fn:11">11</a></sup>.</p>
<p><img alt="The 8-bit ANSI code allows you to render more than the initial 16 available colors" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/256col-table.png">
<span class=imgcaption>The 8-bit ANSI code allows you to render more than the initial 16 available colors</span></p>
<p><strong>Examples</strong></p>
<div class="highlight"><pre><span></span><code><span class="c1"># Using 256-bit ANSI codes</span>
$<span class="w"> </span><span class="nv">TIME</span><span class="o">=</span><span class="s2">"[\e[38;5;33m\t\e[m]"</span><span class="w"> </span><span class="c1"># blue</span>
$<span class="w"> </span><span class="nv">USERNAME</span><span class="o">=</span><span class="s2">"\e[38;5;200m\u\e[m"</span><span class="w"> </span><span class="c1"># pink</span>
$<span class="w"> </span><span class="nv">HOSTNAME</span><span class="o">=</span><span class="s2">"\e[38;5;139m\h\e[m"</span><span class="w"> </span><span class="c1"># purple</span>
$<span class="w"> </span><span class="nv">WORKDIR</span><span class="o">=</span><span class="s2">"\W"</span><span class="w"> </span><span class="c1"># no color</span>
$<span class="w"> </span><span class="nv">DOLLAR</span><span class="o">=</span><span class="s2">"\e[38;5;41m</span>$<span class="s2">\e[m"</span><span class="w"> </span><span class="c1"># green</span>
$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PS1</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">TIME</span><span class="si">}</span><span class="s2"> </span><span class="si">${</span><span class="nv">USERNAME</span><span class="si">}</span><span class="s2">@</span><span class="si">${</span><span class="nv">HOSTNAME</span><span class="si">}</span><span class="s2"> </span><span class="si">${</span><span class="nv">PTH</span><span class="si">}</span><span class="s2"> </span><span class="si">${</span><span class="nv">DOLLAR</span><span class="si">}</span><span class="s2"> "</span>
</code></pre></div>
<p><img alt="These ANSI codes sure are awful to read but they make for pretty colors " decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/256col-prompt.png">
<span class=imgcaption>These ANSI codes sure are awful to read but they make for pretty colors </span></p>
<div class="Note">
<p>Not all terminals support 256 colors, but most of the modern ones
should. To this day, GNOME Terminal, Konsole, Terminator, XFCE4
Terminal, iTerm2, Terminal (macOS) and tmux all support 256 colors.</p>
</div>
<p>Contrary to the 3-bit ANSI codes, the 8-bit codes are insensitive to
color schemes changes, as shown in the following examples, both re-using
the same <code>PS1</code> configuration than in the <span class="ref"
key="fig:256col-prompt">previous screenshot</span>.</p>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/color-scheme-solarized-dark-256.png"></p>
<p><img alt="The colors remain unchanged" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/color-scheme-pastel-dark-256.png">
<span class=imgcaption>The colors remain unchanged</span></p>
<h4 id="adding-color-to-your-zsh-prompt">Adding color to your zsh prompt</h4>
<p>Everything we've explained in the previous section is still valid for
<code>zsh</code>: you can use 3 or 8 bit ANSI color codes just fine. However, <code>zsh</code>
also provides you with a much easier and readable color system:</p>
<ul>
<li>each color can be represented as either <code>black</code>, <code>red</code>, <code>green</code>,
<code>yellow</code>, <code>blue</code>, <code>magenta</code>, <code>cyan</code> or <code>white</code>, or a number between
0 and 255</li>
<li><code>%F{color}Text%f</code>: changes the <code>Text</code> foreground color to <code>color</code></li>
<li><code>%K{color}Text%k</code>: changes the <code>Text</code> background color to <code>color</code></li>
<li><code>%BText%b</code>: displays <code>Text</code> in boldface</li>
<li><code>%UText%u</code>: underlines <code>Text</code></li>
</ul>
<p><strong>Example</strong></p>
<p><img alt="The current working directory in blue and the dollar sign in bold pink" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/zsh-colors-prompt.png">
<span class=imgcaption>The current working directory in blue and the dollar sign in bold pink</span></p>
<h3 id="displaying-dynamic-data-in-the-prompt">Displaying dynamic data in the prompt</h3>
<p>We can make our prompt display dynamic context to make it even more
informative. To do this, we can execute a function as part of our <code>PS1</code>
environment variable. The shell will call that function every time it
renders the prompt.</p>
<p>The idea is to be able to have as much information as possible in your
prompt at the ready, but <em>only when necessary</em>.</p>
<h4 id="displaying-dynamic-data-in-bash">Displaying dynamic data in bash</h4>
<p>Let's say that we want to colorize the <code>$</code> of our prompt in green if the
last command was successful, and in red if it failed. We can wrap that
logic into the following <code>colorized_prompt</code> bash function, and have it
called every time <code>PS1</code> is rendered by including <code>$(colorized_prompt)</code>
in the environment variable.</p>
<p>The <code>$(colorized_prompt)</code> syntax means "call the <code>colorize_prompt</code> function", and will be expanded into the output of the function (what it prints), which will contain ASCII color codes colorizing the prompt.</p>
<div class="highlight"><pre><span></span><code><span class="k">function</span><span class="w"> </span>colorized_prompt<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="c1"># Check if last command exit code equals 0</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">((</span><span class="k">$?</span><span class="o">))</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">printf</span><span class="w"> </span><span class="s2">"\e[32m</span>$<span class="s2">\e[m"</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nb">printf</span><span class="w"> </span><span class="s2">"\e[31m</span>$<span class="s2">\e[m"</span>
<span class="w"> </span><span class="k">fi</span>
<span class="o">}</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">PS1</span><span class="o">=</span><span class="s1">'[\t] \W $(colorized_prompt) '</span>
</code></pre></div>
<div class="Note">
<p><code>$?</code> is a special bash parameter that expands to the <em>exit status</em> of the previously executed command. The norm is to have an exit status of 0 if the command executed successfully, and any other exit status indicates an error.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">pwd</span>
/home/br
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="k">$?</span>
<span class="m">0</span>
$<span class="w"> </span>cmdnotfound
bash:<span class="w"> </span>cmdnotfound:<span class="w"> </span><span class="nb">command</span><span class="w"> </span>not<span class="w"> </span>found
<span class="nb">echo</span><span class="w"> </span><span class="k">$?</span>
<span class="m">127</span>
</code></pre></div>
<p>The syntax <code>if (($?)); then</code> thus translates to “if the last command
executed successfully, then…”.</p>
</div>
<p><img alt="The prompt is green after a successful command and red after a failed one" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/red-green-prompt.png">
<span class=imgcaption>The prompt is green after a successful command and red after a failed one</span></p>
<h4 id="displaying-dynamic-data-in-zsh">Displaying dynamic data in zsh</h4>
<p>Dynamic data can be injected in your prompt the same way than in bash,
by executing functions at rendering time. <code>zsh</code> however provides you
with <em>ternary conditionals</em>, that is to say expressions that either
evaluate to one value or the other depending on a condition, to reach
the same goal. A ternary conditional has the following syntax</p>
<div class="highlight"><pre><span></span><code>%<span class="o">(</span><span class="k"><</span>condition>.<span class="k"><</span>success<span class="w"> </span>value>.<span class="k"><</span>failure<span class="w"> </span>value><span class="o">)</span>
</code></pre></div>
<p>If the condition is true, then the expression is evaluated to the
success value. On the other hand, if the condition is false, the
expression will be evaluated to the failure value.</p>
<p>You can read a ternary conditional as <em>if condition, then, else</em>. It's
actually a common pattern called <em>ternary expression</em> you might
encounter in many programming languages.</p>
<p>Here is a list of useful built-in conditions provided by <code>zsh</code>.</p>
<table>
<thead>
<tr>
<th>Condition</th>
<th>Meaning</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>n?</code></td>
<td>True if the previous command exited with the exit status <em>n</em></td>
</tr>
<tr>
<td><code>nd</code></td>
<td>True if the day of the month is equal to n</td>
</tr>
<tr>
<td><code>nw</code></td>
<td>True if the day of the week is equal to n (Sunday = 0).</td>
</tr>
<tr>
<td><code>!</code></td>
<td>True if the shell is running with super-user privileges (as the <code>root</code> user)</td>
</tr>
</tbody>
</table>
<p><strong>Examples</strong></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PROMPT</span><span class="o">=</span><span class="s1">'%F{%(0?.green.red)}$ %f'</span>
</code></pre></div>
<p><img alt="Displays a dollar prompt in green if the last command was successful, or red if it failed" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/zsh-ternary-red-green.png">
<span class=imgcaption>Displays a dollar prompt in green if the last command was successful, or red if it failed</span></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PROMPT</span><span class="o">=</span><span class="s1">'%* %1~ %(!.#.$) '</span>
</code></pre></div>
<p><img alt="Display a dollar sign if you run your regular user, and a hash if you are running in super-user mode" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/zsh-ternary-root.png">
<span class=imgcaption>Display a dollar sign if you run your regular user, and a hash if you are running in super-user mode</span></p>
<div class="Note">
<p>The full list of ternary conditionals is available in the zsh
documentation<sup id="fnref:12"><a class="footnote-ref" href="#fn:12">12</a></sup>.</p>
</div>
<h3 id="adding-emoji-to-your-prompt">Adding emoji to your prompt</h3>
<p>Modern terminal support non-ASCII characters, such as emoji. Like
colors, they can be convenient to convey information in a very
space-efficient fashion.</p>
<p>For example, during the process of writing that book, I displayed the
associated total word count in my prompt to keep me motivated. That word
count would however only be displayed when I was located in the root
directory of the project, in the spirit of only displaying context when
necessary.</p>
<p><img alt="" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/word-count-emoji.png"></p>
<p><a id="shell-configuration-frameworks"></a></p>
<h2 id="shell-configuration-frameworks">Shell configuration frameworks</h2>
<p>Up until now, we have seen how to tailor your prompt by adding colors,
context, dynamic information computed on-the-fly. While you can
certainly spend hours customizing up to “perfection” (trust me, I have
been there…), you can also take another route and benefit from other
people's work, using a shell configuration <span class="gls"
key="framework">framework</span>.</p>
<p>These frameworks provide you with a large choice of prompt themes,
helpers, options, additional command auto-completions, plug-ins, and are
regularly updated by a community of developers around the world.</p>
<p>To this day, the most famous <code>zsh</code> configuration frameworks are Oh My
Zsh<sup id="fnref:13"><a class="footnote-ref" href="#fn:13">13</a></sup> and Prezto.<sup id="fnref:14"><a class="footnote-ref" href="#fn:14">14</a></sup> While we can't fully attribute <code>zsh</code>'s success
to them (Oh My Zsh was first released around 2010, 20 years after zsh's
first release), they certainly have helped in driving community adoption
in the last couple of years<sup id="fnref:15"><a class="footnote-ref" href="#fn:15">15</a></sup>.</p>
<p><img alt="Comparison of Google Trends associated with zsh and Oh My Zsh" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/zsh-adoption.png">
<span class=imgcaption>Comparison of Google Trends associated with zsh and Oh My Zsh</span></p>
<p>We will introduce you to the concepts behind Oh My Zsh, but it will then
be up to you to explore, and select a theme as well as plug-ins you like
(or even not use them at all!). After all, it is <em>your</em> development
environment, and henceforth, your choice.</p>
<div class="Note">
<p><code>bash</code> has a similar framework, inspired by Oh My Zsh, called
<code>bash-it</code>.<sup id="fnref:16"><a class="footnote-ref" href="#fn:16">16</a></sup> We won't cover it in details but we encourage you to
look at it if don't feel like using <code>zsh</code> but still want to use a
configuration framework.</p>
</div>
<h3 id="oh-my-zsh">Oh My Zsh</h3>
<p>Quoting the official website,</p>
<blockquote>
<p>Oh My Zsh is a delightful, open source, community-driven framework for
managing your Zsh configuration. It comes bundled with thousands of
helpful functions, helpers, plug-ins, themes, and a few things that
make you shout…</p>
</blockquote>
<p>To install it, run the following command in a shell, which will download
an installation script, and run it on your computer.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sh<span class="w"> </span>-c<span class="w"> </span><span class="s2">"</span><span class="k">$(</span>curl<span class="w"> </span>-fsSL<span class="w"> </span>https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh<span class="k">)</span><span class="s2">"</span>
</code></pre></div>
<p>Once the script has finished running, you should see a message stating
that Oh My Zsh has been installed, and that plug-ins, themes and options
should be enabled by changing the configuration living under <code>~/.zshrc</code>.</p>
<p>Before, we do, let's inspect our environment variables, to see how Oh My
Zsh configures itself.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">printenv</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>ZSH
<span class="nv">ZSH</span><span class="o">=</span>/home/br/.oh-my-zsh
</code></pre></div>
<p>That <code>ZSH</code> environment variable points to the Oh My Zsh installation
directory. The framework also injected a couple of other variables
defining specific configuration values.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>ZSH
<span class="nv">ZSH</span><span class="o">=</span>/home/br/.oh-my-zsh
<span class="nv">ZSH_ARGZERO</span><span class="o">=</span>zsh
<span class="nv">ZSH_CACHE_DIR</span><span class="o">=</span>/home/br/.oh-my-zsh/cache
<span class="nv">ZSH_COMPDUMP</span><span class="o">=</span>/home/br/.zcompdump-morenika-5.7.1
<span class="nv">ZSH_CUSTOM</span><span class="o">=</span>/home/br/.oh-my-zsh/custom
<span class="nv">ZSH_EVAL_CONTEXT</span><span class="o">=</span>toplevel
<span class="nv">ZSH_NAME</span><span class="o">=</span>zsh
<span class="nv">ZSH_PATCHLEVEL</span><span class="o">=</span>zsh-5.7.1-0-g8b89d0d
<span class="nv">ZSH_SPECTRUM_TEXT</span><span class="o">=</span><span class="s1">'Arma virumque cano Troiae qui primus ab oris'</span>
<span class="nv">ZSH_SUBSHELL</span><span class="o">=</span><span class="m">1</span>
<span class="nv">ZSH_THEME</span><span class="o">=</span>robbyrussell
<span class="nv">ZSH_VERSION</span><span class="o">=</span><span class="m">5</span>.7.1
</code></pre></div>
<h4 id="picking-a-theme">Picking a theme</h4>
<p>We can see that the default theme is <code>robbyrussell</code> (Robby Russell<sup id="fnref:17"><a class="footnote-ref" href="#fn:17">17</a></sup>
is the creator of Oh My Zsh). The full list of available themes is
available online<sup id="fnref:18"><a class="footnote-ref" href="#fn:18">18</a></sup>, along with screenshots.</p>
<p>You can also get the list by running the following command, as all
themes are defined in <code>$ZSH/themes</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-1<span class="w"> </span><span class="nv">$ZSH</span>/themes<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s/.zsh-theme//'</span>
3den
adben
af-magic
afowler
...
</code></pre></div>
<p>I suggest you scroll through the themes wiki, or simply pick a theme at
random from the previous command output, edit your <code>~/.zshrc</code>
configuration file by updating the value of the <code>ZSH_THEME</code> variable,
and run <code>source ~/.zshrc</code> to reload it. That will get you a whole new
shell theme!</p>
<p>Feel free to rinse and repeat until you find a theme that suits you. In
the case where no built-in theme finds grace in your eyes, you can also
explore the external theme wiki<sup id="fnref:19"><a class="footnote-ref" href="#fn:19">19</a></sup>. If you find an external theme you
like, download its associated <code>.zsh-theme</code> file, and place it under
<code>$ZSH/themes</code>, then edit <code>~/.zshrc</code>, and update the <code>ZSH_THEME</code>
accordingly.</p>
<div class="Note">
<p>If you want to further personalize a theme using some of the techniques
we covered in that chapter, I'd advise you clone it and maintain a
separate version, as your tweaks might get overridden at the next theme
update.</p>
<p>Export the <code>ZSH_CUSTOM</code> environment variable to <code>$ZSH/custom</code>, then run
the following commands.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>-p<span class="w"> </span><span class="nv">$ZSH</span>/custom/themes
$<span class="w"> </span><span class="nb">cp</span><span class="w"> </span><span class="nv">$ZSH</span>/themes/<span class="nv">$ZSH_THEME</span>.zsh-theme<span class="w"> </span><span class="nv">$ZSH</span>/custom/themes/<span class="nv">$ZSH_THEME</span>-custom.zsh-theme
</code></pre></div>
<p>Then add <code>ZSH_THEME=<old zsh theme>-custom</code> to your <code>~/.zshrc</code>.</p>
</div>
<h4 id="useful-configuration-options">Useful configuration options</h4>
<p>Oh My Zsh has a couple of options you can enable or disable by editing
<code>~/.zshrc</code>. I suggest you take a look at them and choose what to
activate. Here are some personal recommendations.</p>
<h5 id="automatic-command-correction">Automatic command correction</h5>
<p>zsh can suggest a command correction if it detects a mistyped command.
To enable the automatic command correction, add
<code>ENABLE_AUTO_CORRECTION='true'</code> to <code>~/.zshrc</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sl
zsh:<span class="w"> </span>correct<span class="w"> </span><span class="s1">'sl'</span><span class="w"> </span>to<span class="w"> </span><span class="s1">'ls'</span><span class="w"> </span><span class="o">[</span>nyae<span class="o">]</span>?<span class="w"> </span>y
Android<span class="w"> </span>bin<span class="w"> </span>Desktop<span class="w"> </span>Downloads<span class="w"> </span>Firefox_wallpaper.png<span class="w"> </span>Pictures
AndroidStudioProjects<span class="w"> </span>code<span class="w"> </span>Documents<span class="w"> </span>Dropbox<span class="w"> </span>Music<span class="w"> </span>Videos
</code></pre></div>
<p>The 4 options are:</p>
<ul>
<li><code>n</code> (no): run the mistyped command</li>
<li><code>y</code> (yes): run the suggested command</li>
<li><code>a</code> (abort): stop and do nothing</li>
<li><code>e</code> (edit): edit your command before re-running it</li>
</ul>
<div class="Note">
<p>zsh's auto-correction feature can sometimes be over-zealous and is not
to everyone's liking<sup id="fnref:20"><a class="footnote-ref" href="#fn:20">20</a></sup>. If you end up repeatedly fighting it for a
given command (e.g. <code>git status</code> wrongly autocorrected to <code>git stats</code>),
you can define an alias for the command by prefixing it with
<code>nocorrect</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nb">alias</span><span class="w"> </span>git<span class="w"> </span><span class="nv">status</span><span class="o">=</span><span class="s1">'nocorrect git status'</span>
</code></pre></div>
</div>
<h5 id="automatic-oh-my-zsh-updates">Automatic Oh My Zsh updates</h5>
<p>To make sure you regularly get new plug-ins and bug fixes, Oh My Zsh can
automatically and regularly update itself. To do so, set the following
options in <code>~/.zshrc</code>:</p>
<ul>
<li><code>DISABLE_UPDATE_PROMPT=true</code>: update Oh My Zsh without asking for
confirmation</li>
<li><code>UPDATE_ZSH_DAYS=30</code>: update Oh My Zsh every 30 days</li>
</ul>
<h4 id="add-plug-ins">Add plug-ins</h4>
<p>Oh My Zsh comes with more than 250 plug-ins, each of them either
defining aliases or improved auto-completion for a given set of
commands. Refer to the Oh My Zsh wiki page<sup id="fnref:21"><a class="footnote-ref" href="#fn:21">21</a></sup> to see the full list of
available plug-ins. To enable a given plug-in, add its name to the
<code>plugins</code> list in <code>~/.zshrc</code>, then run <code>source ~/.zshrc</code>.</p>
<p><strong>Example</strong>:</p>
<div class="highlight"><pre><span></span><code><span class="gd">- plugins=(git)</span>
<span class="gi">+ plugins=(git python)</span>
</code></pre></div>
<p>If you regularly use a command listed in the plug-in wiki page, you
should probably try to enable the associated plug-in! I however suggest
enabling the following general-purpose plug-ins.</p>
<ul>
<li><code>common-aliases</code><sup id="fnref:22"><a class="footnote-ref" href="#fn:22">22</a></sup>: Collection of useful aliases, not enabled by
default since they may change some user defined aliases</li>
<li><code>colored-man-pages</code>: colorize <code>man</code> pages</li>
</ul>
<p><img alt="Colorized man pages are much easier to read!" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/colored-man-pages.png">
<span class=imgcaption>Colorized man pages are much easier to read!</span></p>
<ul>
<li><code>extract</code>: define an <code>extract</code> alias that can extract any type of
archive (.zip, .tar.gz, .bzip, etc)<sup id="fnref:23"><a class="footnote-ref" href="#fn:23">23</a></sup></li>
</ul>
<p>The following plug-ins are not provided by default, I find them so
useful that I suggest you install them and give them a try.</p>
<ul>
<li><code>zsh-autosuggestions</code><sup id="fnref:24"><a class="footnote-ref" href="#fn:24">24</a></sup>: emulate the <code>fish</code> autosuggestion by
suggesting commands as you type them, saving you from using
<kbd>Ctrl</kbd> - <kbd>R</kbd> to look into your shell history. Any
suggestion can be accepted by hitting <kbd>→</kbd> or ignored by
just continuing typing.</li>
</ul>
<p><img alt="I just typed ls and I immediately get a completion suggestion" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/zsh-autosuggestion-before.png">
<span class=imgcaption>I just typed <code>ls</code> and I immediately get a completion suggestion</span></p>
<p><img alt="Suggestion accepted!" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/zsh-autosuggestion-after.png">
<span class=imgcaption>Suggestion accepted!</span></p>
<ul>
<li><code>zsh-syntax-highlighting</code><sup id="fnref:25"><a class="footnote-ref" href="#fn:25">25</a></sup>: provide syntax highlighting within
the zsh command line. It also colorizes the name of the command you
type in green if it is found, and in red if not.</li>
</ul>
<p><img alt="ls is a valid command" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/zsh-syntax-highlight-prompt-cmd-green.png">
<span class=imgcaption><code>ls</code> is a valid command</span></p>
<p><img alt="cmdnotfound is not" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/customize-shell/zsh-syntax-highlight-prompt-cmd-red.png">
<span class=imgcaption><code>cmdnotfound</code> is not</span></p>
<h4 id="uninstalling-oh-my-zsh">Uninstalling Oh My Zsh</h4>
<p>If you find that Oh My Zsh isn't for you, you can uninstall it by
running the <code>uninstall_oh_my_zsh</code> function. Your previous configuration
will be restored.</p>
<p><a id="summary"></a></p>
<h2 id="summary">Summary</h2>
<p>I strongly believe that learning how to configure and personalize your
own shell is an important part of becoming a developer. I'd even go as
far as calling it a ritual. On a personal level, it helped me overcome
the almost mystic reputation of the terminal by making it my own.</p>
<p>Configuring your shell might never really be fully completed. Do you
find yourself executing a long command repeatedly? Make it an alias. If
an alias does not cut it, or if it should take arguments, write a shell
function instead. Are you oftentimes wondering on which branch, project
or profile you are currently running? Add it to your prompt. If your
prompt starts to feel a little crowded, you might be able to condense it
by using colors and emoji.</p>
<p>Making your own tools and customizing your shell is an investment, but
it is also an inherent part of being a software developer, which will
allow you to do more, faster, and will help you feel more at home in
your shell. It's also quite a bit of fun!</p>
<p><a id="going-further"></a></p>
<h2 id="going-further">Going further</h2>
<p><strong>4.1</strong>: Look into your terminal's preferences and try to change the
color scheme, or remap ANSI colors to different RGB colors.</p>
<p><strong>4.2</strong>: Try different fonts, such as <code>Source Code Pro</code>,
<code>Fira Code Pro</code>, <code>Inconsolata</code> or <code>Jetbrains Mono</code> and pick the one you
like most</p>
<p><strong>4.3</strong>: Explore your terminal preferences, and experiment with
different settings.</p>
<p><strong>4.4</strong>: Try to change the colors of the different sections of your
prompt</p>
<footer>
<p>
<em>Essential Tools and Practices for the Aspiring Software Developer</em> is a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>The book will help you set up a productive development environment and get acquainted with tools and practices that, along with your programming languages of choice, will go a long way in helping you grow as a software developer.
It will cover subjects such as mastering the terminal, configuring and getting productive in a shell, the basics of code versioning with <code>git</code>, SQL basics, tools such as <code>Make</code>, <code>jq</code> and regular expressions, networking basics as well as software engineering and collaboration best practices.
</p>
<p>
If you are interested in the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</footer>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p><a href="https://gnometerminator.blogspot.com/p/introduction.html">https://gnometerminator.blogspot.com/p/introduction.html</a> <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p><a href="https://iterm2.com/">https://iterm2.com/</a> <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p><a href="https://github.com/tonsky/FiraCode">https://github.com/tonsky/FiraCode</a> <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p><a href="https://www.jetbrains.com/lp/mono/">https://www.jetbrains.com/lp/mono/</a> <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:5">
<p><a href="https://www.theverge.com/2019/6/4/18651872/apple-macos-catalina-zsh-bash-shell-replacement-features">https://www.theverge.com/2019/6/4/18651872/apple-macos-catalina-zsh-bash-shell-replacement-features</a> <a class="footnote-backref" href="#fnref:5" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:6">
<p><a href="https://fishshell.com">https://fishshell.com</a> <a class="footnote-backref" href="#fnref:6" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:7">
<p><a href="http://ezprompt.net">http://ezprompt.net</a> <a class="footnote-backref" href="#fnref:7" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:8">
<p><a href="http://zsh.sourceforge.net/Doc/Release/Prompt-Expansion.html">http://zsh.sourceforge.net/Doc/Release/Prompt-Expansion.html</a> <a class="footnote-backref" href="#fnref:8" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:9">
<p><a href="https://en.wikipedia.org/wiki/ANSI_escape_code">https://en.wikipedia.org/wiki/ANSI_escape_code</a> <a class="footnote-backref" href="#fnref:9" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn:10">
<p><a href="https://github.com/mbadolato/iTerm2-Color-Schemes">https://github.com/mbadolato/iTerm2-Color-Schemes</a> <a class="footnote-backref" href="#fnref:10" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn:11">
<p>Source: <a href="https://en.wikipedia.org/wiki/ANSI_escape_code#Colors">https://en.wikipedia.org/wiki/ANSI_escape_code#Colors</a> <a class="footnote-backref" href="#fnref:11" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn:12">
<p><a href="http://zsh.sourceforge.net/Doc/Release/Prompt-Expansion.html#Conditional-Substrings-in-Prompts">http://zsh.sourceforge.net/Doc/Release/Prompt-Expansion.html#Conditional-Substrings-in-Prompts</a> <a class="footnote-backref" href="#fnref:12" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
<li id="fn:13">
<p><a href="https://ohmyz.sh">https://ohmyz.sh</a> <a class="footnote-backref" href="#fnref:13" title="Jump back to footnote 13 in the text">↩</a></p>
</li>
<li id="fn:14">
<p><a href="https://github.com/sorin-ionescu/prezto">https://github.com/sorin-ionescu/prezto</a> <a class="footnote-backref" href="#fnref:14" title="Jump back to footnote 14 in the text">↩</a></p>
</li>
<li id="fn:15">
<p>Source:
<a href="https://trends.google.com/trends/explore?date=all&q=oh%20my%20zsh,%2Fm%2F0nrgk">https://trends.google.com/trends/explore?date=all&q=oh%20my%20zsh,%2Fm%2F0nrgk</a> <a class="footnote-backref" href="#fnref:15" title="Jump back to footnote 15 in the text">↩</a></p>
</li>
<li id="fn:16">
<p><a href="https://github.com/Bash-it/bash-it">https://github.com/Bash-it/bash-it</a> <a class="footnote-backref" href="#fnref:16" title="Jump back to footnote 16 in the text">↩</a></p>
</li>
<li id="fn:17">
<p><a href="https://github.com/robbyrussell">https://github.com/robbyrussell</a> <a class="footnote-backref" href="#fnref:17" title="Jump back to footnote 17 in the text">↩</a></p>
</li>
<li id="fn:18">
<p><a href="https://github.com/ohmyzsh/ohmyzsh/wiki/Themes">https://github.com/ohmyzsh/ohmyzsh/wiki/Themes</a> <a class="footnote-backref" href="#fnref:18" title="Jump back to footnote 18 in the text">↩</a></p>
</li>
<li id="fn:19">
<p><a href="https://github.com/ohmyzsh/ohmyzsh/wiki/External-themes">https://github.com/ohmyzsh/ohmyzsh/wiki/External-themes</a> <a class="footnote-backref" href="#fnref:19" title="Jump back to footnote 19 in the text">↩</a></p>
</li>
<li id="fn:20">
<p><a href="https://github.com/ohmyzsh/ohmyzsh/issues/534">https://github.com/ohmyzsh/ohmyzsh/issues/534</a> <a class="footnote-backref" href="#fnref:20" title="Jump back to footnote 20 in the text">↩</a></p>
</li>
<li id="fn:21">
<p><a href="https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins">https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins</a> <a class="footnote-backref" href="#fnref:21" title="Jump back to footnote 21 in the text">↩</a></p>
</li>
<li id="fn:22">
<p><a href="https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins#common-aliases">https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins#common-aliases</a> <a class="footnote-backref" href="#fnref:22" title="Jump back to footnote 22 in the text">↩</a></p>
</li>
<li id="fn:23">
<p><a href="https://www.xkcd.com/1168/">https://www.xkcd.com/1168/</a> <a class="footnote-backref" href="#fnref:23" title="Jump back to footnote 23 in the text">↩</a></p>
</li>
<li id="fn:24">
<p><a href="https://github.com/zsh-users/zsh-autosuggestions/blob/master/INSTALL.md">https://github.com/zsh-users/zsh-autosuggestions/blob/master/INSTALL.md</a> <a class="footnote-backref" href="#fnref:24" title="Jump back to footnote 24 in the text">↩</a></p>
</li>
<li id="fn:25">
<p><a href="https://github.com/zsh-users/zsh-syntax-highlighting/blob/master/INSTALL.md">https://github.com/zsh-users/zsh-syntax-highlighting/blob/master/INSTALL.md</a> <a class="footnote-backref" href="#fnref:25" title="Jump back to footnote 25 in the text">↩</a></p>
</li>
</ol>
</div>The shell's building blocks2020-04-04T00:00:00+02:002020-04-04T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2020-04-04:/the-shells-building-blocks<p>Something I still find striking after years of using a shell almost daily is how simple yet powerful its building blocks are. Chapter 1 covered commands, I/O streams and pipes. This chapter will cover environment variables, aliases and functions.</p><header>
<p>
This article is part of a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>
If you are interested in the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</header>
<h2 id="table-of-contents">Table of Contents</h2>
<!-- MarkdownTOC autolink="true" levels="2" autoanchor="true" -->
<ul>
<li><a href="#environment-variables">Environment variables</a></li>
<li><a href="#aliases">Aliases</a></li>
<li><a href="#functions">Functions</a></li>
<li><a href="#real-life-examples">Real life examples</a></li>
<li><a href="#summary">Summary</a></li>
<li><a href="#going-further">Going further</a></li>
</ul>
<!-- /MarkdownTOC -->
<h1 id="the-shells-building-blocks">The shell's building blocks</h1>
<p>As we have seen in the <a href="https://blog.balthazar-rouberol.com/discovering-the-terminal">previous chapters</a>, the shell is a program
allowing you to run other programs. It is an invaluable tool in the life
of a software engineer, as it provides you with a simple text-based
interface to control your computer and any program you might install or
write.</p>
<p>Something I still find striking after years of using a shell almost
daily is how simple yet powerful its building blocks are.</p>
<p><a href="https://blog.balthazar-rouberol.com/discovering-the-terminal">Chapter 1</a> covered commands, I/O streams and pipes. This chapter will
cover environment variables, aliases and functions.</p>
<p><a id="environment-variables"></a></p>
<h2 id="environment-variables">Environment variables</h2>
<p>Environment variables are key/value pairs that affect how running
programs behave. Another way to say that would be that environment
variables can allow you to tweak and personalize how certain programs,
amongst which your shell, work. They can also define what programs will
be called to perform a certain task.</p>
<p>Here are a few examples:</p>
<ul>
<li><code>SHELL</code> defines what shell your terminal runs (‘/bin/bash',
<code>/bin/zsh</code>, <code>/bin/fish</code>, etc)</li>
<li><code>HOME</code> defines where your home directory is located</li>
<li><code>EDITOR</code> defines what text editor program should be used to edit
text within your terminal (eg <code>nano</code>, <code>vim</code>, <code>emacs</code>, etc)</li>
</ul>
<h3 id="displaying-an-environment-variables-value">Displaying an environment variable's value</h3>
<p>To display the value of given environment variable, you can use the
<code>echo</code> command, followed by a dollar sign and the name of the variable:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$SHELL</span>
/bin/zsh
</code></pre></div>
<p>You can use the <code>printenv</code> command to list all environment variables
along with their value.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">printenv</span>
<span class="nv">USER</span><span class="o">=</span>br
<span class="nv">HOME</span><span class="o">=</span>/home/br
<span class="nv">LC_TERMINAL</span><span class="o">=</span>terminator
<span class="nv">SHELL</span><span class="o">=</span>/bin/zsh
<span class="nv">EDITOR</span><span class="o">=</span><span class="nb">vim</span>
<span class="nv">PWD</span><span class="o">=</span>/home/br/
<span class="nv">PAGER</span><span class="o">=</span><span class="nb">less</span>
</code></pre></div>
<p>For the sake of brevity, I've only displayed a subset of the environment
variables defined on my computer. These variables tell the following
story:</p>
<ul>
<li>my username is <code>br</code></li>
<li>all my personal data is stored in my <em>home directory</em>, located at
<code>/home/br</code></li>
<li>my default terminal is called <code>terminator</code></li>
<li>and whenever I open <code>terminator</code>, it runs the commands via the <code>zsh</code>
shell</li>
<li>my default text editor is <code>vim</code></li>
<li>I am currently located in my home directory</li>
<li>my default pager program is <code>less</code></li>
</ul>
<h3 id="changing-an-environment-variable">Changing an environment variable</h3>
<p>What is interesting about these environment variables is that they can
be changed, and with them, the behavior of other programs.</p>
<p>For example, let's change the value of our <code>HOME</code> environment variable,
defining where our home directory is.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">HOME</span><span class="o">=</span>/tmp
$<span class="w"> </span><span class="nb">cd</span>
$<span class="w"> </span><span class="nb">pwd</span>
/tmp
</code></pre></div>
<p>In the first line, I redefined the value of my <code>HOME</code> environment
variable from <code>/home/br</code> to <code>/tmp</code>. Remember when you learned that
running <code>cd</code> without arguments would take you back to your home
directory? Well, it's actually using the <code>HOME</code> environment variable to
figure out where your home directory is. Now that <code>HOME</code> has changed, so
has <code>cd</code>'s behavior.</p>
<p>Another example is <code>PAGER</code>. We saw that my environment had <code>PAGER=less</code>
defined by default, which explains why you find yourself reading text
within <code>less</code> when you open a man page. <code>man</code> fetches the actual
documentation and displays it in a pager, which itself is specified by
the <code>PAGER</code> environment variable. If you were to change that variable to
something else, like <code>more</code> or <code>bat</code>,<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup> it would then change <code>man</code>'s
behavior.</p>
<div class="Note">
<p>There is a difference between <code>SHELL</code> and <code>$SHELL</code>. The first one is the
name of an environment variable, and the latter represents its value.
Consequently, when we executed <code>echo $SHELL</code>, we told our shell to
lookup what value was associated with the <code>SHELL</code> environment variable,
and then display it to the screen via the <code>echo</code> command. <code>$</code> is what we
call a <em>dereference operator</em> in that context.</p>
</div>
<h3 id="defining-new-variables">Defining new variables</h3>
<p>Not only can you change an existing environment variable, but you can
also define a new one. If a non-existing variable is <code>echo</code>-ed, it will
simply be replaced by an empty string.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$NEW_VAR</span>
$<span class="w"> </span><span class="nv">NEW_VAR</span><span class="o">=</span>my-new-env-var
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$NEW_VAR</span>
my-new-env-var
</code></pre></div>
<p>If you define an environment variable this way, it will only be visible
by the shell itself, but not by any command executed by your shell (also
called <em>subprocesses</em>). To make an environment variable visible by a
subprocess, you need to define it after the <code>export</code> keyword.</p>
<p>To illustrate that, we will create our first <em>shell script</em>: a program
executing shell commands one after the others.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span><span class="s"><<EOF > echo_var.sh</span>
<span class="s">echo $NEW_VAR</span>
<span class="s">EOF</span>
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>echo_var.sh
<span class="nb">echo</span><span class="w"> </span><span class="nv">$NEW_VAR</span>
</code></pre></div>
<p>As you can see, the <code>echo_var.sh</code> <em>script</em> only contains one shell
command: <code>echo $NEW_VAR</code>.</p>
<p>To execute that bash script, we can run <code>bash echo_var.sh</code>, and all
instructions within that script will be executed by <code>bash</code>. Let's have a
look at what executing that script displays on the screen with and
without <code>export</code>-ing that variable.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">NEW_VAR</span><span class="o">=</span>my-new-var
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$NEW_VAR</span>
my-new-var
$<span class="w"> </span><span class="nb">bash</span><span class="w"> </span>echo_var.sh
$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">NEW_VAR</span><span class="o">=</span>my-new-var
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$NEW_VAR</span>
my-new-var
$<span class="w"> </span><span class="nb">bash</span><span class="w"> </span>echo_var.sh
my-new-var
</code></pre></div>
<p>As you can see, the <code>echo_var.sh</code> subprocess can see the <code>NEW_VAR</code>
environment variable after it has been <code>export</code>-ed by its parent shell.</p>
<p>This can very useful if you write programs: some parameters can have a
sane default value but can also be overridden by specifying an
environment variable. <code>grep</code> does this for example: reading the <code>grep</code>
<code>man</code> page, we see:</p>
<blockquote>
<p><code>GREP_OPTIONS</code> May be used to specify default options that will be
placed at the beginning of the argument list.</p>
</blockquote>
<h3 id="removing-environment-variables">Removing environment variables</h3>
<p>You can remove an environment variable by using the <code>unset</code> keyword:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">unset</span><span class="w"> </span>NEW_VAR
$<span class="w"> </span><span class="nb">bash</span><span class="w"> </span>echo_var.sh
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$NEW_VAR</span>
$
</code></pre></div>
<h3 id="the-case-of-path">The case of <code>PATH</code></h3>
<p>Until that point, we've executed commands in the shell, and things
happened. It was a simple world and it was nice. You might wonder <em>what
would happen if I gave the shell a non-existent command though?</em>. Well,
I'm glad you asked. Ten points for Gryffindor.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cmdnotfound
zsh:<span class="w"> </span><span class="nb">command</span><span class="w"> </span>not<span class="w"> </span>found:<span class="w"> </span>cmdnotfound
</code></pre></div>
<p>The <code>cmdnotfound</code> command, like its name implies, is not found. But what
makes a command be found then? What makes the shell happily comply when
we type <code>ls</code>, and makes it complain when we type <code>cmdnotfound</code>? It turns
out that this is due to an environment variable called <code>PATH</code>, listing
all directories in which executable programs can be found.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$PATH</span>
/home/br/bin:/home/br/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
</code></pre></div>
<p>This means that for any command passed to the shell, it will look into
these directories (separated by a colon) in search for the program I'm
trying to run.</p>
<p>For example, if I type</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span>
</code></pre></div>
<p>into my shell, it will look into <code>/home/br/bin</code>, <code>/home/br/.local/bin</code>,
<code>/usr/local/sbin</code>, etc, until it finds it in <code>/bin</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>/bin
...
<span class="nb">chmod</span><span class="w"> </span>dd<span class="w"> </span>ed<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>ps<span class="w"> </span>sh<span class="w"> </span>tcsh<span class="w"> </span>zsh
</code></pre></div>
<p>If the command is not found in any of the directories listed in <code>PATH</code>,
then it is not found.</p>
<p>This means that you can also redefine <code>PATH</code> to force your shell to look
into new directories. In fact, this is exactly what I've done to make it
look into <code>/home/br/bin</code>, where I store tools of my making.</p>
<div class="Warning">
<p>You can mess with up your shell by running <code>unset PATH</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">unset</span><span class="w"> </span>PATH
$<span class="w"> </span><span class="nb">ls</span>
zsh:<span class="w"> </span><span class="nb">command</span><span class="w"> </span>not<span class="w"> </span>found:<span class="w"> </span><span class="nb">ls</span>
</code></pre></div>
<p>However, calling a command by using its absolute or relative path still
works, as <code>PATH</code> is only used to look for commands that only have been
invoked by name.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">unset</span><span class="w"> </span>PATH
$<span class="w"> </span>/bin/ls
bin<span class="w"> </span>code<span class="w"> </span>Documents<span class="w"> </span>..
...
</code></pre></div>
</div>
<p>There is a useful command you can use to know what a program is and
where it is found: <code>which</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">which</span><span class="w"> </span><span class="nb">less</span>
/usr/bin/less
$<span class="w"> </span><span class="nb">which</span><span class="w"> </span><span class="nb">bash</span>
/usr/local/bin/bash
$<span class="w"> </span><span class="nb">which</span><span class="w"> </span><span class="nb">ls</span>
ls:<span class="w"> </span>aliased<span class="w"> </span>to<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-G
</code></pre></div>
<p>Wait. What? What's an alias?</p>
<p><a id="aliases"></a></p>
<h2 id="aliases">Aliases</h2>
<p>An <em>alias</em> allows you to define custom commands. In the previous
example, running <code>ls</code> would actually run <code>ls -G</code>, which enables
colorized output.</p>
<p>You can define an alias by using the <code>alias</code> keyword.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">alias</span><span class="w"> </span><span class="nb">ls</span><span class="o">=</span><span class="s1">'ls -G'</span>
</code></pre></div>
<p>There are a couple of reasons you might want to define aliases:</p>
<ul>
<li>redefining a command's behavior (ex: always using <code>ls</code> with the <code>-G</code>
option)</li>
<li>shortening a command's name to make it quicker to type (ex:
<code>alias ..='cd ..'</code>)</li>
<li>creating new commands altogether (ex:
<code>alias filesize='ls --size --human-readable -1'</code>)</li>
</ul>
<p>To see the underlying command that will be executed by an alias, you can
type <code>alias <name></code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">alias</span><span class="w"> </span><span class="nv">filesize</span><span class="o">=</span><span class="s1">'ls --size --human-readable -1'</span>
$<span class="w"> </span><span class="nb">alias</span><span class="w"> </span>filesize
<span class="nb">alias</span><span class="w"> </span><span class="nv">filesize</span><span class="o">=</span><span class="s1">'ls --size --human-readable -1'</span>
</code></pre></div>
<p>Aliases are very simple yet powerful. They allow you to customize your
shell to your liking, create new commands without having to remember a
lot of options, and decrease the time you spend typing, all of which
should make you feel more productive.</p>
<div class="Note">
<p>Aliases can be “nested”. If you define <code>ls</code> as an alias of <code>ls -G</code> and
<code>filesize</code> as an alias of <code>ls --size --human-readable -1</code>, your shell
will unwrap both aliases and execute <code>ls -G --size --human-readable -1</code>
when you type <code>filesize</code>.</p>
</div>
<p>When we're executing <code>filesize bin</code>, the shell will see that <code>filesize</code>
is an alias for <code>ls --size --human-readable -1</code> and will actually
execute the command <code>ls --size --human-readable -1 bin</code> behind the
scenes. This simply is done by replacing the alias by its definition in
the command itself. Aliases can however fall short if we want to do
something a more complex than this.</p>
<p>For example, one of my favorite productivity tools is <code>mkcd</code>, which
creates a directory and steps into it right after. It saves you from
typing</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>new-dir
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>new-dir
</code></pre></div>
<p>where you can just type</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkcd<span class="w"> </span>new-dir
</code></pre></div>
<p>An alias can't really help here, because we are talking about aliasing
two commands with a single alias, which does not work. Enter
<em>functions</em>.</p>
<p><a id="functions"></a></p>
<h2 id="functions">Functions</h2>
<p>According to the <code>bash</code> <code>man</code> page:</p>
<blockquote>
<p>A shell function is an object that is called like a simple command and
executes a compound command with a new set of positional parameters.</p>
</blockquote>
<p>Let's see what that looks like in practice. A function is declared this
way.</p>
<div class="highlight"><pre><span></span><code><span class="k">function</span><span class="w"> </span>name<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="c1"># ...</span>
<span class="o">}</span>
</code></pre></div>
<p>If your function is expecting arguments, these can be accessed by using
<code>$n</code> where <code>n</code> is a number. For example, <code>$1</code> is the first function
argument, <code>$2</code> its second argument, etc. With that in mind, we can now
declare our <code>mkcd</code> function.</p>
<div class="highlight"><pre><span></span><code><span class="k">function</span><span class="w"> </span>mkcd<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>-p<span class="w"> </span><span class="nv">$1</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="nv">$1</span>
<span class="o">}</span>
</code></pre></div>
<p>Let's now see <code>mkcd</code> in action!</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="k">function</span><span class="w"> </span>mkcd<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">target</span><span class="o">=</span><span class="nv">$1</span>
<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>-p<span class="w"> </span><span class="nv">$target</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="nv">$target</span>
<span class="o">}</span>
$<span class="w"> </span><span class="nb">pwd</span>
/home/br
$<span class="w"> </span>mkcd<span class="w"> </span><span class="nb">test</span>
$<span class="w"> </span><span class="nb">pwd</span>
/home/br/test
</code></pre></div>
<p>You can use the <code>typeset -f</code> command to see how a function was defined
(or <code>which <function-name></code>, although that only works in <code>zsh</code>).</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">typeset</span><span class="w"> </span>-f<span class="w"> </span>mkcd
mkcd<span class="w"> </span><span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>-p<span class="w"> </span><span class="nv">$1</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="nv">$1</span>
<span class="o">}</span>
</code></pre></div>
<p><a id="real-life-examples"></a></p>
<h2 id="real-life-examples">Real life examples</h2>
<p>These are some of the environment variables, aliases and functions I
have defined for myself.</p>
<h3 id="shorter-navigation-aliases">Shorter navigation aliases</h3>
<div class="highlight"><pre><span></span><code><span class="nb">alias</span><span class="w"> </span>..<span class="o">=</span><span class="s1">'cd ..'</span>
<span class="nb">alias</span><span class="w"> </span>...<span class="o">=</span><span class="s1">'cd ../..'</span>
</code></pre></div>
<h3 id="colorize-commands-output">Colorize commands output</h3>
<div class="highlight"><pre><span></span><code><span class="nb">alias</span><span class="w"> </span><span class="nb">ls</span><span class="o">=</span><span class="s1">'ls --color=auto'</span>
<span class="nb">alias</span><span class="w"> </span><span class="nb">grep</span><span class="o">=</span><span class="s1">'grep --color=auto'</span>
<span class="nb">alias</span><span class="w"> </span><span class="nv">ip</span><span class="o">=</span><span class="s1">'ip --color'</span>
</code></pre></div>
<h3 id="alias-commands-i-never-remember">Alias commands I never remember</h3>
<div class="highlight"><pre><span></span><code><span class="c1"># https://xkcd.com/1168/</span>
<span class="nb">alias</span><span class="w"> </span><span class="nv">untar</span><span class="o">=</span><span class="s1">'tar -zxvf'</span>
</code></pre></div>
<h3 id="have-homebin-be-part-of-path">Have <code>$HOME/bin</code> be part of <code>PATH</code></h3>
<div class="highlight"><pre><span></span><code><span class="nb">export</span><span class="w"> </span><span class="nv">PATH</span><span class="o">=</span><span class="nv">$PATH</span>:<span class="nv">$HOME</span>/bin
</code></pre></div>
<p>By extending my <code>PATH</code> this way, I can then put every single tool I
create into <code>$HOME/bin</code> and have it be usable right-away.</p>
<h3 id="a-backup-function">A backup function</h3>
<div class="highlight"><pre><span></span><code><span class="k">function</span><span class="w"> </span>bak<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">cp</span><span class="w"> </span>-r<span class="w"> </span><span class="nv">$1</span><span class="w"> </span><span class="nv">$1</span>.bak
<span class="o">}</span>
</code></pre></div>
<p>This function can be used to backup a file or directory. I regularly use
this when I'm about to edit a critical file and I want to make sure I
can revert my changes if needed.</p>
<h3 id="password-generation-function">Password generation function</h3>
<p>This function generate a password composed of alphanumeric characters,
of default length 32.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="k">function</span><span class="w"> </span>genpass<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">passlen</span><span class="o">=</span><span class="si">${</span><span class="nv">1</span><span class="k">:-</span><span class="nv">32</span><span class="si">}</span>
<span class="w"> </span><span class="c1"># Note: LC_ALL=C is needed for macos compatibility</span>
<span class="w"> </span><span class="nv">LC_ALL</span><span class="o">=</span>C<span class="w"> </span><span class="nb">tr</span><span class="w"> </span>-cd<span class="w"> </span><span class="s1">'[:alnum:]'</span><span class="w"> </span><span class="k"><</span><span class="w"> </span>/dev/urandom<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">fold</span><span class="w"> </span>-w<span class="w"> </span><span class="nv">$passlen</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">head</span><span class="w"> </span>-n1
<span class="o">}</span>
$<span class="w"> </span>genpass
GQROc0tnABqfYH0qpMMwSPYFgcY7OANB
$<span class="w"> </span>genpass<span class="w"> </span><span class="m">50</span>
WkeQ14E8FIQZN7XlN7yPkYK4yhMOvpAuNzZivKwODNkskh0uq0
</code></pre></div>
<h3 id="the-weather-in-your-terminal">The weather in your terminal</h3>
<div class="highlight"><pre><span></span><code><span class="k">function</span><span class="w"> </span>weather<span class="w"> </span><span class="o">{</span>
<span class="w"> </span>curl<span class="w"> </span><span class="s2">"wttr.in/</span><span class="si">${</span><span class="nv">1</span><span class="k">:-</span><span class="nv">lyon</span><span class="si">}</span><span class="s2">?m"</span>
<span class="o">}</span>
</code></pre></div>
<p>This function uses <code>curl</code> to send an HTTP request to the
<code>http://wttr.in</code> website, that displays weather forecasts in a
terminal-friendly way. So I can just type <code>weather mycity</code> and <em>voila</em>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>weather<span class="w"> </span>lyon
Weather<span class="w"> </span>report:<span class="w"> </span>lyon
<span class="w"> </span><span class="se">\ </span><span class="w"> </span>/<span class="w"> </span>Sunny
<span class="w"> </span>.-.<span class="w"> </span><span class="m">17</span><span class="w"> </span>°C
<span class="w"> </span>―<span class="w"> </span><span class="o">(</span><span class="w"> </span><span class="o">)</span><span class="w"> </span>―<span class="w"> </span>↖<span class="w"> </span><span class="m">6</span><span class="w"> </span>km/h
<span class="w"> </span><span class="sb">`</span>-<span class="s1">' 10 km</span>
<span class="s1"> / \ 0.0 mm</span>
<span class="s1"> ┌─────────────┐</span>
<span class="s1">┌──────────────────────────────┬───────────────────────┤ Sat 04 Apr ├───────────────────────┬──────────────────────────────┐</span>
<span class="s1">│ Morning │ Noon └──────┬──────┘ Evening │ Night │</span>
<span class="s1">├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤</span>
<span class="s1">│ \ / Partly cloudy │ \ / Partly cloudy │ \ / Partly cloudy │ \ / Partly cloudy │</span>
<span class="s1">│ _ /"".-. 7..8 °C │ _ /"".-. 13 °C │ _ /"".-. 13 °C │ _ /"".-. 10..11 °C │</span>
<span class="s1">│ \_( ). ← 5-6 km/h │ \_( ). ↙ 5 km/h │ \_( ). ← 5-10 km/h │ \_( ). ↖ 8-17 km/h │</span>
<span class="s1">│ /(___(__) 10 km │ /(___(__) 10 km │ /(___(__) 10 km │ /(___(__) 10 km │</span>
<span class="s1">│ 0.0 mm | 0% │ 0.0 mm | 0% │ 0.0 mm | 0% │ 0.0 mm | 0% │</span>
<span class="s1">└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘</span>
<span class="s1"> ┌─────────────┐</span>
<span class="s1">┌──────────────────────────────┬───────────────────────┤ Sun 05 Apr ├───────────────────────┬──────────────────────────────┐</span>
<span class="s1">│ Morning │ Noon └──────┬──────┘ Evening │ Night │</span>
<span class="s1">├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤</span>
<span class="s1">│ \ / Sunny │ \ / Sunny │ \ / Sunny │ \ / Partly cloudy │</span>
<span class="s1">│ .-. 10..12 °C │ .-. 16 °C │ .-. 14..15 °C │ _ /"".-. 10..12 °C │</span>
<span class="s1">│ ― ( ) ― ↖ 14-18 km/h │ ― ( ) ― ↑ 23-27 km/h │ ― ( ) ― ↑ 15-25 km/h │ \_( ). ↑ 13-26 km/h │</span>
<span class="s1">│ `-'</span><span class="w"> </span><span class="m">10</span><span class="w"> </span>km<span class="w"> </span>│<span class="w"> </span><span class="sb">`</span>-<span class="s1">' 10 km │ `-'</span><span class="w"> </span><span class="m">10</span><span class="w"> </span>km<span class="w"> </span>│<span class="w"> </span>/<span class="o">(</span>___<span class="o">(</span>__<span class="o">)</span><span class="w"> </span><span class="m">10</span><span class="w"> </span>km<span class="w"> </span>│
│<span class="w"> </span>/<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="m">0</span>.0<span class="w"> </span>mm<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span>│<span class="w"> </span>/<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="m">0</span>.0<span class="w"> </span>mm<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span>│<span class="w"> </span>/<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="m">0</span>.0<span class="w"> </span>mm<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span>│<span class="w"> </span><span class="m">0</span>.0<span class="w"> </span>mm<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span>│
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
<span class="w"> </span>┌─────────────┐
┌──────────────────────────────┬───────────────────────┤<span class="w"> </span>Mon<span class="w"> </span><span class="m">06</span><span class="w"> </span>Apr<span class="w"> </span>├───────────────────────┬──────────────────────────────┐
│<span class="w"> </span>Morning<span class="w"> </span>│<span class="w"> </span>Noon<span class="w"> </span>└──────┬──────┘<span class="w"> </span>Evening<span class="w"> </span>│<span class="w"> </span>Night<span class="w"> </span>│
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│<span class="w"> </span><span class="se">\ </span><span class="w"> </span>/<span class="w"> </span>Sunny<span class="w"> </span>│<span class="w"> </span><span class="se">\ </span><span class="w"> </span>/<span class="w"> </span>Sunny<span class="w"> </span>│<span class="w"> </span><span class="se">\ </span><span class="w"> </span>/<span class="w"> </span>Sunny<span class="w"> </span>│<span class="w"> </span><span class="se">\ </span><span class="w"> </span>/<span class="w"> </span>Clear<span class="w"> </span>│
│<span class="w"> </span>.-.<span class="w"> </span><span class="m">12</span>..13<span class="w"> </span>°C<span class="w"> </span>│<span class="w"> </span>.-.<span class="w"> </span><span class="m">16</span><span class="w"> </span>°C<span class="w"> </span>│<span class="w"> </span>.-.<span class="w"> </span><span class="m">14</span>..15<span class="w"> </span>°C<span class="w"> </span>│<span class="w"> </span>.-.<span class="w"> </span><span class="m">11</span><span class="w"> </span>°C<span class="w"> </span>│
│<span class="w"> </span>―<span class="w"> </span><span class="o">(</span><span class="w"> </span><span class="o">)</span><span class="w"> </span>―<span class="w"> </span>↖<span class="w"> </span><span class="m">18</span>-22<span class="w"> </span>km/h<span class="w"> </span>│<span class="w"> </span>―<span class="w"> </span><span class="o">(</span><span class="w"> </span><span class="o">)</span><span class="w"> </span>―<span class="w"> </span>↑<span class="w"> </span><span class="m">22</span>-28<span class="w"> </span>km/h<span class="w"> </span>│<span class="w"> </span>―<span class="w"> </span><span class="o">(</span><span class="w"> </span><span class="o">)</span><span class="w"> </span>―<span class="w"> </span>↑<span class="w"> </span><span class="m">14</span>-24<span class="w"> </span>km/h<span class="w"> </span>│<span class="w"> </span>―<span class="w"> </span><span class="o">(</span><span class="w"> </span><span class="o">)</span><span class="w"> </span>―<span class="w"> </span>↑<span class="w"> </span><span class="m">8</span>-16<span class="w"> </span>km/h<span class="w"> </span>│
│<span class="w"> </span><span class="sb">`</span>-<span class="s1">' 10 km │ `-'</span><span class="w"> </span><span class="m">10</span><span class="w"> </span>km<span class="w"> </span>│<span class="w"> </span><span class="sb">`</span>-<span class="s1">' 10 km │ `-'</span><span class="w"> </span><span class="m">10</span><span class="w"> </span>km<span class="w"> </span>│
│<span class="w"> </span>/<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="m">0</span>.0<span class="w"> </span>mm<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span>│<span class="w"> </span>/<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="m">0</span>.0<span class="w"> </span>mm<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span>│<span class="w"> </span>/<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="m">0</span>.0<span class="w"> </span>mm<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span>│<span class="w"> </span>/<span class="w"> </span><span class="se">\ </span><span class="w"> </span><span class="m">0</span>.0<span class="w"> </span>mm<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span>│
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
Location:<span class="w"> </span>Lyon,<span class="w"> </span>Métropole<span class="w"> </span>de<span class="w"> </span>Lyon,<span class="w"> </span>Circonscription<span class="w"> </span>départementale<span class="w"> </span>du<span class="w"> </span>Rhône,<span class="w"> </span>Auvergne-Rhône-Alpes,<span class="w"> </span>France<span class="w"> </span><span class="o">[</span><span class="m">45</span>.7578137,4.8320114<span class="o">]</span>
</code></pre></div>
<p><a id="summary"></a></p>
<h2 id="summary">Summary</h2>
<p>Environment variables, aliases and functions are simple yet powerful to
change the shell's behavior into something that feels more intuitive.
You feel like <code>nano</code> is not shiny enough and prefer using <code>vim</code> instead?
Sure. Define <code>EDITOR=vim</code>. Any command interacting with an editor would
then use <code>vim</code> instead of <code>nano</code>.</p>
<p>Aliases are a great way to reduce mental friction in the shell by hiding
away complex commands, or just reducing the amount of typing you have to
do. When aliases start being not powerful enough because you want to
execute multiple commands, you can then have a look at functions
instead.</p>
<p>Everything we have seen so far however had an ephemeral effect, as
changes you made would disappear when you close your shell session. In
the next chapter, we will go dive into how to persistently configure
your shell to improve your day-to-day experience and productivity.</p>
<p><a id="going-further"></a></p>
<h2 id="going-further">Going further</h2>
<p><strong>3.1</strong>: Write a <code>cat</code> alias that displays <code>meow</code> on screen.</p>
<p><strong>3.2</strong>: Write a <code>restorebak</code> function that takes a filename as only
argument and renames <code>$1.bak</code> into <code>$1</code>.</p>
<p><strong>3.3</strong>: Unset the <code>PATH</code> environment variable and then export it back
so that you can use <code>ls</code> again.</p>
<footer>
<p>
<em>Essential Tools and Practices for the Aspiring Software Developer</em> is a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>The book will help you set up a productive development environment and get acquainted with tools and practices that, along with your programming languages of choice, will go a long way in helping you grow as a software developer.
It will cover subjects such as mastering the terminal, configuring and getting productive in a shell, the basics of code versioning with <code>git</code>, SQL basics, tools such as <code>Make</code>, <code>jq</code> and regular expressions, networking basics as well as software engineering and collaboration best practices.
</p>
<p>
If you are interested in the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</footer>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p><a href="https://github.com/sharkdp/bat">https://github.com/sharkdp/bat</a> <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>My pizza recipe2020-03-18T00:00:00+01:002020-03-18T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2020-03-18:/my-pizza-recipe<p><img alt="pizza" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-done.jpg"> I've always enjoyed a good looking Neapolitan pizza. You know, the ones with the <a href="https://www.napolike.it/wp-content/uploads/2017/12/Pizze-gratis-al-Napoli-Pizza-village-per-il-riconoscimento-unesco.jpeg">puffy, slightly burned crust</a>. I have probably baked dozens of them during the last couple of years, but only recently did I become satisfied enough with my recipe to feel comfortable sharing it.</p><div class="Note">
<p>If you're reading this, do yourself a favor. Stop right there, and go to this <a href="/neapolitan-pizza-dough-recipe">post</a> instead. I was young and foolish, and didn't know any better.</p>
</div>
<p>I've always enjoyed a good looking Neapolitan pizza. You know, the ones with the <a href="https://www.napolike.it/wp-content/uploads/2017/12/Pizze-gratis-al-Napoli-Pizza-village-per-il-riconoscimento-unesco.jpeg">puffy, slightly burned crust</a>. I have probably baked dozens of them during the last couple of years, but only recently did I become satisfied enough with my recipe to feel comfortable sharing it.</p>
<h2 id="what-youll-need">What you'll need</h2>
<p>I don't think you <em>need</em> a professional oven, a dough mixer or a pizza stone to get nice results. Hear me out: I'm not saying you don't need any of these, but you certainly can do without and still enjoy a delicious slice.</p>
<p>This is what I use for this recipe:</p>
<ul>
<li>a pizza tray</li>
<li>a tupperware</li>
<li>a <a href="https://www.amazon.fr/Buyer-4858-00N-Raclette-Corne-Blanche/dp/B000ECUDVK/ref=pd_sbs_328_2/260-9517784-3248614?_encoding=UTF8&pd_rd_i=B000ECUDVK&pd_rd_r=9e2c7ea4-d71a-42dc-a173-3997bc28e9c8&pd_rd_w=rYY4y&pd_rd_wg=e4W69&pf_rd_p=72159c7a-2bb2-4a15-aa35-b315ce8f5c64&pf_rd_r=53M9R9PNK41E8TSHJ9SM&psc=1&refRID=53M9R9PNK41E8TSHJ9SM">scraper</a></li>
<li>a bowl</li>
<li>a tall glass</li>
</ul>
<h3 id="ingredients">Ingredients</h3>
<p>For 2 pizzas, you'll need:</p>
<ul>
<li>400g of flour</li>
<li>6g of salt (I personally use kosher salt, although more out of habit than anything else)</li>
<li>220mL of water</li>
<li>6g of fresh baker's yeast (dried yeast works as well. I prefer it fresh.)</li>
<li>time</li>
</ul>
<p>When it comes to the flour, the traditional flour is called <em>Double zero</em> (00) flour. This means that it's ground extremely fine. It's usually found in specialized Italian shops.</p>
<div class="Note">
<p>What's important to note here is that a flour is characterized by <a href="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-mix.jpeg">several things</a>:</p>
<ul>
<li>how fine it was ground</li>
<li>its protein content</li>
</ul>
<p>Pizza dough needs a high amount of gluten in the flour (14% is recommended) to develop its elasticity, and you can have 00 flour with a very low amount of protein (think cake flour), which will probably give you disappointing results.</p>
</div>
<p>I've never managed to find 00 pizza flour close to my place, but I found a good organic T65 flour (French T65 flours are commonly used in bread making) with 12% of protein which gives good results.</p>
<h2 id="recipe">Recipe</h2>
<h3 id="petit-levain">Petit levain</h3>
<p>The first thing we're going to make is a <em>petit levain</em>: a mix of water, flour and yeast aiming at gently activating the yeast coming out of your fridge of freezer.</p>
<p>Mix 50g of flour, 100mL and the yeast. Stir with a wooden spoon, as the yeast isn't too fond of metal.</p>
<h3 id="autolysis">Autolysis</h3>
<p>Mix the rest of the flour and water with the salt, and let it rest for half and hour, for an <em>autolysis</em> phase. During that phase, the water will get fully absorbed by the flour, which will make it easier to work with. The water will also start to degrade the flour into simpler sugars that will be more easily digested by the yeast when we add it, increasing the yeast activity.</p>
<div class="Note">
In theory, we should add the salt after the autolysis process, but as I use kosher salt, I find it easier to add it at this step rather than working it into the dough afterwards.
</div>
<h3 id="kneading">Kneading</h3>
<p>After having waited a good half-hour, incorporate the <em>petit levain</em> with the mix of flour, salt and water, by using your dough scraper. It's ok if there is a bit of flour left in the bowl, it will get absorbed during the kneading. Wait for 10, 15 minutes.</p>
<p><img alt="mix" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-mix.jpeg"></p>
<p>Pop the dough on your table (add a little bit of flour on the table if there isn't any left in the bowl), and knead it for about 10 minutes. Feel free to stop when the dough is pretty smooth and supple.</p>
<p><img alt="ball" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-ball.jpeg"></p>
<p>Put the dough in a closed container, and put it to rest in the fridge for 48 to 72h. Yep, you read that right. If you're thinking <em>"I am my own person, I do what I want, and I want to eat that pizza tonight"</em>, sure, let the dough rest for about 2h on your kitchen counter instead, under a damp towel.</p>
<p>Letting your dough rest for a long time will give it time to ferment and develop flavors and aromas. Trust me, it will smell fantastic when you open the container.</p>
<h3 id="shaping">Shaping</h3>
<p>Take the container of the fridge about 6h (if you can) before cooking time, and divide it into 2 even parts.</p>
<div class="Note">
Letting the dough out early will give it time to "relax" and will make it easier to stretch.
</div>
<p><img alt="tuppwerware" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-tupperware.jpeg">
<img alt="balls" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-balls.jpeg"></p>
<p><a href="https://www.youtube.com/watch?v=v5t5MEZt6LM">Shape</a> each half into a nice stretched ball, and put them in a closed container until you start cooking (or put the second one in the fridge if you're only planning to make one).</p>
<h3 id="stretching">Stretching</h3>
<p>When stretching your dough, you want to make sure to <em>never</em> use a rolling pin, and never touch the outside rim, otherwise you'll chase the air out and the crust won't expand as much as we'd want. I learned a ton from that <a href="https://www.youtube.com/watch?v=9f9-xTcKzZo">super short video</a>, which really helped me getting that Neapolitan look I wanted.</p>
<p><img alt="streched" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-streched.jpeg"></p>
<p>Sprinkle flour on your pizza tray, and pop the dough on it.</p>
<h3 id="toppings">Toppings</h3>
<p>First off, add some tomato sauce. One important thing is to avoid putting too much, which would dampen the dough and make the pizza quite watery. I usually use 2 big spoons of sauce, and spread it in circles, starting from the center. Make sure to avoid touching the outside rim.</p>
<p><img alt="with tomato" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-tomato.jpeg"></p>
<p>Pre-heat your oven at 250°C (480°F) with convection, if possible.</p>
<p>At that point, you're going to have to make choices. I won't tell you what you should put on your pizza, but I can describe my favorite one though, which has (put on the pizza in that order)</p>
<ul>
<li>small onions slices, previously cooked in olive oil</li>
<li>shreds of mozzarella di bufala</li>
<li>some genovese pesto here and there</li>
<li>black olives</li>
<li>oregano</li>
</ul>
<p>Drizzle the rim with olive oil, which will give it a nice golden color at cooking time.</p>
<p><img alt="pre oven" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-pre-oven.jpeg"></p>
<h3 id="cooking">Cooking</h3>
<p>When your oven is hot, cook your pizza for about 8 minutes.</p>
<p>The rest is on you.</p>
<p><img alt="post oven" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/pizza-recipe/pizza-post-oven.jpeg"></p>Text processing in the shell2020-03-14T00:00:00+01:002020-03-14T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2020-03-14:/text-processing-in-the-shell<p>One of the things that makes the shell an invaluable tool is the amount of available text processing commands, and the ability to easily pipe them into each other to build complex text processing workflows. These commands can make it trivial to perform text and data analysis, convert data between different formats, filter lines, etc. This chapter will go over some of the most common and useful text processing commands the shell has to offer, and will demonstrate real-life workflows piping them together.</p><header>
<p>
This article is part of a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>
If you are interested in the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</header>
<h2 id="table-of-contents">Table of Contents</h2>
<!-- MarkdownTOC autolink="true" levels="2" autoanchor="true" -->
<ul>
<li><a href="#cat"><code>cat</code></a></li>
<li><a href="#head"><code>head</code></a></li>
<li><a href="#tail"><code>tail</code></a></li>
<li><a href="#wc"><code>wc</code></a></li>
<li><a href="#grep"><code>grep</code></a></li>
<li><a href="#cut"><code>cut</code></a></li>
<li><a href="#paste"><code>paste</code></a></li>
<li><a href="#sort"><code>sort</code></a></li>
<li><a href="#uniq"><code>uniq</code></a></li>
<li><a href="#awk"><code>awk</code></a></li>
<li><a href="#tr"><code>tr</code></a></li>
<li><a href="#fold"><code>fold</code></a></li>
<li><a href="#sed"><code>sed</code></a></li>
<li><a href="#real-life-examples">Real-life examples</a></li>
<li><a href="#going-further-for-loops-and-xargs">Going further: <code>for</code> loops and <code>xargs</code></a></li>
<li><a href="#summary">Summary</a></li>
<li><a href="#going-further">Going further</a></li>
</ul>
<!-- /MarkdownTOC -->
<h1 id="text-processing-in-the-shell">Text processing in the shell</h1>
<p>One of the things that makes the shell an invaluable tool is the amount
of available text processing commands, and the ability to easily pipe
them into each other to build complex text processing workflows. These
commands can make it trivial to perform text and data analysis, convert
data between different formats, filter lines, etc.</p>
<p>When working with text data, the philosophy is to break any complex
problem you have into a set of smaller ones, and to solve each of them
with a specialized tool.</p>
<blockquote>
<p>Make each program do one thing well.<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup></p>
</blockquote>
<p>The examples in that chapter might seem a little contrived at first, but
this is also by design. Each one of these tools was designed to solve one
small problem. They however become extremely powerful when combined.</p>
<p>We will go over some of the most common and useful text processing
commands the shell has to offer, and will demonstrate real-life
workflows piping them together. I suggest you take a look at the <code>man</code>
of these commands to see the full breadth of options at your disposal.</p>
<div class="Note">
<p>The example CSV (comma-separated values) file is available online.<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>
Feel free to download it yourself to test these commands.</p>
</div>
<p><a id="cat"></a></p>
<h2 id="cat"><code>cat</code></h2>
<p>As seen in the previous chapter, <code>cat</code> is used to concatenate a list of
one or more files and displays their content on screen.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>Documents/readme
Thanks<span class="w"> </span>again<span class="w"> </span><span class="k">for</span><span class="w"> </span>reading<span class="w"> </span>this<span class="w"> </span>book!
I<span class="w"> </span>hope<span class="w"> </span>you<span class="s1">'re following so far!</span>
<span class="s1">$ cat Documents/computers</span>
<span class="s1">Computers are not intelligent</span>
<span class="s1">They'</span>re<span class="w"> </span>just<span class="w"> </span>fast<span class="w"> </span>at<span class="w"> </span>making<span class="w"> </span>dumb<span class="w"> </span>things.
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>Documents/readme<span class="w"> </span>Documents/computers
Thanks<span class="w"> </span>again<span class="w"> </span><span class="k">for</span><span class="w"> </span>reading<span class="w"> </span>this<span class="w"> </span>book!
I<span class="w"> </span>hope<span class="w"> </span>you<span class="w"> </span>are<span class="w"> </span>following<span class="w"> </span>so<span class="w"> </span>far!
Computers<span class="w"> </span>are<span class="w"> </span>not<span class="w"> </span>intelligent
They<span class="err">'</span>re<span class="w"> </span>just<span class="w"> </span>fast<span class="w"> </span>at<span class="w"> </span>making<span class="w"> </span>dumb<span class="w"> </span>things.
</code></pre></div>
<p><a id="head"></a></p>
<h2 id="head"><code>head</code></h2>
<p><code>head</code> prints the first n lines in a file. It can be very useful to peek
into a file of unknown structure and format without burying your shell
under a wall of text.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">head</span><span class="w"> </span>-n<span class="w"> </span><span class="m">2</span><span class="w"> </span>metadata.csv
metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name
mysql.galera.wsrep_cluster_size,gauge,,node,,The<span class="w"> </span>current<span class="w"> </span>number<span class="w"> </span>of<span class="w"> </span>nodes<span class="w"> </span><span class="k">in</span><span class="w"> </span>the<span class="w"> </span>Galera<span class="w"> </span>cluster.,0,mysql,galera<span class="w"> </span>cluster<span class="w"> </span>size
</code></pre></div>
<p>If <code>-n</code> is unspecified, <code>head</code> will print the first 10 lines in its
argument file or input stream.</p>
<p><a id="tail"></a></p>
<h2 id="tail"><code>tail</code></h2>
<p><code>tail</code> is <code>head</code>’s counterpart. It prints the last n lines in a file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">1</span><span class="w"> </span>metadata.csv
mysql.performance.queries,gauge,,query,second,The<span class="w"> </span>rate<span class="w"> </span>of<span class="w"> </span>queries.,0,mysql,queries
</code></pre></div>
<p>If you want to print all lines in a file located after the nth line
(included), you can use the <code>-n +n</code> argument.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span>+42<span class="w"> </span>metadata.csv
mysql.replication.slaves_connected,gauge,,,,Number<span class="w"> </span>of<span class="w"> </span>slaves<span class="w"> </span>connected<span class="w"> </span>to<span class="w"> </span>a<span class="w"> </span>replication<span class="w"> </span>master.,0,mysql,slaves<span class="w"> </span>connected
mysql.performance.queries,gauge,,query,second,The<span class="w"> </span>rate<span class="w"> </span>of<span class="w"> </span>queries.,0,mysql,queries
</code></pre></div>
<p>Our file has 43 lines, so <code>tail -n +42</code> only prints the 42nd and 43rd
line in our file.</p>
<p>If <code>-n</code> is unspecified, <code>tail</code> will print the last 10 lines in its
argument file or input stream.</p>
<p><code>tail -f</code> or <code>tail --follow</code> displays the last lines in a file and
displays each new line as the file is being written to. It is very
useful to see real time activity that is written to a log file, for
example a web server log file, etc.</p>
<p><a id="wc"></a></p>
<h2 id="wc"><code>wc</code></h2>
<p><code>wc</code> (for <em>word count</em>) prints either the number of characters (when
using <code>-c</code>), words (when using <code>-w</code>) or lines (when using <code>-l</code>) in its
argument files or input stream.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">wc</span><span class="w"> </span>-l<span class="w"> </span>metadata.csv
<span class="m">43</span><span class="w"> </span>metadata.csv
$<span class="w"> </span><span class="nb">wc</span><span class="w"> </span>-w<span class="w"> </span>metadata.csv
<span class="m">405</span><span class="w"> </span>metadata.csv
$<span class="w"> </span><span class="nb">wc</span><span class="w"> </span>-c<span class="w"> </span>metadata.csv
<span class="m">5094</span><span class="w"> </span>metadata.csv
</code></pre></div>
<p>By default, <code>wc</code> prints all of the above.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">wc</span><span class="w"> </span>metadata.csv
<span class="m">43</span><span class="w"> </span><span class="m">405</span><span class="w"> </span><span class="m">5094</span><span class="w"> </span>metadata.csv
</code></pre></div>
<p>Only the count will be printed out if the text data is piped in or
redirected into <code>stdin</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>metadata.csv<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">wc</span>
<span class="m">43</span><span class="w"> </span><span class="m">405</span><span class="w"> </span><span class="m">5094</span>
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>metadata.csv<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">wc</span><span class="w"> </span>-l
<span class="m">43</span>
$<span class="w"> </span><span class="nb">wc</span><span class="w"> </span>-w<span class="w"> </span><span class="k"><</span><span class="w"> </span>metadata.csv
<span class="m">405</span>
</code></pre></div>
<p><a id="grep"></a></p>
<h2 id="grep"><code>grep</code></h2>
<p><code>grep</code> is the Swiss Army knife of line filtering. It allows you to
filter lines matching a given pattern.</p>
<p>For example, we can use <code>grep</code> to find all occurrences of the word
<em>mutex</em> in our <code>metadata.csv</code> file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>mutex<span class="w"> </span>metadata.csv
mysql.innodb.mutex_os_waits,gauge,,event,second,The<span class="w"> </span>rate<span class="w"> </span>of<span class="w"> </span>mutex<span class="w"> </span>OS<span class="w"> </span>waits.,0,mysql,mutex<span class="w"> </span>os<span class="w"> </span>waits
mysql.innodb.mutex_spin_rounds,gauge,,event,second,The<span class="w"> </span>rate<span class="w"> </span>of<span class="w"> </span>mutex<span class="w"> </span>spin<span class="w"> </span>rounds.,0,mysql,mutex<span class="w"> </span>spin<span class="w"> </span>rounds
mysql.innodb.mutex_spin_waits,gauge,,event,second,The<span class="w"> </span>rate<span class="w"> </span>of<span class="w"> </span>mutex<span class="w"> </span>spin<span class="w"> </span>waits.,0,mysql,mutex<span class="w"> </span>spin<span class="w"> </span>waits
</code></pre></div>
<p><code>grep</code> can either filter files passed as arguments, or a stream of text passed
to its <code>stdin</code>. We can thus chain multiple <code>grep</code> commands to further
filter our text. In the next example, we filter lines in our
<code>metadata.csv</code> file that contain both the <em>mutex</em> and <em>OS</em> words.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>mutex<span class="w"> </span>metadata.csv<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>OS
mysql.innodb.mutex_os_waits,gauge,,event,second,The<span class="w"> </span>rate<span class="w"> </span>of<span class="w"> </span>mutex<span class="w"> </span>OS<span class="w"> </span>waits.,0,mysql,mutex<span class="w"> </span>os<span class="w"> </span>waits
</code></pre></div>
<p>Let’s go over some of the options you can pass to grep and their
associated behavior.</p>
<p><code>grep -v</code> performs an invert matching: it filters the lines that do
<em>not</em> match the argument pattern.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-v<span class="w"> </span>gauge<span class="w"> </span>metadata.csv
metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name
</code></pre></div>
<p><code>grep -i</code> performs a case-insensitive matching. In the next example
<code>grep -i os</code> matches both <em>OS</em> and <em>os</em>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-i<span class="w"> </span>os<span class="w"> </span>metadata.csv
mysql.innodb.mutex_os_waits,gauge,,event,second,The<span class="w"> </span>rate<span class="w"> </span>of<span class="w"> </span>mutex<span class="w"> </span>OS<span class="w"> </span>waits.,0,mysql,mutex<span class="w"> </span>os<span class="w"> </span>waits
mysql.innodb.os_log_fsyncs,gauge,,write,second,The<span class="w"> </span>rate<span class="w"> </span>of<span class="w"> </span>fsync<span class="w"> </span>writes<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>log<span class="w"> </span>file.,0,mysql,log<span class="w"> </span>fsyncs
</code></pre></div>
<p><code>grep -l</code> only lists files containing a match.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-l<span class="w"> </span>mysql<span class="w"> </span>metadata.csv
metadata.csv
</code></pre></div>
<p><code>grep -c</code> counts the number of times a pattern was found.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-c<span class="w"> </span><span class="k">select</span><span class="w"> </span>metadata.csv
<span class="m">3</span>
</code></pre></div>
<p><code>grep -r</code> recursively searches files in the current working directory
and all subdirectories below it.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-r<span class="w"> </span>are<span class="w"> </span>~/Documents
/home/br/Documents/computers:Computers<span class="w"> </span>are<span class="w"> </span>not<span class="w"> </span>intelligent
/home/br/Documents/readme:I<span class="w"> </span>hope<span class="w"> </span>you<span class="w"> </span>are<span class="w"> </span>following<span class="w"> </span>so<span class="w"> </span>far!
</code></pre></div>
<p><code>grep -w</code> only matches whole words.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>follow<span class="w"> </span>~/Documents/readme
I<span class="w"> </span>hope<span class="w"> </span>you<span class="w"> </span>are<span class="w"> </span>following<span class="w"> </span>so<span class="w"> </span>far!
$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-w<span class="w"> </span>follow<span class="w"> </span>~/Documents/readme
$
</code></pre></div>
<p><a id="cut"></a></p>
<h2 id="cut"><code>cut</code></h2>
<p><code>cut</code> cuts out a portion of a file (or, as always, its input stream).
<code>cut</code> works by defining a field delimited (what separates two columns)
with the <code>-d</code> option, and what column(s) should be extracted, with the
<code>-f</code> option.</p>
<p>For example, the following command extracts the first column of the last
5 lines our CSV file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">5</span><span class="w"> </span>metadata.csv<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">cut</span><span class="w"> </span>-d<span class="w"> </span>,<span class="w"> </span>-f<span class="w"> </span><span class="m">1</span>
mysql.performance.user_time
mysql.replication.seconds_behind_master
mysql.replication.slave_running
mysql.replication.slaves_connected
mysql.performance.queries
</code></pre></div>
<p>As we are dealing with a CSV file, we can extract each column by cutting
over the <code>,</code> character, and extract the first column with <code>-f 1</code>.</p>
<p>We could also select both the first and second columns by using the
<code>-f 1,2</code> option.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">tail</span><span class="w"> </span>-n<span class="w"> </span><span class="m">5</span><span class="w"> </span>metadata.csv<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">cut</span><span class="w"> </span>-d<span class="w"> </span>,<span class="w"> </span>-f<span class="w"> </span><span class="m">1</span>,2
mysql.performance.user_time,gauge
mysql.replication.seconds_behind_master,gauge
mysql.replication.slave_running,gauge
mysql.replication.slaves_connected,gauge
mysql.performance.queries,gauge
</code></pre></div>
<p><a id="paste"></a></p>
<h2 id="paste"><code>paste</code></h2>
<p><code>paste</code> can merge together two different files into one multi-column
file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>ingredients
eggs
milk
butter
tomatoes
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>prices
<span class="m">1</span>$
<span class="m">1</span>.99$
<span class="m">1</span>.50$
<span class="m">2</span>$/kg
$<span class="w"> </span><span class="nb">paste</span><span class="w"> </span>ingredients<span class="w"> </span>prices
eggs<span class="w"> </span><span class="m">1</span>$
milk<span class="w"> </span><span class="m">1</span>.99$
butter<span class="w"> </span><span class="m">1</span>.50$
tomatoes<span class="w"> </span><span class="m">2</span>$/kg
</code></pre></div>
<p>By default, <code>paste</code> uses a tab delimiter, but you can change that using
the <code>-d</code> option.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">paste</span><span class="w"> </span>ingredients<span class="w"> </span>prices<span class="w"> </span>-d:
eggs:1$
milk:1.99$
butter:1.50$
tomatoes:2$/kg
</code></pre></div>
<p>Another common use of <code>paste</code> it to join all lines within a stream or a
file using a given delimiter, using a combination of the <code>-s</code> and <code>-d</code>
argument.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">paste</span><span class="w"> </span>-s<span class="w"> </span>-d,<span class="w"> </span>ingredients
eggs,milk,butter,tomatoes
</code></pre></div>
<p>If <code>-</code> is specified as an input file, <code>stdin</code> will be read instead.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>ingredients<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">paste</span><span class="w"> </span>-s<span class="w"> </span>-d,<span class="w"> </span>-
eggs,milk,butter,tomatoes
</code></pre></div>
<p><a id="sort"></a></p>
<h2 id="sort"><code>sort</code></h2>
<p><code>sort</code>, well, sorts argument files or input.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>ingredients
eggs
milk
butter
tomatoes
salt
$<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>ingredients
butter
eggs
milk
salt
tomatoes
</code></pre></div>
<p><code>sort -r</code> performs a reverse sort.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>-r<span class="w"> </span>ingredients
tomatoes
salt
milk
eggs
butter
</code></pre></div>
<p><code>sort -n</code> performs a numerical sort, by sorting fields by their
arithmetic value.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>numbers
<span class="m">0</span>
<span class="m">2</span>
<span class="m">1</span>
<span class="m">10</span>
<span class="m">3</span>
$<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>numbers
<span class="m">0</span>
<span class="m">1</span>
<span class="m">10</span>
<span class="m">2</span>
<span class="m">3</span>
$<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>-n<span class="w"> </span>numbers
<span class="m">0</span>
<span class="m">1</span>
<span class="m">2</span>
<span class="m">3</span>
<span class="m">10</span>
</code></pre></div>
<p><a id="uniq"></a></p>
<h2 id="uniq"><code>uniq</code></h2>
<p><code>uniq</code> detects or filters out adjacent identical lines in its argument
file or input stream.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>duplicates
and<span class="w"> </span>one
and<span class="w"> </span>one
and<span class="w"> </span>two
and<span class="w"> </span>one
and<span class="w"> </span>two
and<span class="w"> </span>one,<span class="w"> </span>two,<span class="w"> </span>three
$<span class="w"> </span><span class="nb">uniq</span><span class="w"> </span>duplicates
and<span class="w"> </span>one
and<span class="w"> </span>two
and<span class="w"> </span>one
and<span class="w"> </span>two
and<span class="w"> </span>one,<span class="w"> </span>two,<span class="w"> </span>three
</code></pre></div>
<p>As <code>uniq</code> only filters out <em>adjacent</em> identical lines, we can still see
more than one unique lines in its output. To filter out all identical
lines from our <code>duplicates</code> file, we need to sort its content first.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>duplicates<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">uniq</span>
and<span class="w"> </span>one
and<span class="w"> </span>one,<span class="w"> </span>two,<span class="w"> </span>three
and<span class="w"> </span>two
</code></pre></div>
<p><code>uniq -c</code> prepends all lines with its number of occurrences.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>duplicates<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">uniq</span><span class="w"> </span>-c
<span class="w"> </span><span class="m">3</span><span class="w"> </span>and<span class="w"> </span>one
<span class="w"> </span><span class="m">1</span><span class="w"> </span>and<span class="w"> </span>one,<span class="w"> </span>two,<span class="w"> </span>three
<span class="w"> </span><span class="m">2</span><span class="w"> </span>and<span class="w"> </span>two
</code></pre></div>
<p><code>uniq -u</code> only displays the unique lines within its input.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>duplicates<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">uniq</span><span class="w"> </span>-u
and<span class="w"> </span>one,<span class="w"> </span>two,<span class="w"> </span>three
</code></pre></div>
<div class="Note">
<p><code>uniq</code> is particularly useful used in conjunction with <code>sort</code>, as
<code>| sort | uniq</code> allows you to remove any duplicate line in a file or a
stream.</p>
</div>
<p><a id="awk"></a></p>
<h2 id="awk"><code>awk</code></h2>
<p><code>awk</code> is a little more than a text processing tool: it’s actually a
whole programming language of its own<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>. One thing <code>awk</code> is <em>really</em>
good at is splitting files into columns, and it especially shines when
these files contain a mix and match of spaces and tabs.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>-t<span class="w"> </span>multi-columns
John<span class="w"> </span>Smith<span class="w"> </span>Doctor^ITardis
Sarah-James<span class="w"> </span>Smith^I<span class="w"> </span>Companion^ILondon
Rose<span class="w"> </span>Tyler<span class="w"> </span>Companion^ILondon
</code></pre></div>
<div class="Note">
<p><code>cat -t</code> displays tabs as <code>^I</code>.</p>
</div>
<p>We can see that these columns are either separated by spaces or tabs,
and that they are not always separated by the same number of spaces.
<code>cut</code> would be of no use there, because it only works on a single
character delimiter. <code>awk</code> however, can easily make sense of that file.</p>
<p><code>awk '{ print $n }'</code> prints the nth column in the text.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>multi-columns<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $1 }'</span>
John
Sarah-James
Rose
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>multi-columns<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $3 }'</span>
Doctor
Companion
Companion
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>multi-columns<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $1,$2 }'</span>
John<span class="w"> </span>Smith
Sarah-James<span class="w"> </span>Smith
Rose<span class="w"> </span>Tyler
</code></pre></div>
<p>There is so much more we can do with <code>awk</code>, however, printing columns
probably accounts for 99% of my personal usage.</p>
<div class="Note">
<p><code>{ print $NF }</code> prints the last column in the line.</p>
</div>
<p><a id="tr"></a></p>
<h2 id="tr"><code>tr</code></h2>
<p><code>tr</code> stands for <em>translate</em>, and it replaces characters into others. It
either works on characters or character <em>classes</em>, such as lowercase,
printable, spaces, alphanumeric, etc.</p>
<p><code>tr <char1> <char2></code> translates all occurrences of <code><char1></code> from its
standard input into <code><char2></code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Computers are fast"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span>a<span class="w"> </span>A
computers<span class="w"> </span>Are<span class="w"> </span>fAst
</code></pre></div>
<p><code>tr</code> can also translate character classes by using the <code>[:class:]</code>
notation. The full list of available classes is described in the <code>tr</code>
man page, but we’ll demonstrate some of them here.</p>
<p><code>[:space:]</code> represent all types of spaces, from a simple space, to a tab
or a newline.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"computers are fast"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span><span class="s1">'[:space:]'</span><span class="w"> </span><span class="s1">','</span>
computers,are,fast,%
</code></pre></div>
<p>All spaces-like characters were translated into a comma. Note that the
<code>%</code> character at the end of the output represents the lack of a trailing
newline. Indeed, that newline was translated to a comma as well.</p>
<p><code>[:lower:]</code> represents all lowercase characters, and <code>[:upper:]</code>
represents all uppercase characters. Converting between cases is thus
made very easy.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"computers are fast"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span><span class="s1">'[:lower:]'</span><span class="w"> </span><span class="s1">'[:upper:]'</span>
COMPUTERS<span class="w"> </span>ARE<span class="w"> </span>FAST
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"COMPUTERS ARE FAST"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span><span class="s1">'[:upper:]'</span><span class="w"> </span><span class="s1">'[:lower:]'</span>
computers<span class="w"> </span>are<span class="w"> </span>fast
</code></pre></div>
<p><code>tr SET1 SET2</code> will transform any character in SET1 into the
characters in SET2. The following example replaces all vowels by
spaces.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"computers are fast"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span><span class="s1">'[aeiouy]'</span><span class="w"> </span><span class="s1">' '</span>
c<span class="w"> </span>mp<span class="w"> </span>t<span class="w"> </span>rs<span class="w"> </span>r<span class="w"> </span>f<span class="w"> </span>st
</code></pre></div>
<p><code>tr -c SET1 SET2</code> does the opposite: it transforms any character <em>not</em> in SET1 into the
characters in SET2. The following example replaces all non vowels by
spaces.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"computers are fast"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span>-c<span class="w"> </span><span class="s1">'[aeiouy]'</span><span class="w"> </span><span class="s1">' '</span>
<span class="w"> </span>o<span class="w"> </span>u<span class="w"> </span>e<span class="w"> </span>a<span class="w"> </span>e<span class="w"> </span>a
</code></pre></div>
<p><code>tr -d</code> deletes the matched characters, instead of replacing them. It’s
the equivalent of <code>tr <char> ''</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Computers Are Fast"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span>-d<span class="w"> </span><span class="s1">'[:lower:]'</span>
C<span class="w"> </span>A<span class="w"> </span>F
</code></pre></div>
<p><code>tr</code> can also replace character ranges, for example all letters between
<em>a</em> and <em>e</em>, or all numbers between 1 and 8, by using the notation
<code>s-e</code>, where <code>s</code> is the start character and <code>e</code> is the end one.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"computers are fast"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span><span class="s1">'a-e'</span><span class="w"> </span><span class="s1">'x'</span>
xomputxrs<span class="w"> </span>xrx<span class="w"> </span>fxst
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"5uch l337 5p34k"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span><span class="s1">'1-4'</span><span class="w"> </span><span class="s1">'x'</span>
5uch<span class="w"> </span>lxx7<span class="w"> </span>5pxxk
</code></pre></div>
<p><code>tr -s string1</code> compresses any multiple occurrences of the characters in <code>string1</code> into a single one. One of the most useful uses of <code>tr -s</code> is to replace multiple consecutive spaces by a single one.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Computers are fast"</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">tr</span><span class="w"> </span>-s<span class="w"> </span><span class="s1">' '</span>
Computers<span class="w"> </span>are<span class="w"> </span>fast
</code></pre></div>
<p><a id="fold"></a></p>
<h2 id="fold"><code>fold</code></h2>
<p><code>fold</code> wraps each input line to fit in a specified width. It can be
useful to make sure an argument text fits in a small display size for
example. <code>fold -w n</code> folds the lines at <code>n</code> characters.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>~/Documents/readme<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">fold</span><span class="w"> </span>-w<span class="w"> </span><span class="m">16</span>
Thanks<span class="w"> </span>again<span class="w"> </span><span class="k">for</span>
<span class="w"> </span>reading<span class="w"> </span>this<span class="w"> </span>bo
ok!
I<span class="w"> </span>hope<span class="w"> </span>you<span class="err">'</span>re<span class="w"> </span>fo
llowing<span class="w"> </span>so<span class="w"> </span>far!
</code></pre></div>
<p><code>fold -s</code> will only break lines on a space character, and can be
combined with <code>-w</code> to fold up to a given number of characters.</p>
<div class="highlight"><pre><span></span><code><span class="nv">Thanks</span><span class="w"> </span><span class="nv">again</span>
<span class="k">for</span><span class="w"> </span><span class="nv">reading</span>
<span class="nv">this</span><span class="w"> </span><span class="nv">book</span><span class="o">!</span>
<span class="nv">I</span><span class="w"> </span><span class="nv">hope</span><span class="w"> </span><span class="nv">you</span><span class="err">'re</span>
<span class="err">following so</span>
<span class="err">far!</span>
</code></pre></div>
<p><a id="sed"></a></p>
<h2 id="sed"><code>sed</code></h2>
<p><code>sed</code> is a non-interactive stream editor, used to perform text
transformation on its input stream, on a line-per-line basis. It can
take its output from a file our its <code>stdin</code> and will output its result
either in a file or its <code>stdout</code>.</p>
<p>It works by taking one or many optional <em>addresses</em>, a <em>function</em> and
<em>parameters</em>. A <code>sed</code> command thus looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span><span class="n">address[,address</span><span class="o">]</span><span class="err">]</span><span class="k">function</span><span class="o">[</span><span class="n">arguments</span><span class="o">]</span>
</code></pre></div>
<p>While <code>sed</code> can perform many functions, we will cover only substitution,
as it is probably <code>sed</code>’s most common use.</p>
<h3 id="substituting-text">Substituting text</h3>
<p>A <code>sed</code> substitution command looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="n">s</span><span class="o">/</span><span class="n">PATTERN</span><span class="o">/</span><span class="n">REPLACEMENT</span><span class="o">/[</span><span class="n">options</span><span class="o">]</span>
</code></pre></div>
<p><strong>Example</strong>: replacing the first instance of a word for each line in a
file</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>hello
hello<span class="w"> </span>hello
hello<span class="w"> </span>world!
hi
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>hello<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s/hello/Hey I just met you/'</span>
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>hello
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>world
hi
</code></pre></div>
<p>We can see that only the first occurrence of <code>hello</code> was replaced in the
first line. To replace <em>all</em> occurrences of <code>hello</code> in each line, we can
use the <code>g</code> (for <em>global</em>) option.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>hello<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s/hello/Hey I just met you/g'</span>
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>world
hi
</code></pre></div>
<p><code>sed</code> allows you to specify any other separator than <code>/</code>, which is
especially useful to keep the command readable if the search of
replacement pattern contains forward slashes.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>hello<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s@hello@Hey I just met you@g'</span>
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>world
hi
</code></pre></div>
<p>By specifying an address, we can tell sed on which line or line-range to
actually perform the substitution.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>hello<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'1s/hello/Hey I just met you/g'</span>
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>hello
hello<span class="w"> </span>world
hi
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>hello<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'2s/hello/Hey I just met you/g'</span>
hello<span class="w"> </span>hello
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>world
hi
</code></pre></div>
<p>The address <code>1</code> tells <code>sed</code> to only replace <code>hello</code> by
<code>Hey I just met you</code> on line 1. We can specify an address range with the
notation <code><start>,<end></code> where <code><end></code> can either be a line number or
<code>$</code>, meaning the last line in the file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>hello<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'1,2s/hello/Hey I just met you/g'</span>
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>world
hi
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>hello<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'2,3s/hello/Hey I just met you/g'</span>
hello<span class="w"> </span>hello
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>world
hi
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>hello<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'2,$s/hello/Hey I just met you/g'</span>
hello<span class="w"> </span>hello
Hey<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you<span class="w"> </span>world
hi
</code></pre></div>
<p>By default, <code>sed</code> displays its result in its <code>stdout</code>, but it can also
edit the initial file in-place, with the use of the <code>-i</code> option.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">''</span><span class="w"> </span><span class="s1">'s/hello/Bonjour/'</span><span class="w"> </span>sed-data
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>sed-data
Bonjour<span class="w"> </span>hello
Bonjour<span class="w"> </span>world
hi
</code></pre></div>
<div class="Note">
<p>On Linux, only <code>-i</code> needs to be specified. However, due to the fact that
<code>sed</code>’s behavior on macOS is slightly different, the <code>''</code> needs to be
added right after <code>-i</code>.</p>
</div>
<p><a id="real-life-examples"></a></p>
<h2 id="real-life-examples">Real-life examples</h2>
<h3 id="filtering-a-csv-using-grep-and-awk">Filtering a CSV using <code>grep</code> and <code>awk</code></h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-w<span class="w"> </span>gauge<span class="w"> </span>metadata.csv<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">awk</span><span class="w"> </span>-F,<span class="w"> </span><span class="s1">'{ if ($4 == "query") { print $1, "per", $5 } }'</span>
mysql.performance.com_delete<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.com_delete_multi<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.com_insert<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.com_insert_select<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.com_replace_select<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.com_select<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.com_update<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.com_update_multi<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.questions<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.slow_queries<span class="w"> </span>per<span class="w"> </span>second
mysql.performance.queries<span class="w"> </span>per<span class="w"> </span>second
</code></pre></div>
<p>This example filters the lines containing the word <code>gauge</code> in our
<code>metadata.csv</code> file using <code>grep</code>, then the filters the lines with the
string <code>query</code> as their 4th column, and displays the metric name (1st
column) with its associated <code>per_unit_name</code> value (5th column).</p>
<h3 id="printing-the-ipv4-address-associated-with-a-network-interface">Printing the IPv4 address associated with a network interface</h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ifconfig</span><span class="w"> </span>en0<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>inet<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-v<span class="w"> </span>inet6<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $2 }'</span>
<span class="m">192</span>.168.0.38
</code></pre></div>
<p><code>ifconfig <interface name></code> prints details associated with the argument
network interface name. For example:</p>
<div class="highlight"><pre><span></span><code>en0:<span class="w"> </span><span class="nv">flags</span><span class="o">=</span><span class="m">8863</span><span class="k"><</span>UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST><span class="w"> </span>mtu<span class="w"> </span><span class="m">1500</span>
<span class="w"> </span>ether<span class="w"> </span><span class="m">19</span>:64:92:de:20:ba
<span class="w"> </span>inet6<span class="w"> </span>fe80::8a3:a1cb:56ae:7c7c%en0<span class="w"> </span>prefixlen<span class="w"> </span><span class="m">64</span><span class="w"> </span>secured<span class="w"> </span>scopeid<span class="w"> </span>0x7
<span class="w"> </span>inet<span class="w"> </span><span class="m">192</span>.168.0.38<span class="w"> </span>netmask<span class="w"> </span>0xffffff00<span class="w"> </span>broadcast<span class="w"> </span><span class="m">192</span>.168.0.255
<span class="w"> </span>nd6<span class="w"> </span><span class="nv">options</span><span class="o">=</span><span class="m">201</span><span class="k"><</span>PERFORMNUD,DAD>
<span class="w"> </span>media:<span class="w"> </span>autoselect
<span class="w"> </span>status:<span class="w"> </span>active
</code></pre></div>
<p>We then <code>grep</code> for <code>inet</code>, which will match 2 lines.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ifconfig</span><span class="w"> </span>en0<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>inet
<span class="w"> </span>inet6<span class="w"> </span>fe80::8a3:a1cb:56ae:7c7c%en0<span class="w"> </span>prefixlen<span class="w"> </span><span class="m">64</span><span class="w"> </span>secured<span class="w"> </span>scopeid<span class="w"> </span>0x7
<span class="w"> </span>inet<span class="w"> </span><span class="m">192</span>.168.0.38<span class="w"> </span>netmask<span class="w"> </span>0xffffff00<span class="w"> </span>broadcast<span class="w"> </span><span class="m">192</span>.168.0.255
</code></pre></div>
<p>We then exclude the line with <code>ipv6</code> by using a <code>grep -v</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ifconfig</span><span class="w"> </span>en0<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>inet<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-v<span class="w"> </span>inet6
inet<span class="w"> </span><span class="m">192</span>.168.0.38<span class="w"> </span>netmask<span class="w"> </span>0xffffff00<span class="w"> </span>broadcast<span class="w"> </span><span class="m">192</span>.168.0.255
</code></pre></div>
<p>We finally use <code>awk</code> to get the 2nd column in that line: the IPv4
address associated with our <code>en0</code> network interface.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ifconfig</span><span class="w"> </span>en0<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>inet<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-v<span class="w"> </span>inet6<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $2 }'</span>
<span class="m">192</span>.168.0.38
</code></pre></div>
<div class="Note">
<p>It has been <a href="https://www.reddit.com/r/bash/comments/finbd2/beginner_friendly_introduction_to_the_shell_text/fki8523/?context=3">suggested</a> to me that <code>grep inet | grep -v inet6</code> could be replaced by the following future-proof <code>awk</code> command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ifconfig</span><span class="w"> </span>en0<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">' $1 == "inet" { print $2 }'</span>
<span class="m">192</span>.168.0.38
</code></pre></div>
<p>It is shorter and specifically targets IPv4 using the <code>$1 == "inet"</code> condition.</p>
</div>
<h3 id="extracting-a-value-from-a-config-file">Extracting a value from a config file</h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s1">'editor ='</span><span class="w"> </span>~/.gitconfig<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">cut</span><span class="w"> </span>-d<span class="w"> </span><span class="o">=</span><span class="w"> </span>-f2<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s/ //g'</span>
/usr/bin/vim
</code></pre></div>
<p>We look for the <code>editor =</code> value in the current user’s git configuration
file, then cut over the <code>=</code> sign, get the second column and remove any
space around that column.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s1">'editor ='</span><span class="w"> </span>~/.gitconfig
<span class="w"> </span><span class="nv">editor</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>/usr/bin/vim
$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s1">'editor ='</span><span class="w"> </span>~/.gitconfig<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">cut</span><span class="w"> </span>-d<span class="s1">'='</span><span class="w"> </span>-f2
<span class="w"> </span>/usr/bin/vim
$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s1">'editor ='</span><span class="w"> </span>~/.gitconfig<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">cut</span><span class="w"> </span>-d<span class="s1">'='</span><span class="w"> </span>-f2<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s/ //'</span>
/usr/bin/vim
</code></pre></div>
<h3 id="extracting-ip-addresses-from-a-log-file">Extracting IP addresses from a log file</h3>
<p>The following real life example looks for the message
<code>Too many connections from</code> in a database log file (which is followed by
an IP address) and displays the 10 biggest offenders.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s1">'Too many connections from'</span><span class="w"> </span>db.log<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $12 }'</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s@/@@'</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">sort</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">uniq</span><span class="w"> </span>-c<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>-rn<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">head</span><span class="w"> </span>-n<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $2 }'</span>
<span class="w"> </span><span class="m">10</span>.11.112.108
<span class="w"> </span><span class="m">10</span>.11.111.70
<span class="w"> </span><span class="m">10</span>.11.97.57
<span class="w"> </span><span class="m">10</span>.11.109.72
<span class="w"> </span><span class="m">10</span>.11.116.156
<span class="w"> </span><span class="m">10</span>.11.100.221
<span class="w"> </span><span class="m">10</span>.11.96.242
<span class="w"> </span><span class="m">10</span>.11.81.68
<span class="w"> </span><span class="m">10</span>.11.99.112
<span class="w"> </span><span class="m">10</span>.11.107.120
</code></pre></div>
<p>Let’s break down what this pipeline of command does. First, let’s look
at what a log line looks like.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s2">"Too many connections from"</span><span class="w"> </span>db.log<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">head</span><span class="w"> </span>-n<span class="w"> </span><span class="m">1</span>
<span class="m">2020</span>-01-01<span class="w"> </span><span class="m">08</span>:02:37,617<span class="w"> </span><span class="o">[</span>myid:1<span class="o">]</span><span class="w"> </span>-<span class="w"> </span>WARN<span class="w"> </span><span class="o">[</span>NIOServerCxn.Factory:1.2.3.4/1.2.3.4:2181:NIOServerCnxnFactory@193<span class="o">]</span><span class="w"> </span>-<span class="w"> </span>Too<span class="w"> </span>many<span class="w"> </span>connections<span class="w"> </span>from<span class="w"> </span>/10.11.112.108<span class="w"> </span>-<span class="w"> </span>max<span class="w"> </span>is<span class="w"> </span><span class="m">60</span>
</code></pre></div>
<p><code>awk '{ print $12 }'</code> then extracts the IP from the line.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s2">"Too many connections from"</span><span class="w"> </span>db.log<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $12 }'</span>
/10.11.112.108
...
</code></pre></div>
<p><code>sed 's@/@@'</code> removes the trailing slash from the IPs.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s2">"Too many connections from"</span><span class="w"> </span>db.log<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $12 }'</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s@/@@'</span>
<span class="m">10</span>.11.112.108
...
</code></pre></div>
<div class="Note">
<p>As we have previously seen, we can use whatever separator we want for
<code>sed</code>. While <code>/</code> is commonly used as a separator, we are currently
replacing that very character, which would make the substitution
expression sightly less readable.</p>
<div class="highlight"><pre><span></span><code><span class="nb">sed</span><span class="w"> </span><span class="s1">'s/\///'</span>
</code></pre></div>
</div>
<p><code>sort | uniq -c</code> sorts the IPs lexicographically, and then removed
duplicates while prefixing IPs by their associated number of
occurrences.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s1">'Too many connections from'</span><span class="w"> </span>db.log<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $12 }'</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s@/@@'</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">sort</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">uniq</span><span class="w"> </span>-c
<span class="w"> </span><span class="m">1379</span><span class="w"> </span><span class="m">10</span>.11.100.221
<span class="w"> </span><span class="m">1213</span><span class="w"> </span><span class="m">10</span>.11.103.168
<span class="w"> </span><span class="m">1138</span><span class="w"> </span><span class="m">10</span>.11.105.177
<span class="w"> </span><span class="m">946</span><span class="w"> </span><span class="m">10</span>.11.106.213
<span class="w"> </span><span class="m">1211</span><span class="w"> </span><span class="m">10</span>.11.106.4
<span class="w"> </span><span class="m">1326</span><span class="w"> </span><span class="m">10</span>.11.107.120
<span class="w"> </span>...
</code></pre></div>
<p><code>sort -rn | head -n 10</code> sorts the lines by the number of occurrences,
numerically and in the reversed order, which displays the biggest
offenders first, 10 of which are displayed. The final <code>awk { print $2 }</code>
extracts the IPs themselves.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span><span class="s1">'Too many connections from'</span><span class="w"> </span>db.log<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $12 }'</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">sed</span><span class="w"> </span><span class="s1">'s@/@@'</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">sort</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">uniq</span><span class="w"> </span>-c<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>-rn<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">head</span><span class="w"> </span>-n<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">awk</span><span class="w"> </span><span class="s1">'{ print $2 }'</span>
<span class="w"> </span><span class="m">10</span>.11.112.108
<span class="w"> </span><span class="m">10</span>.11.111.70
<span class="w"> </span><span class="m">10</span>.11.97.57
<span class="w"> </span><span class="m">10</span>.11.109.72
<span class="w"> </span><span class="m">10</span>.11.116.156
<span class="w"> </span><span class="m">10</span>.11.100.221
<span class="w"> </span><span class="m">10</span>.11.96.242
<span class="w"> </span><span class="m">10</span>.11.81.68
<span class="w"> </span><span class="m">10</span>.11.99.112
<span class="w"> </span><span class="m">10</span>.11.107.120
</code></pre></div>
<h3 id="renaming-a-function-in-a-source-file">Renaming a function in a source file</h3>
<p>Let’s imagine that we are working a code project, and we would like to
rename rename a poorly named function (or class, variable, etc) in a
code file. We can do this by using <code>sed -i</code>, which performs an in-place
replacement in a file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>izk/utils.py
def<span class="w"> </span>bool_from_str<span class="o">(</span>s<span class="o">)</span>:
<span class="w"> </span><span class="k">if</span><span class="w"> </span>s.isdigit<span class="o">()</span>:
<span class="w"> </span><span class="k">return</span><span class="w"> </span>int<span class="o">(</span>s<span class="o">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span>s.lower<span class="o">()</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">[</span><span class="s1">'yes'</span>,<span class="w"> </span><span class="s1">'true'</span>,<span class="w"> </span><span class="s1">'y'</span><span class="o">]</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/def bool_from_str/def is_affirmative/'</span><span class="w"> </span>izk/utils.py
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>izk/utils.py
def<span class="w"> </span>is_affirmative<span class="o">(</span>s<span class="o">)</span>:
<span class="w"> </span><span class="k">if</span><span class="w"> </span>s.isdigit<span class="o">()</span>:
<span class="w"> </span><span class="k">return</span><span class="w"> </span>int<span class="o">(</span>s<span class="o">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span>s.lower<span class="o">()</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">[</span><span class="s1">'yes'</span>,<span class="w"> </span><span class="s1">'true'</span>,<span class="w"> </span><span class="s1">'y'</span><span class="o">]</span>
</code></pre></div>
<div class="Note">
<p>Use <code>sed -i ''</code> instead of <code>sed -i</code> on macOs, as the <code>sed</code> version
behaves slightly differently.</p>
</div>
<p>We’ve however only renamed this function in the file it was defined in.
Any other file we import <code>bool_from_str</code> will now be broken, as this
function is not defined anymore. We’d need a way to rename
<code>bool_from_str</code> everywhere it is found in our project. We can achieve
just that by using <code>grep</code>, <code>sed</code>, and either <code>for</code> loops or <code>xargs</code>.</p>
<p><a id="going-further-for-loops-and-xargs"></a></p>
<h2 id="going-further-for-loops-and-xargs">Going further: <code>for</code> loops and <code>xargs</code></h2>
<p>To replace all occurrences of <code>bool_from_str</code> in our project, we first
need to recursively find them using <code>grep -r</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-r<span class="w"> </span>bool_from_str<span class="w"> </span>.
./tests/test_utils.py:from<span class="w"> </span>izk.utils<span class="w"> </span>import<span class="w"> </span>bool_from_str
./tests/test_utils.py:def<span class="w"> </span>test_bool_from_str<span class="o">(</span>s,<span class="w"> </span>expected<span class="o">)</span>:
./tests/test_utils.py:<span class="w"> </span>assert<span class="w"> </span>bool_from_str<span class="o">(</span>s<span class="o">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span>expected
./izk/utils.py:def<span class="w"> </span>bool_from_str<span class="o">(</span>s<span class="o">)</span>:
./izk/prompt.py:from<span class="w"> </span>.utils<span class="w"> </span>import<span class="w"> </span>bool_from_str
./izk/prompt.py:<span class="w"> </span><span class="nv">default</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>bool_from_str<span class="o">(</span>os.environ<span class="o">[</span>envvar<span class="o">])</span>
</code></pre></div>
<p>As we are only interested in the matching files, we also need to use the
<code>-l/--files-with-matches</code> option:</p>
<div class="highlight"><pre><span></span><code><span class="o">-</span><span class="nv">l</span>,<span class="w"> </span><span class="o">--</span><span class="nv">files</span><span class="o">-</span><span class="nv">with</span><span class="o">-</span><span class="nv">matches</span>
<span class="w"> </span><span class="nv">Only</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">names</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">files</span><span class="w"> </span><span class="nv">containing</span><span class="w"> </span><span class="nv">selected</span><span class="w"> </span><span class="nv">lines</span><span class="w"> </span><span class="nv">are</span><span class="w"> </span><span class="nv">written</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">standard</span><span class="w"> </span><span class="nv">out</span><span class="o">-</span>
<span class="w"> </span><span class="nv">put</span>.<span class="w"> </span><span class="nv">grep</span><span class="w"> </span><span class="nv">will</span><span class="w"> </span><span class="nv">only</span><span class="w"> </span><span class="nv">search</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">file</span><span class="w"> </span><span class="k">until</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">match</span><span class="w"> </span><span class="nv">has</span><span class="w"> </span><span class="nv">been</span><span class="w"> </span><span class="nv">found</span>,<span class="w"> </span><span class="nv">making</span>
<span class="w"> </span><span class="nv">searches</span><span class="w"> </span><span class="nv">potentially</span><span class="w"> </span><span class="nv">less</span><span class="w"> </span><span class="nv">expensive</span>.<span class="w"> </span><span class="nv">Pathnames</span><span class="w"> </span><span class="nv">are</span><span class="w"> </span><span class="nv">listed</span><span class="w"> </span><span class="nv">once</span><span class="w"> </span><span class="nv">per</span><span class="w"> </span><span class="nv">file</span>
<span class="w"> </span><span class="nv">searched</span>.<span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">standard</span><span class="w"> </span><span class="nv">input</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">searched</span>,<span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">string</span><span class="w"> </span>``<span class="ss">(</span><span class="nv">standard</span><span class="w"> </span><span class="nv">input</span><span class="ss">)</span><span class="s1">''</span>
<span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">written</span>.
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-r<span class="w"> </span>--files-with-matches<span class="w"> </span>bool_from_str<span class="w"> </span>.
./tests/test_utils.py
./izk/utils.py
./izk/prompt.py
</code></pre></div>
<p>We can then use the <code>xargs</code> command to perform an action on each line in
the output (each file containing the <code>bool_from_str</code> string).</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-r<span class="w"> </span>--files-with-matches<span class="w"> </span>bool_from_str<span class="w"> </span>.<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">xargs</span><span class="w"> </span>-n<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span>
</code></pre></div>
<p><code>-n 1</code> tells <code>xargs</code> that each line in the output should cause a
separate <code>sed</code> command to be executed.</p>
<p>The following commands are then executed:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span><span class="w"> </span>./tests/test_utils.py
$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span><span class="w"> </span>./izk/utils.py
$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span><span class="w"> </span>./izk/prompt.py
</code></pre></div>
<p>If the command you call with <code>xargs</code> (<code>sed</code>, in our case) support
multiple arguments, you can (and shoud, as a single command will execute faster) drop the <code>-n 1</code> argument and run</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-r<span class="w"> </span>--files-with-matches<span class="w"> </span>bool_from_str<span class="w"> </span>.<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">xargs</span><span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span>
</code></pre></div>
<p>which will then execute</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span><span class="w"> </span>./tests/test_utils.py<span class="w"> </span>./izk/utils.py<span class="w"> </span>./izk/prompt.py
</code></pre></div>
<div class="Note">
<p>We can see that <code>sed</code> can take multiple arguments by looking at its
synopsis, in its <code>man</code> page.</p>
<div class="highlight"><pre><span></span><code>SYNOPSIS
sed [-Ealn] command [file ...]
sed [-Ealn] [-e command] [-f command_file] [-i extension] [file ...]
</code></pre></div>
<p>Indeed, as we’ve seen in the previous chapter, <code>file ...</code> means that
multiple arguments representing file names are accepted.</p>
</div>
<p>We can see that all <code>bool_from_str</code> occurrences have been replaced.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">grep</span><span class="w"> </span>-r<span class="w"> </span>is_affirmative<span class="w"> </span>.
./tests/test_utils.py:from<span class="w"> </span>izk.utils<span class="w"> </span>import<span class="w"> </span>is_affirmative
./tests/test_utils.py:def<span class="w"> </span>test_is_affirmative<span class="o">(</span>s,<span class="w"> </span>expected<span class="o">)</span>:
./tests/test_utils.py:<span class="w"> </span>assert<span class="w"> </span>is_affirmative<span class="o">(</span>s<span class="o">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span>expected
./izk/utils.py:def<span class="w"> </span>is_affirmative<span class="o">(</span>s<span class="o">)</span>:
./izk/prompt.py:from<span class="w"> </span>.utils<span class="w"> </span>import<span class="w"> </span>is_affirmative
./izk/prompt.py:<span class="w"> </span><span class="nv">default</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>is_affirmative<span class="o">(</span>os.environ<span class="o">[</span>envvar<span class="o">])</span>
</code></pre></div>
<p>As it is often the case, there are multiple ways of achieving the same
result. Instead of using <code>xargs</code>, we could have used <code>for</code> lops, which
allow you to iterate over a list of lines and perform an action on each
element. These <code>for</code> loops have the following syntax:</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span>item<span class="w"> </span><span class="k">in</span><span class="w"> </span>list<span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nb">command</span><span class="w"> </span><span class="nv">$item</span>
<span class="k">done</span>
</code></pre></div>
<p>By wrapping our <code>grep</code> command by <code>$()</code>, it will cause the shell to
execute the it in a <em>subshell</em>, which result will then be iterated on by
the <code>for</code> loop.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="k">for</span><span class="w"> </span>file<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="k">$(</span><span class="nb">grep</span><span class="w"> </span>-r<span class="w"> </span>--files-with-matches<span class="w"> </span>bool_from_str<span class="w"> </span>.<span class="k">)</span><span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span><span class="w"> </span><span class="nv">$file</span>
<span class="k">done</span>
</code></pre></div>
<p>which will execute</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span><span class="w"> </span>./tests/test_utils.py
$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span><span class="w"> </span>./izk/utils.py
$<span class="w"> </span><span class="nb">sed</span><span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/bool_from_str/is_affirmative/'</span><span class="w"> </span>./izk/prompt.py
</code></pre></div>
<p>I tend to find the for loop syntax clearer than <code>xargs</code>’s. <code>xargs</code> can
however execute the commands in parallel using its <code>-P n</code> options, where
<code>n</code> is the maximum number of parallel commands to be executed at a time,
which can be a performance win if your command takes time to run.</p>
<p><a id="summary"></a></p>
<h2 id="summary">Summary</h2>
<p>All these tools open up a world of possibilities, as they allow you to
extract data and transform its format, to make it possible to build
entire workflows of commands that were possibly never intended to work
together. Each of these commands accomplishes has a relatively small
function (<code>sort</code> sorts, <code>cat</code> concatenates, <code>grep</code> filters, <code>sed</code> edits,
<code>cut</code> cuts, etc).</p>
<p>Any given task involving text, can then be reduced to a pipeline of
smaller tasks, each of them performing a simple action and piping its
output into the next task.</p>
<p>For example, if we wanted to know how many unique IPs could be found in
a log file, and that these IPs always appeared at the same column, we
could:</p>
<ul>
<li><code>grep</code> lines on a pattern specific to lines containing an IP address</li>
<li>locate the column the IPs appear, and extract them with <code>awk</code></li>
<li>sort the list of IPs with <code>sort</code></li>
<li>compute the list of <em>unique</em> IPs with <code>uniq</code></li>
<li>count the number of lines (aka, of unique IPs) with <code>wc -l</code></li>
</ul>
<p>As there is a plethora of text processing tools, either available by
default or installable, there is bound to be many ways to solve any
given task.</p>
<p>The examples in this article were contrived, but I suggest you read the
amazing article “Command-line Tools can be 235x Faster than your Hadoop
Cluster”<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup> to get a sense of how useful and powerful these text
processing commands really are, and what real-life problems they can
solve.</p>
<p><a id="going-further"></a></p>
<h2 id="going-further">Going further</h2>
<p><strong>2.1</strong>: Count the number of files and directories located in your home
directory.</p>
<p><strong>2.2</strong>: Display the content of a file in all caps.</p>
<p><strong>2.3</strong>: Count how many times each word was found in a file.</p>
<p><strong>2.4</strong>: Count the number of vowels present in a file. Display the
result from the most common to the least.</p>
<footer>
<p>
<em>Essential Tools and Practices for the Aspiring Software Developer</em> is a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>The book will help you set up a productive development environment and get acquainted with tools and practices that, along with your programming languages of choice, will go a long way in helping you grow as a software developer.
It will cover subjects such as mastering the terminal, configuring and getting productive in a shell, the basics of code versioning with <code>git</code>, SQL basics, tools such as <code>Make</code>, <code>jq</code> and regular expressions, networking basics as well as software engineering and collaboration best practices.
</p>
<p>
If you are interested in the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</footer>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p><a href="https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html">https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html</a> <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p><a href="https://raw.githubusercontent.com/DataDog/integrations-core/master/mysql/metadata.csv">https://raw.githubusercontent.com/DataDog/integrations-core/master/mysql/metadata.csv</a> <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p><a href="https://www.gnu.org/software/gawk/manual/gawk.html">https://www.gnu.org/software/gawk/manual/gawk.html</a> <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p><a href="https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html">https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html</a> <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>Discovering the terminal2020-03-05T00:00:00+01:002020-03-05T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2020-03-05:/discovering-the-terminal<p>This chapter will help you take your first steps in the terminal, this almost mythical (The Matrix, anyone?) application we oftentimes imagine hackers type in really fast. We will see how it works and how it can empower you, increase your productivity and give you insights about how your computer works.</p><header>
<p>
This article is part of a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>
If you are interested in following the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</header>
<h2 id="table-of-contents">Table of Contents</h2>
<!-- MarkdownTOC autolink="true" levels="2" autoanchor="true" -->
<ul>
<li><a href="#what-is-a-terminal">What is a terminal?</a></li>
<li><a href="#your-first-steps">Your first steps</a></li>
<li><a href="#managing-files">Managing files</a></li>
<li><a href="#learning-new-options">Learning new options</a></li>
<li><a href="#command-inputoutput-streams">Command Input/Output streams</a></li>
<li><a href="#composing-commands">Composing commands</a></li>
<li><a href="#escaping-from-bad-situations">Escaping from bad situations</a></li>
<li><a href="#summary">Summary</a></li>
<li><a href="#going-further">Going further</a></li>
</ul>
<!-- /MarkdownTOC -->
<h1 id="discovering-the-terminal">Discovering the terminal</h1>
<p>When people picture a programmer, it’s not uncommon for them to imagine
someone sitting in front of a computer screen displaying undecipherable
streams of text going by really fast, like in The Matrix. Let’s set the
record straight. This is not true, at least for the most part. The
Matrix however got some things right. A programmer works with <em>code</em>,
which, as its name indicates, has to be learned before it can be
understood. Anyone not versed in the trade of reading and writing code
would only see gibberish. Another thing these movies usually get right
is the fact that a programmer types commands in a <em>terminal</em>.</p>
<p><a id="what-is-a-terminal"></a></p>
<h2 id="what-is-a-terminal">What is a terminal?</h2>
<p>Most of the applications people use everyday have a <span class="gls"
key="gui">Graphical User Interface (GUI)</span>. Think about Photoshop,
Firefox, or your smartphone apps. These application have immense
capabilities, but the user is mostly bound by the features implemented
in them in the first place. What if you suddenly wanted to have a new
feature in Photoshop that just wasn’t available? You would possibly end
up either waiting for the newest version to be released, or have to
install another application altogether.</p>
<p>One of the most important tools in a programmer toolbox is of a
different kind though. It’s called the <em><span class="gls"
key="terminal">terminal</span></em>, which is a <em><span class="gls"
key="cli">command-line</span> application</em>. That is to say that you
enter a command, your computer executes that command, and displays the
output in the terminal.</p>
<p>In other words, this is an applications in which you give your computer
orders. If you know how to ask, your computer will be happy to comply.
However, if you order it to do something stupid, it will obey.</p>
<blockquote>
<p>— You: “Computer, create that folder.”</p>
<p>— Computer: “Sure.”</p>
<p>— You: “Now put all the files on my Desktop in that new folder.”</p>
<p>— Computer: “No problem.”</p>
<p>— You: “Now delete that folder forever with everything inside.”</p>
<p>— Computer: “Done.”</p>
<p>— You: “Wait, no, my mistake, I want it back.”</p>
<p>— Computer: “Sorry, it’s all gone, as you requested.”</p>
<p>— You: “…”</p>
<p>— Computer: “I’m not even sorry.”</p>
</blockquote>
<p>Never has this famous quote been more true:</p>
<blockquote>
<p>With great power come great responsibility</p>
</blockquote>
<p>Learning your way around a terminal really is a fundamental shift in how
you usually interact with computers. Instead of working inside the
boundaries of an application, a terminal gives you free and unlimited
access to every part of the computer. The littles wheels are off, and
you are only limited by the number of commands you know. Consequently,
learning how to use the terminal will give you insights about how your
computer works. Let’s see what we can do. We’ll start small, but trust
me, it gets better.</p>
<p><a id="your-first-steps"></a></p>
<h2 id="your-first-steps">Your first steps</h2>
<p>First off, let’s define a couple of words.</p>
<p>A terminal is an application you can open on your computer, in which
you’ll be able to type commands in a command line interface (CLI). When
you hit the <kbd>Enter</kbd> key, the command will be executed by a
program called a <em>shell</em>, and the result is displayed back in the terminal.</p>
<p>In the early days of computing, video terminals were actual physical
devices, used to execute commands onto a remote computer that could take
a whole room.</p>
<p><img alt="The DEC VT100, a physical video terminal dating back 1978" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/physical-terminal.png">
<span class=imgcaption>The DEC VT100, a physical video terminal dating back 1978</span></p>
<p>Nowadays, terminals are programs run into a graphical window, emulating
the behavior of the video terminals of old.</p>
<p><img alt="This is what a terminal looks like nowadays" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/shell.png">
<span class=imgcaption>This is what a terminal looks like nowadays.</span></p>
<p>Different operating systems come with different terminals and different
shells pre-installed, but the most common shell out there is certainly <code>bash</code>.</p>
<p>Before we go any deeper, let’s open a terminal! The way you do this however depends on your operating system.</p>
<h3 id="opening-a-terminal">Opening a terminal</h3>
<h4 id="on-macos">On MacOS</h4>
<p>Open the <code>Finder</code> app, click on <code>Applications</code> on the left pane, then
enter the <code>Utilities</code> directory, then execute the <code>Terminal</code> app. You
can also use the Spotlight search by clicking on the magnifying glass
icon on the top right corner of your screen (or use the <kbd>Cmd</kbd>
<kbd>Space</kbd> keyboard shortcut), and type <code>Terminal</code>.</p>
<h4 id="on-linux">On Linux</h4>
<p>Depending on the Linux distribution you use, it might come with XTerm,
Gnome-Terminal or Konsole pre-installed. Look for any of these in your
applications menu. A lot of Linux installation use the
<kbd>Ctrl</kbd> - <kbd>Alt</kbd> - <kbd>T</kbd> keyboard shortcut to
open a terminal window.</p>
<h4 id="on-windows">On Windows</h4>
<p>Windows is a special case: Linux and MacOS come with bash pre-installed, whereas Windows does not. It comes with 2 built-in shells: <code>cmd</code> and <code>Powershell</code>. The rest of this tutorial and its following chapters however assume you are running bash. The reason for that is that <code>bash</code> is pretty much ubiquitous, whether it's on a personal workstations or on servers. On top of that, bash comes with a myriad of tools and commands that will be detailed in the next chapter.</p>
<p>Fortunately, Windows 10 can now natively run bash since 2019 by using the <em>Windows Subsystem for Linux</em> (WSL). We suggest you follow the instructions from this <a href="https://itsfoss.com/install-bash-on-windows/">tutorial</a>.</p>
<p><img alt="Running bash on Windows is now possible" decoding="async" loading="lazy" src="https://i1.wp.com/itsfoss.com/wp-content/uploads/2016/08/install-ubuntu-windows-10-linux-subsystem-10.jpeg">
<span class=imgcaption>Running bash on Windows is now possible</span></p>
<h3 id="running-our-first-command">Running our first command</h3>
<p>When you open your terminal, the first thing you will see is a <em><span
class="gls" key="prompt">prompt</span></em>. It is what is displayed every
time the shell is ready for its next order. It is common for the prompt
to display information useful for the user. In my case, <code>br</code> is my
username, and <code>morenika</code> is my computer’s name (its <em><span class="gls"
key="hostname">hostname</span></em>).</p>
<p><img alt="br@morenika:~$ is my prompt" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/shell.png">
<span class=imgcaption><code>br@morenika:~$</code> is my prompt</span></p>
<p>The black rectangle is called a <em>cursor</em>. It represents your current
typing position.</p>
<div class="Note">
<p>What <em>your</em> prompt actually looks like depends on your operating system
and your shell. Don’t worry if it does not look exactly the same as the
one in the following examples.</p>
</div>
<p>The first command we will run is <code>ls</code> (which stands for <em>list
directory</em>). By default, that command lists all directories and files
present in the directory we currently are located into. To run that
command, we need to type <code>ls</code> after the prompt, and then hit
<kbd>Enter</kbd></p>
<p>The text that is displayed after our command and before the next prompt
is the command’s <em>output</em>.</p>
<div class="highlight"><pre><span></span><code>br@morenika:~$<span class="w"> </span><span class="nb">ls</span>
Android<span class="w"> </span>code<span class="w"> </span>Downloads<span class="w"> </span>Music
AndroidStudioProjects<span class="w"> </span>Desktop<span class="w"> </span>Dropbox<span class="w"> </span>Pictures
bin<span class="w"> </span>Documents<span class="w"> </span>Firefox_wallpaper.png<span class="w"> </span>Videos
</code></pre></div>
<p>These are all the files and directories located in my personal directory
(also called <em><span class="gls" key="homedir">home directory</span></em>).
Let’s open a graphical file explorer and check, just to be sure.</p>
<p><img alt="As expected, we weren’t lied to" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/home.png">
<span class=imgcaption>As expected, we weren’t lied to</span></p>
<div class="Warning">
<p>The shell is sensitive to casing: a lower-case command is not the same
thing as it’s upper case equivalent.</p>
<div class="highlight"><pre><span></span><code>br@morenika:~$<span class="w"> </span>LS
bash:<span class="w"> </span>LS:<span class="w"> </span><span class="nb">command</span><span class="w"> </span>not<span class="w"> </span>found
</code></pre></div>
</div>
<p>As of now, we will ignore the <code>br@morenika:~$</code> prompt prefix and will
only use <code>$</code>, to keep our examples short.</p>
<h3 id="commands-arguments">Commands arguments</h3>
<p>In our last example, we listed all files and directories located in my
home directory. What if I wanted to list all files located in the <code>bin</code>
directory that we can see in the output? In that case, I could pass
<code>bin</code> as an <em>argument</em> to the <code>ls</code> command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>bin
bat<span class="w"> </span>fix-vlc-size<span class="w"> </span>lf<span class="w"> </span>terraform<span class="w"> </span>vpnconnect
clean-desktop<span class="w"> </span>itresize<span class="w"> </span>nightlight<span class="w"> </span>tv-mode
</code></pre></div>
<p>By passing the <code>bin</code> argument to the <code>ls</code> command, we told it where to
look, and we thus changed its behavior. Note that it is possible to pass
more than one argument to a command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>Android<span class="w"> </span>bin
Android:
Sdk
bin:
bat<span class="w"> </span>clean-desktop<span class="w"> </span>fix-vlc-size<span class="w"> </span>itresize<span class="w"> </span>lf<span class="w"> </span>nightlight<span class="w"> </span>terraform<span class="w"> </span>tv-mode<span class="w"> </span>vpnconnect
</code></pre></div>
<p>In that example, we passed two arguments to <code>ls</code>: <code>bin</code> and <code>Android</code>.
<code>ls</code> then proceeded to list the content of each these 2 directories.</p>
<p>Think about how you would have done that in a File explorer GUI. You
probably would have gone into the first directory, then gone back to the
parent directory and finally proceeded with the next directory. The
terminal allows you to be more efficient.</p>
<h3 id="command-options">Command options</h3>
<p>Now, let’s say I’d also like to see how big files located under <code>bin</code>
are. No problem! The <code>ls</code> command has <em>options</em> we can use to adjust its
behavior. The <code>-s</code> option causes <code>ls</code> to display each file size, in
kilobytes.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-s<span class="w"> </span>bin
total<span class="w"> </span><span class="m">52336</span>
<span class="w"> </span><span class="m">4772</span><span class="w"> </span>bat<span class="w"> </span><span class="m">4</span><span class="w"> </span>itresize<span class="w"> </span><span class="m">44296</span><span class="w"> </span>terraform
<span class="w"> </span><span class="m">4</span><span class="w"> </span>clean-desktop<span class="w"> </span><span class="m">3244</span><span class="w"> </span>lf<span class="w"> </span><span class="m">4</span><span class="w"> </span>tv-mode
<span class="w"> </span><span class="m">4</span><span class="w"> </span>fix-vlc-size<span class="w"> </span><span class="m">4</span><span class="w"> </span>nightlight<span class="w"> </span><span class="m">4</span><span class="w"> </span>vpnconnect
</code></pre></div>
<p>While this is nice, I’d prefer to see the file size in a human-readable
unit. I can add the <code>-h</code> option to further specify what <code>ls</code> has to do.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-s<span class="w"> </span>-h<span class="w"> </span>bin
total<span class="w"> </span>52M
<span class="m">4</span>.7M<span class="w"> </span>bat<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>itresize<span class="w"> </span>44M<span class="w"> </span>terraform
<span class="m">4</span>.0K<span class="w"> </span>clean-desktop<span class="w"> </span><span class="m">3</span>.2M<span class="w"> </span>lf<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>tv-mode
<span class="m">4</span>.0K<span class="w"> </span>fix-vlc-size<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>nightlight<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>vpnconnect
</code></pre></div>
<p>I can separate both options with a space, or also group them as one
option.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-sh<span class="w"> </span>bin
total<span class="w"> </span>52M
<span class="m">4</span>.7M<span class="w"> </span>bat<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>itresize<span class="w"> </span>44M<span class="w"> </span>terraform
<span class="m">4</span>.0K<span class="w"> </span>clean-desktop<span class="w"> </span><span class="m">3</span>.2M<span class="w"> </span>lf<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>tv-mode
<span class="m">4</span>.0K<span class="w"> </span>fix-vlc-size<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>nightlight<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>vpnconnect
</code></pre></div>
<p>I’d finally like each file and its associated size to be displayed on
its own line. Enter the <code>-1</code> option!</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-s<span class="w"> </span>-h<span class="w"> </span>-1<span class="w"> </span>bin
total<span class="w"> </span>52M
<span class="m">4</span>.7M<span class="w"> </span>bat
<span class="m">4</span>.0K<span class="w"> </span>clean-desktop
<span class="m">4</span>.0K<span class="w"> </span>fix-vlc-size
<span class="m">4</span>.0K<span class="w"> </span>itresize
<span class="m">3</span>.2M<span class="w"> </span>lf
<span class="m">4</span>.0K<span class="w"> </span>nightlight
<span class="w"> </span>44M<span class="w"> </span>terraform
<span class="m">4</span>.0K<span class="w"> </span>tv-mode
<span class="m">4</span>.0K<span class="w"> </span>vpnconnect
</code></pre></div>
<p>Short options make it easy to type a command quickly, but the result can
be hard to decipher after a certain amount of options, and you might
find yourself wondering what the command is doing in the first place.
Luckily, options can have a <em>long form</em> and a <em>short form</em>. For example,
<code>-s</code> can be replaced by its long form <code>--size</code>, and <code>-h</code> by
<code>--human-readable</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>--size<span class="w"> </span>--human-readable<span class="w"> </span>-1<span class="w"> </span>bin
total<span class="w"> </span>52M
<span class="m">4</span>.7M<span class="w"> </span>bat
<span class="m">4</span>.0K<span class="w"> </span>clean-desktop
<span class="m">4</span>.0K<span class="w"> </span>fix-vlc-size
<span class="m">4</span>.0K<span class="w"> </span>itresize
<span class="m">3</span>.2M<span class="w"> </span>lf
<span class="m">4</span>.0K<span class="w"> </span>nightlight
<span class="w"> </span>44M<span class="w"> </span>terraform
<span class="m">4</span>.0K<span class="w"> </span>tv-mode
<span class="m">4</span>.0K<span class="w"> </span>vpnconnect
</code></pre></div>
<p>The command feels way more self-explanatory this way! You’ll notice that
we still used the short form for the <code>-1</code> option. The reason for that is
that this option simply does not have a long form.</p>
<h3 id="takeaways">Takeaways</h3>
<ul>
<li>A terminal is an application through which you interact with a shell</li>
<li>You can execute commands by typing them in the shell’s command-line
and hitting <kbd>Enter</kbd></li>
<li>A command can take 0, 1 or more arguments</li>
<li>A command’s behavior can be changed by passing options</li>
<li>By convention, options can have have multiple forms: a short and/or
a long one.</li>
</ul>
<p><img alt="Here is a summary of the different parts of a command" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/terminal-parts.png">
<span class=imgcaption>Here is a summary of the different parts of a command</span></p>
<p><a id="managing-files"></a></p>
<h2 id="managing-files">Managing files</h2>
<p>So far, we’ve seen how to run a command, changing its behavior by
passing command-line arguments and options, and that <code>ls</code> is used to
list the content of a directory. It’s now time to learn about how to
managing your files, by creating files and directories, copying and
moving them around, creating links, etc. The goal of this section is to
teach you how to do everything you usually do in your file explorer, but
in your terminal.</p>
<h3 id="pwd-cd-navigating-between-directories"><code>pwd</code>, <code>cd</code>: navigating between directories</h3>
<p>Up to now, every command we’ve run were executed from our <em><span
class="gls" key="homedir">home directory</span></em> (the directory in which
you have all your documents, downloads, etc). The same way you can
navigate directories in a graphical file editor, you can do it in a
terminal as well.</p>
<p>Before going anywhere, we first need to figure out where we are. Enters
<code>pwd</code>, standing for <em>print working directory</em>. This command displays
your current working directory (<em>a.k.a</em> where you are).</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">pwd</span>
/home/br
</code></pre></div>
<p>Now that we found our bearings, we can finally move around. We
can do that with the <code>cd</code> command, standing for (you might have guessed
it) <em>change directory</em>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>Documents
$<span class="w"> </span><span class="nb">pwd</span>
/home/br/Documents
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>./invoices
$<span class="w"> </span><span class="nb">pwd</span>
/home/br/Documents/invoices
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="m">2020</span>
$<span class="w"> </span><span class="nb">pwd</span>
/home/br/Documents/invoices/2020
</code></pre></div>
<p>As <code>2020</code> is empty, we can’t go any further. However, we can also
go back to the <em>parent directory</em> (the directory containing the one we
are currently into) using <code>cd ..</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">pwd</span>
/home/br/Documents/invoices/2020
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>..
$<span class="w"> </span><span class="nb">pwd</span>
/home/br/Documents/invoices
</code></pre></div>
<p>We don’t have to always change directory one level at the time. We can
go up multiples directories at a time.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">pwd</span>
/home/br/Documents/invoices
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>../..
$<span class="w"> </span><span class="nb">pwd</span>
/home/br
</code></pre></div>
<p>We can also go several directories down at the same time</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">pwd</span>
/home/br
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>Documents/invoices/2020
</code></pre></div>
<p>Running <code>cd</code> without arguments takes you back to your home directory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">pwd</span>
/home/br/Documents/invoices/2020
$<span class="w"> </span><span class="nb">cd</span>
$<span class="w"> </span><span class="nb">pwd</span>
/home/br
</code></pre></div>
<p>Running <code>cd -</code> takes you back to your previous location.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">pwd</span>
/home/br/Documents/invoices/2020
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/home/br
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>-
$<span class="w"> </span><span class="nb">pwd</span>
/home/br/Documents/invoices/2020
</code></pre></div>
<p>You might wonder why <code>cd ..</code> takes you back to the parent directory?
What does <code>..</code> mean? To understand this, we need to explore how <em>paths</em>
work.</p>
<h3 id="paths-root-absolute-and-relative">Paths: root, absolute and relative</h3>
<p>If you have never used a terminal before, and have only navigated
between directories using a graphical file explorer, the notion of
<em>path</em> might be a bit foreign. A path is a unique location to a file or
a folder on your file system. The easiest way to explain it is by
describing how files and directories are organized on your disk.</p>
<p>The base directory (also called <span class="gls" key="rootdir">root
directory</span>, and referred as <code>/</code>) is the highest directory in the
hierarchy: it contains every single file and directory in your
system, each of these directories possibly containing others, to form a
structure looking like a tree.</p>
<p><img alt="Your disk is organized like a tree" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/paths.png">
<span class=imgcaption>Your disk is organized like a tree</span></p>
<p>Let’s look at what that <code>/</code> root directory contains.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>/
bin<span class="w"> </span>boot<span class="w"> </span>dev<span class="w"> </span>etc<span class="w"> </span>home<span class="w"> </span>lib<span class="w"> </span>lib64<span class="w"> </span>lost+found<span class="w"> </span>media
mnt<span class="w"> </span>opt<span class="w"> </span>proc<span class="w"> </span>root<span class="w"> </span>run<span class="w"> </span>sbin<span class="w"> </span>srv<span class="w"> </span>sys<span class="w"> </span>tmp<span class="w"> </span>usr<span class="w"> </span>var
</code></pre></div>
<p>Ok so, there are a couple of things in there. We have talked about home
directories before, remember? It turns out that all the users’ home
directories are located under the <code>home</code> directory. As <code>home</code> is located
under <code>/</code>, we can refer it via its <em><span class="gls"
key="abspath">absolute path</span></em>, that is to say the full path to a
given directory, starting from the root directory. In the case of
<code>home</code>, its absolute path is <code>/home</code>, as it is directly located under
<code>/</code>.</p>
<div class="Note">
<p>Any path starting with <code>/</code> is an absolute path.</p>
</div>
<p>We can then use that path to inspect the content of the <code>home</code> directory
with the <code>ls</code> command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>/home
br
</code></pre></div>
<p>The absolute path of <code>br</code> is <code>/home/br</code>. Each directory is separated
from its parent by a <code>/</code>. This is why the root directory is called <code>/</code>:
it is the only directory without a parent.</p>
<p>Any path that does not start with <code>/</code> will be a <em><span class="gls"
key="relpath">relative path</span></em>, meaning that it will be relative to
the current directory. When we executed the <code>ls bin</code> command, <code>bin</code> was
actually a relative path. Indeed, we executed that command while we were
located in <code>/home/br</code>, meaning that the absolute path of <code>bin</code> was
<code>/home/br/bin</code>.</p>
<p>Each folder on disk has a link to itself called <code>.</code>, and and link to its
parent folder called <code>..</code>.</p>
<p><img alt="The . link points to the folder itself and the .. link points to
the folder’s parent" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/dotpaths.png">
<span class=imgcaption>The <code>.</code> link points to the folder itself and the <code>..</code> link points to the folder’s parent</span>.</p>
<p>We can use these <code>.</code> and <code>..</code> links when constructing relative paths.
For example, if you were located in <code>/home/br</code>, you could refer to
the <code>Android</code> folder as <code>./Android</code>, meaning “the <code>Android</code> folder
located under <code>.</code> (the current directory)”.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>./Android
Sdk
</code></pre></div>
<p>Were you located under <code>/home/br/Android</code>, you could also refer to
<code>/home/br/Downloads</code> as <code>../Downloads</code>.</p>
<p><img alt="Following Android’s .. link takes you back to the home directory" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/dotdot.png">
<span class=imgcaption>Following <code>Android</code>’s <code>..</code> link takes you back to the <code>home</code> director</span></p>
<div class="Note">
<p><code>ls -a</code> allows you to see <em>hidden files</em>, a.k.a all files starting with
a dot. We can indeed see the <code>.</code> and <code>..</code> links!</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-a
.<span class="w"> </span>..<span class="w"> </span>Sdk
</code></pre></div>
</div>
<h3 id="mkdir-creating-directories"><code>mkdir</code>: creating directories</h3>
<div class="Note">
<p>In order to make sure that we don’t mess with your personal files when
testing out the commands from this chapter, we will start by creating a
new directory to experiment in, called <code>experiments</code>.</p>
</div>
<p>You can create a new directory using the <code>mkdir</code> command, which stands
for <em>make directories</em>. By executing the command <code>mkdir experiments</code>,
you will create the <code>experiments</code> directory in your current directory.
Let’s test this out.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span>
Android<span class="w"> </span>code<span class="w"> </span>Downloads<span class="w"> </span>Music
AndroidStudioProjects<span class="w"> </span>Desktop<span class="w"> </span>Dropbox<span class="w"> </span>Pictures
bin<span class="w"> </span>Documents<span class="w"> </span>Firefox_wallpaper.png<span class="w"> </span>Videos
$<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>experiments
$
</code></pre></div>
<p>Notice that the <code>mkdir</code> command did not display anything. It might feel
unusual at first, but this is the philosophy of these commands: only
display something if something went wrong. In other terms, no news if good
news.</p>
<p>We can now check that the directory has been created.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span>
Android<span class="w"> </span>bin<span class="w"> </span>Desktop<span class="w"> </span>Downloads<span class="w"> </span>experiments<span class="w"> </span>Music<span class="w"> </span>Videos
AndroidStudioProjects<span class="w"> </span>code<span class="w"> </span>Documents<span class="w"> </span>Dropbox<span class="w"> </span>Firefox_wallpaper.png<span class="w"> </span>Pictures
</code></pre></div>
<p>We can also see that directory by opening our file explorer.</p>
<p><img alt="The directory we have just created in the terminal can be seen in our file explorer. The terminal displays the information as text, and the file explorer displays it in a graphical form." decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/new-dir.png">
<span class=imgcaption>The directory we have just created in the terminal can be seen in our file explorer. The terminal displays the information as text, and the file explorer displays it in a graphical form.</span></p>
<p>Running <code>mkdir</code> on a pre-existing command causes it to fail and display
an error message.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>experiments
mkdir:<span class="w"> </span>experiments:<span class="w"> </span>File<span class="w"> </span>exists
</code></pre></div>
<p>What if we wanted to create a directory in <code>experiments</code> called <code>art</code>,
and another directory called <code>paintings</code> itself located into <code>art</code>?</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>experiments/art/paintings
mkdir:<span class="w"> </span>experiments/art:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
</code></pre></div>
<p>Something clearly went wrong here. <code>mkdir</code> is complaining that it cannot
create <code>paintings</code> within <code>experiments/art</code> as it does not exist. We
could create <code>art</code> and then <code>paintings</code>, in two separate commands, but
fortunately, <code>mkdir</code> provides us with a <code>-p</code> option that causes <code>mkdir</code>
to succeed even if directories already exist, and that will create each
parent directory.</p>
<div class="highlight"><pre><span></span><code>-p,<span class="w"> </span>--parents:<span class="w"> </span>no<span class="w"> </span>error<span class="w"> </span><span class="k">if</span><span class="w"> </span>existing,<span class="w"> </span>make<span class="w"> </span>parent<span class="w"> </span>directories<span class="w"> </span>as<span class="w"> </span>needed
</code></pre></div>
<p>This looks like exactly what we need in that case! Let’s see if it works
as expected.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mkdir</span><span class="w"> </span>-p<span class="w"> </span>experiments/art/paintings
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments
art
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments/art
paintings
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments/art/paintings
$
</code></pre></div>
<h3 id="cp-mv-moving-files-around"><code>cp</code>, <code>mv</code>: moving files around</h3>
<p><code>cp</code> (standing for <code>copy</code>) allows you to copy a file or a directory to
another location.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cp</span><span class="w"> </span>Documents/readme<span class="w"> </span>experiments/art
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments/art
paintings<span class="w"> </span>readme
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>Documents
readme
</code></pre></div>
<p>You can also move the file from a location to another by using <code>mv</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mv</span><span class="w"> </span>experiments/art/readme<span class="w"> </span>experiments
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments
art<span class="w"> </span>readme
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments/art
paintings
</code></pre></div>
<p>That does not seem to work on directories however.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cp</span><span class="w"> </span>experiments/art<span class="w"> </span>experiments/art-copy
cp:<span class="w"> </span>experiments/art<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span>directory<span class="w"> </span><span class="o">(</span>not<span class="w"> </span>copied<span class="o">)</span>.
</code></pre></div>
<p>By default, <code>cp</code> only works on files, and not on directories. We need to
use the <code>-r</code> option to tell <code>cp</code> to recursively copy <code>experiments/art</code>
to <code>experiments/art-copy</code>, meaning <code>cp</code> will copy the directory and
every file and directories it contains.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cp</span><span class="w"> </span>-r<span class="w"> </span>experiments/art<span class="w"> </span>experiments/art-copy
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments
art-copy<span class="w"> </span>art<span class="w"> </span>readme
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments/art
paintings
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments/art-copy
paintings
</code></pre></div>
<p>Finally, you can use <code>mv</code> to rename a file or a directory. It might
sound surprising that there is not <code>rn</code> or <code>rename</code> command, but
renaming a file is actually just moving it to another location in the
same directory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">mv</span><span class="w"> </span>experiments/readme<span class="w"> </span>experiments/README
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments
README<span class="w"> </span>art-copy<span class="w"> </span>art
</code></pre></div>
<h3 id="rm-removing-files-and-directories"><code>rm</code>: removing files and directories</h3>
<p>The <code>rm</code> copy allows you to delete files and directories.</p>
<div class="Warning">
<p><strong>Be careful with <code>rm</code></strong>, when a file is deleted, it is not moved to the
trash, it is gone.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">rm</span><span class="w"> </span>experiments/README
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments
art-copy<span class="w"> </span>art
</code></pre></div>
</div>
<p><code>rm</code> behaves like <code>cp</code>: it only allows you to remove directories by
using the <code>-r</code> option.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">rm</span><span class="w"> </span>experiments/art
rm:<span class="w"> </span>experiments/art:<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span>directory
$<span class="w"> </span><span class="nb">rm</span><span class="w"> </span>-r<span class="w"> </span>experiments/art
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments
art-copy
$<span class="w"> </span><span class="nb">rm</span><span class="w"> </span>-r<span class="w"> </span>experiments/art-copy
$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>experiments
$
</code></pre></div>
<h3 id="ln-creating-links"><code>ln</code>: creating links</h3>
<p>Have you ever created a shortcut to a file on your desktop? Behind the
scenes, this works using a <em>symbolic link</em>. A link points to the
original file, and allows you to access that file from multiple places,
without actually having to store multiple copies on disk.</p>
<p>We can create such a link by using the <code>ln -s</code> command (<code>-s</code> stands for
<em>symbolic</em>).</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">pwd</span>
/home/br
$<span class="w"> </span><span class="nb">ln</span><span class="w"> </span>-s<span class="w"> </span>Documents/readme<span class="w"> </span>Desktop/my-readme
</code></pre></div>
<p>Using the <code>-l</code> option of <code>ls</code>, we can see where a link points to.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-l<span class="w"> </span>Desktop
total<span class="w"> </span><span class="m">0</span>
lrwxr-xr-x<span class="w"> </span><span class="m">1</span><span class="w"> </span>br<span class="w"> </span>br<span class="w"> </span><span class="m">21</span><span class="w"> </span>Jan<span class="w"> </span><span class="m">17</span><span class="w"> </span><span class="m">16</span>:48<span class="w"> </span>my-readme<span class="w"> </span>-><span class="w"> </span>/home/br/Documents/readme
</code></pre></div>
<div class="Note">
<p>My personal mnemonic to remember the order of arguments is by
remembering <em>s for source</em>: the source file goes after the <code>-s</code> option.
<code>ln -s <source> <destination></code></p>
</div>
<h3 id="tree-visualizing-files-and-subfolders"><code>tree</code>: visualizing files and subfolders</h3>
<p><code>tree</code> displays the content of the current directory (or argument
directory) and its subfolders in a tree-like representation. It is very
useful to have a quick look at the current content of a directory,</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">tree</span><span class="w"> </span>experiments
experiments
<span class="k">|</span>__<span class="w"> </span>art
<span class="w"> </span><span class="k">|</span>__<span class="w"> </span>paintings
<span class="m">2</span><span class="w"> </span>directories,<span class="w"> </span><span class="m">0</span><span class="w"> </span>files
</code></pre></div>
<div class="Note">
<p><code>tree</code> might not be installed by default, depending on your system. We
mention it here as we will re-use it throughout the chapters.</p>
</div>
<p><a id="learning-new-options"></a></p>
<h2 id="learning-new-options">Learning new options</h2>
<h3 id="getting-help">Getting help</h3>
<p>If you are wondering how you will be able to remember all these options,
don’t worry. Nobody expects you to know all of the options of all the
commands by heart. You can rely on the commands’ documentation instead
of having to memorize them all.</p>
<p>Most of the commands out there take a <code>-h</code> (or <code>--help</code>) option that
will display the list of options the command itself can take, and what
they do.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>--help
Usage:<span class="w"> </span><span class="nb">ls</span><span class="w"> </span><span class="o">[</span>OPTION<span class="o">]</span>...<span class="w"> </span><span class="o">[</span>FILE<span class="o">]</span>...
List<span class="w"> </span>information<span class="w"> </span>about<span class="w"> </span>the<span class="w"> </span>FILEs<span class="w"> </span><span class="o">(</span>the<span class="w"> </span>current<span class="w"> </span>directory<span class="w"> </span>by<span class="w"> </span>default<span class="o">)</span>.
Sort<span class="w"> </span>entries<span class="w"> </span>alphabetically<span class="w"> </span><span class="k">if</span><span class="w"> </span>none<span class="w"> </span>of<span class="w"> </span>-cftuvSUX<span class="w"> </span>nor<span class="w"> </span>--sort<span class="w"> </span>is<span class="w"> </span>specified.
Mandatory<span class="w"> </span>arguments<span class="w"> </span>to<span class="w"> </span>long<span class="w"> </span>options<span class="w"> </span>are<span class="w"> </span>mandatory<span class="w"> </span><span class="k">for</span><span class="w"> </span>short<span class="w"> </span>options<span class="w"> </span>too.
<span class="w"> </span>-a,<span class="w"> </span>--all<span class="w"> </span><span class="k">do</span><span class="w"> </span>not<span class="w"> </span>ignore<span class="w"> </span>entries<span class="w"> </span>starting<span class="w"> </span>with<span class="w"> </span>.
<span class="w"> </span>-A,<span class="w"> </span>--almost-all<span class="w"> </span><span class="k">do</span><span class="w"> </span>not<span class="w"> </span>list<span class="w"> </span>implied<span class="w"> </span>.<span class="w"> </span>and<span class="w"> </span>..
<span class="w"> </span>--author<span class="w"> </span>with<span class="w"> </span>-l,<span class="w"> </span>print<span class="w"> </span>the<span class="w"> </span>author<span class="w"> </span>of<span class="w"> </span>each<span class="w"> </span>file
<span class="w"> </span>-b,<span class="w"> </span>--escape<span class="w"> </span>print<span class="w"> </span>C-style<span class="w"> </span>escapes<span class="w"> </span><span class="k">for</span><span class="w"> </span>nongraphic<span class="w"> </span>characters
<span class="w"> </span>--block-size<span class="o">=</span>SIZE<span class="w"> </span>with<span class="w"> </span>-l,<span class="w"> </span>scale<span class="w"> </span>sizes<span class="w"> </span>by<span class="w"> </span>SIZE<span class="w"> </span>when<span class="w"> </span>printing<span class="w"> </span>them<span class="p">;</span>
<span class="w"> </span>e.g.,<span class="w"> </span><span class="s1">'--block-size=M'</span><span class="p">;</span><span class="w"> </span>see<span class="w"> </span>SIZE<span class="w"> </span>format<span class="w"> </span>below
<span class="w"> </span>-B,<span class="w"> </span>--ignore-backups<span class="w"> </span><span class="k">do</span><span class="w"> </span>not<span class="w"> </span>list<span class="w"> </span>implied<span class="w"> </span>entries<span class="w"> </span>ending<span class="w"> </span>with<span class="w"> </span><span class="k">~</span>
<span class="w"> </span>-c<span class="w"> </span>with<span class="w"> </span>-lt:<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>by,<span class="w"> </span>and<span class="w"> </span>show,<span class="w"> </span>ctime<span class="w"> </span><span class="o">(</span><span class="nb">time</span><span class="w"> </span>of<span class="w"> </span>last
<span class="w"> </span>modification<span class="w"> </span>of<span class="w"> </span>file<span class="w"> </span>status<span class="w"> </span>information<span class="o">)</span><span class="p">;</span>
<span class="w"> </span>with<span class="w"> </span>-l:<span class="w"> </span>show<span class="w"> </span>ctime<span class="w"> </span>and<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>by<span class="w"> </span>name<span class="p">;</span>
<span class="w"> </span>otherwise:<span class="w"> </span><span class="nb">sort</span><span class="w"> </span>by<span class="w"> </span>ctime,<span class="w"> </span>newest<span class="w"> </span>first
<span class="o">[</span><span class="nb">cut</span><span class="w"> </span><span class="k">for</span><span class="w"> </span>brevity<span class="o">]</span>
</code></pre></div>
<p>It’s interesting to note that some options accept both short and long
forms, like <code>-a/--all</code>, while some others only accept a short form
(<code>-c</code>) or a long form (<code>--author</code>). There’s no real rule there, only
conventions. A command might not even accept a <code>--help</code> option, but most
if not all the common ones do.</p>
<div class="Note">
<p><code>-h</code> is not always the short option for <code>--help</code>. Indeed, we’ve seen
that <code>ls --help</code> prints an overview of all available commands, whereas
<code>ls -h</code> displays units in a human-readable format!</p>
</div>
<h3 id="reading-the-manual">Reading the manual</h3>
<p>Sometimes, there’s no <code>--help</code> option available, or its output isn’t
clear or verbose enough for your taste, or the output is too long to
navigate easily. It’s often a good idea to read the command’s <em><code>man</code></em>
page (<em>man</em> stands for <em>manual</em>).</p>
<p>Let’s give it a go, by typing the following command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">man</span><span class="w"> </span><span class="nb">ls</span>
</code></pre></div>
<p><img alt="man ls displays the manual of the ls command: everything you need to know about what ls can be used for." decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/man-ls.png">
<span class=imgcaption><code>man ls</code> displays the manual of the <code>ls</code> command: everything you need to know about what <code>ls</code> can be used for.</span></p>
<h4 id="reading-the-synopsis">Reading the synopsis</h4>
<p><code>man</code> provides you with a <em>synopsis</em>, describing a specific usage of the
command on each line, along with the associated options and arguments.</p>
<p>The <code>ls</code> synopsis is</p>
<div class="highlight"><pre><span></span><code><span class="n">SYNOPSIS</span>
<span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">[</span><span class="n">OPTION</span><span class="o">]</span><span class="p">...</span><span class="w"> </span><span class="o">[</span><span class="n">FILE</span><span class="o">]</span><span class="p">...</span>
</code></pre></div>
<p>The square brackets around <code>[OPTION]</code> and <code>[FILE]</code> mean that both options and files are
<em>optional</em>. As we’ve seen at the beginning of this chapter, just running
<code>ls</code> on its own prints the content of the current working directory.</p>
<p>The <code>...</code> following <code>[OPTION]</code> and <code>[FILE]</code> means that several options
and several files arguments can be passed as arguments to <code>ls</code>, as
illustrated by the following example.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-sh<span class="w"> </span>Android<span class="w"> </span>bin
Android:
total<span class="w"> </span><span class="m">4</span>.0K
<span class="m">4</span>.0K<span class="w"> </span>Sdk
bin:
total<span class="w"> </span>52M
<span class="m">4</span>.7M<span class="w"> </span>bat<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>fix-vlc-size<span class="w"> </span><span class="m">3</span>.2M<span class="w"> </span>lf<span class="w"> </span>44M<span class="w"> </span>terraform<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>vpnconnect
<span class="m">4</span>.0K<span class="w"> </span>clean-desktop<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>itresize<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>nightlight<span class="w"> </span><span class="m">4</span>.0K<span class="w"> </span>tv-mode
</code></pre></div>
<p>If we look at the <code>mkdir</code> synopsis, we see that options are, well,
optional, but we must provide it with one or more directories to create,
because <code>DIRECTORY</code> is not between square brackets.</p>
<div class="highlight"><pre><span></span><code><span class="n">SYNOPSIS</span>
<span class="w"> </span><span class="n">mkdir</span><span class="w"> </span><span class="o">[</span><span class="n">OPTION</span><span class="o">]</span><span class="p">...</span><span class="w"> </span><span class="n">DIRECTORY</span><span class="p">...</span>
</code></pre></div>
<p>The <code>DESCRIPTION</code> section will list all possible options (short and long
forms), along with their effect.</p>
<h4 id="navigating-the-manual">Navigating the manual</h4>
<p>When you run <code>man</code>, the manual of the command will be displayed in a
<em><span class="gls" key="pager"> pager</span></em>, a piece of software that
helps the user get the output one page at a time. One of the most common
pager commands is <code>less</code> (which is incidentally the more featureful
successor of <code>more</code>, because <em>less is more</em>). Being dropped into a pager
for the first time is confusing, as you might not know how to to
navigate.</p>
<p>The most useful commands you can type within <code>less</code> are:</p>
<ul>
<li><code>h</code>: display the <code>less</code> help</li>
<li><code>q</code>: exit <code>less</code></li>
<li><code>/pattern</code>: look for the input text located after the cursor’s
current position</li>
<li><code>n</code>: go to next pattern occurrence</li>
<li><code>?pattern</code>: look for the input text located before the cursor’s
current position</li>
<li><code>N</code> go to the pattern previous occurrence</li>
<li>up or down arrow to navigate up or down a line</li>
<li>PageUp and PageDown keys to navigate up or down a page</li>
<li><code>g</code> go to the beginning of the file</li>
<li><code>G</code> go to the end of the file</li>
</ul>
<p>For example, if you’re not sure what the <code>-s</code> <code>ls</code> option is doing, you
can type <code>man ls</code> and then <code>/-s</code> when you are in <code>less</code>. Type <code>n</code> until
you find the documentation for <code>-s, --size</code> (or <code>N</code> to go back if you
went too far). Once you’re done, you can exit <code>less</code> by typing <code>q</code>.</p>
<p>While <code>man</code> uses <code>less</code> under the hood to help you read documentation,
you can simply use <code>less</code> to page through any file your disk. For
example, I can use this command on my computer.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">less</span><span class="w"> </span>Documents/readme
</code></pre></div>
<p><img alt="" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/less-readme.png"></p>
<p>You can look into the <code>less</code> help itself, by typing <code>h</code> when reading a
man page, by typing <code>less --help</code> in a terminal, or even <code>man less</code>!</p>
<p>Exactly like <code>ls</code>, <code>man</code> itself is a command, and as most of the
commands, it has a manual! You can read more about <code>man</code> itself by
typing</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">man</span><span class="w"> </span><span class="nb">man</span>
</code></pre></div>
<p><img alt="Low and behold, the manual’s manual." decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/man-man.png">
<span class="imgcaption">Low and behold, the manual’s manual.</span></p>
<p><a id="command-inputoutput-streams"></a></p>
<h2 id="command-inputoutput-streams">Command Input/Output streams</h2>
<p>Before we can fully explain what makes the shell so powerful, we need to
explain what is an <em><span class="gls" key="iostream">Input Output
stream</span></em>. Every time we run a command, the shell executes a
<em>process</em>, which will then be in charge of running the command, and
communicating its output back to the terminal. Input/Output streams are
the way the shell sends input to a process and dispatches output from
it.</p>
<p>Each process has 3 streams by default:</p>
<ul>
<li><code>stdin</code> (or <em>standard input</em>): provides input to the command</li>
<li><code>stdout</code> (or <em>standard output</em>): displays the command’s output</li>
<li><code>stderr</code> (or <em>standard error</em>): displays the command’s error</li>
</ul>
<p>Each one of these streams has an associated <em><span class="gls"
key="fd">file descriptor</span></em>, a number used by the shell to
reference that stream. <code>stdin</code> has the file descriptor 0, <code>stdout</code> has
1, and <code>stderr</code> has 2.</p>
<p><img alt="stdin (file descriptor 0) is the process input stream, stdout (file descriptor 1) is the process output stream and stderr (file descriptor 2) is the process error stream" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/process-stream.png">
<span class=imgcaption><code>stdin</code> (file descriptor 0) is the process input stream, <code>stdout</code> (file descriptor 1) is the process output stream and <code>stderr</code> (file descriptor 2) is the process error stream.</span></p>
<h3 id="redirecting-output-to-a-file">Redirecting output to a file</h3>
<p>It can be convenient to “save” the output of a command to a file, to
further process it at a later time, or to send it to someone else. You
can use the <code>></code> operator to redirect the <code>stdout</code> of a command to a
file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>/home/br<span class="w"> </span><span class="k">></span><span class="w"> </span>ls-home.txt
</code></pre></div>
<p>We can then display the content of the <code>ls-home.txt</code> file using the
<code>cat</code> command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>ls-home.txt
Android<span class="w"> </span>code<span class="w"> </span>Downloads<span class="w"> </span>Music
AndroidStudioProjects<span class="w"> </span>Desktop<span class="w"> </span>Dropbox<span class="w"> </span>Pictures
bin<span class="w"> </span>Documents<span class="w"> </span>Firefox_wallpaper.png<span class="w"> </span>Videos
</code></pre></div>
<p>If the file doesn’t already exist, it will be created by the shell at
the moment of the redirection. If the file however does exist at
redirection time, it will be overwritten, meaning that anything that
file used to contain will be replaced by the output of the redirected
command.</p>
<p>In that example, we use the <code>echo</code> command, that simply sends the
argument text to its <code>stdout</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>ls-home.txt
Android<span class="w"> </span>code<span class="w"> </span>Downloads<span class="w"> </span>Music
AndroidStudioProjects<span class="w"> </span>Desktop<span class="w"> </span>Dropbox<span class="w"> </span>Pictures
bin
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Hello world!"</span><span class="w"> </span><span class="k">></span><span class="w"> </span>ls-home.txt
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>ls-home.txt
Hello<span class="w"> </span>world!
</code></pre></div>
<p>If you want to append the output of a command to a file without
overwriting its content, you can use the <code>>></code> operator instead of <code>></code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>echoes
cat:<span class="w"> </span>echoes:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Hey, I just met you, and this is crazy"</span><span class="w"> </span><span class="k">>></span><span class="w"> </span>echoes
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"so here's my echo, so cat it maybe"</span><span class="w"> </span><span class="k">>></span><span class="w"> </span>echoes
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>echoes
Hey,<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you,<span class="w"> </span>and<span class="w"> </span>this<span class="w"> </span>is<span class="w"> </span>crazy
so<span class="w"> </span>here<span class="err">'</span>s<span class="w"> </span>my<span class="w"> </span>echo,<span class="w"> </span>so<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>it<span class="w"> </span>maybe
</code></pre></div>
<h3 id="redirecting-a-file-to-a-commands-input">Redirecting a file to a command’s input</h3>
<p>The same way you can redirect a command’s <code>stdout</code> to a file, you can
redirect a file to a command’s <code>sdtin</code>.</p>
<p>In that example, we’ll redirect the content of the <code>echoes</code> file to the
input of the <code>wc -l</code> command, counting the number of lines of its input
stream or the file(s) passed by argument.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>echoes
Hey,<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you,<span class="w"> </span>and<span class="w"> </span>this<span class="w"> </span>is<span class="w"> </span>crazy
so<span class="w"> </span>here<span class="err">'</span>s<span class="w"> </span>my<span class="w"> </span>echo,<span class="w"> </span>so<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>it<span class="w"> </span>maybe
$<span class="w"> </span><span class="nb">wc</span><span class="w"> </span>-l<span class="w"> </span><span class="k"><</span><span class="w"> </span>echoes
<span class="m">2</span>
</code></pre></div>
<p>You can of course combined the <code><</code>, <code>></code> and <code>>></code> operators in a single
command. In the following example, we will redirect the content of the
<code>echoes</code> file to the <code>wc -l</code> command, and redirect the output of that
command to the <code>echoes-lines</code> files.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">wc</span><span class="w"> </span>-l<span class="w"> </span><span class="k"><</span><span class="w"> </span>echoes<span class="w"> </span><span class="k">></span><span class="w"> </span>echoes-lines
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>echoes-lines
<span class="m">2</span>
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>echoes
Hey,<span class="w"> </span>I<span class="w"> </span>just<span class="w"> </span>met<span class="w"> </span>you,<span class="w"> </span>and<span class="w"> </span>this<span class="w"> </span>is<span class="w"> </span>crazy
so<span class="w"> </span>here<span class="err">'</span>s<span class="w"> </span>my<span class="w"> </span>echo,<span class="w"> </span>so<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>it<span class="w"> </span>maybe
</code></pre></div>
<h3 id="redirecting-multiple-lines-to-a-commands-input">Redirecting multiple lines to a command’s input</h3>
<p>You might find yourself in a situation where you want to pass multiple
lines of input to a command, and the <code><</code> operator fails you in that
case, as it only deals with files. Luckily, your shell provides you with
the <em>heredoc</em> (here document) <code><<</code> operator to accomplish this.</p>
<p>A heredoc redirection has the following syntax:</p>
<div class="highlight"><pre><span></span><code>command <<DELIMITER
a multi-line
string
DELIMITER
</code></pre></div>
<p>The <code>DELIMITER</code> can be any string of your choosing, although <code>EOF</code> (“end
of file”) is pretty commonly used.</p>
<p>Let’s consider the following example:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span><span class="s"><<EOF</span>
<span class="s">My username is br</span>
<span class="s">I'm living at /home/br</span>
<span class="s">EOF</span>
</code></pre></div>
<p>This command will output the following block of text:</p>
<div class="highlight"><pre><span></span><code>My username is br
I'm living at /home/br
</code></pre></div>
<p>You can redirect that block into a file by combining both the <code><<</code> and
<code>></code> operators.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span><span class="s"><<EOF > aboutme</span>
<span class="s">My username is br</span>
<span class="s">I'm living at /home/br</span>
<span class="s">EOF</span>
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>aboutme
My<span class="w"> </span>username<span class="w"> </span>is<span class="w"> </span>br
I<span class="err">'</span>m<span class="w"> </span>living<span class="w"> </span>at<span class="w"> </span>/home/br
</code></pre></div>
<h3 id="redirecting-stderr">Redirecting <code>stderr</code></h3>
<p>Let’s consider the following example.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>-n<span class="w"> </span>notthere<span class="w"> </span><span class="k">></span><span class="w"> </span>notthere-with-line-numbers
cat:<span class="w"> </span>notthere:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>notthere-with-line-numbers
</code></pre></div>
<p>How come the <code>notthere-with-line-numbers</code> file is empty even after we
redirected the <code>cat -n notthere</code> command’s output to it? The reason for
that is, we didn’t really redirect the command’s output to that file, we
redirected the command’s <code>stdout</code>. As the file <code>notthere</code> does not
exist, the <code>cat</code> command fails, and displays an error message on it’s
<code>stderr</code> stream, which wasn’t redirected.</p>
<p>You can redirect a process stream by using its file descriptor.
Remember? 0 for <code>stdin</code>, 1 for <code>stdout</code> and 2 for <code>stderr</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>-n<span class="w"> </span>notthere<span class="w"> </span><span class="m">2</span>>errors.txt
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>errors.txt
cat:<span class="w"> </span>notthere:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
</code></pre></div>
<p>This <code>stderr</code> redirection can be illustrated by the following diagram.</p>
<p><img alt="Any errors displayed by cat will be redirected into the errors.txt file" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/redirect-stderr.png">
<span class=imgcaption>Any errors displayed by <code>cat</code> will be redirected into the <code>errors.txt</code> file</span></p>
<p>You can also redirect the command’s <code>stdout</code> to a file, and its <code>stderr</code>
to another file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>-n<span class="w"> </span>notthere<span class="w"> </span>>output.txt<span class="w"> </span><span class="m">2</span>>errors.txt
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>output.txt
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>errors.txt
cat:<span class="w"> </span>notthere:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
</code></pre></div>
<p><img alt="Normal output will be redirected into output.txt whereas errors are redirected to into errors.txt" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/redirect-stdout-stderr.png">
<span class=imgcaption>Normal output will be redirected into <code>output.txt</code> whereas errors are redirected to into <code>errors.txt</code></span></p>
<p>It is also possible to redirect the command’s <code>stderr</code> into its <code>stdout</code>
using <code>2>&1</code>. This will effectively merge both streams into a single
one.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>notthere<span class="w"> </span><span class="k">></span><span class="w"> </span>output.txt<span class="w"> </span><span class="m">2</span><span class="k">>&</span><span class="m">1</span>
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>output.txt
cat:<span class="w"> </span>notthere:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
</code></pre></div>
<p><img alt="cat’s stdout and stderr are merged together into a single stream" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/redirect-stderr-into-stdout.png">
<span class=imgcaption><code>cat</code>’s stdout and stderr are merged together into a single stream</span></p>
<div class="Note">
<p>The order of redirections has always felt a little bit weird to me.
You’d expect the following syntax to work, as it feels (at least to me)
more logical, by saying “redirect all errors to stdout, and redirect the
whole thing to a file”. It does not work though.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>notthere<span class="w"> </span><span class="m">2</span><span class="k">>&</span><span class="m">1</span><span class="w"> </span><span class="k">></span><span class="w"> </span>output.txt
cat:<span class="w"> </span>notthere:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>output.txt
$
</code></pre></div>
</div>
<p><a id="composing-commands"></a></p>
<h2 id="composing-commands">Composing commands</h2>
<p>Being able to use a myriad of commands, each one with its own purpose,
is powerful. However, the true power of the shell comes from the fact
that these commands can be <strong>combined</strong>. This is where the terminal
takes a radical shift from the philosophy of graphical applications.
Where a <span class="gls" key="gui">GUI</span> allows you to use a set
of predefined tools, the shell allows you to assemble commands into your
own specialized tools.</p>
<p>This is done via the <em><span class="gls" key="pipe">pipe</span></em>: <code>|</code>,
allowing the redirection of a command’s output stream to another
command’s input stream.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>command1<span class="w"> </span><span class="k">|</span><span class="w"> </span>command2
</code></pre></div>
<p>A pipe simply works by connecting the <code>stdout</code> stream of a command to
the <code>stdin</code> stream of the next command. Simply said, the output of a
command becomes the input of the next.</p>
<p><img alt="ls is *piped* into wc by redirecting its output into wc’s input. A pipe allows to compose and assemble commands into pipelines, which makes the terminal so powerful." decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/process-pipe-stream.png">
<span class=imgcaption><code>ls</code> is <em>piped</em> into <code>wc</code> by redirecting its output into <code>wc</code>’s input. A pipe allows to compose and assemble commands into pipelines, which makes the terminal so powerful.</span></p>
<p>You can of course chain as many commands as possible and create command
pipelines.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>command1<span class="w"> </span><span class="k">|</span><span class="w"> </span>command2<span class="w"> </span><span class="k">|</span><span class="w"> </span>command3<span class="w"> </span><span class="k">|</span><span class="w"> </span>...<span class="w"> </span><span class="k">|</span><span class="w"> </span>commandN
</code></pre></div>
<div class="Note">
<p>When you execute <code>command1 | command2</code>, your shell starts <em>all</em> commands
at the same time, and a command’s output is streamed into the next one
as the commands run.</p>
</div>
<p>For example, let’s imagine I’d like to count the number of files in my
<code>Downloads</code> folder. To that effect, I can combine <code>ls</code> and the <code>wc</code> (for
<em>word count</em>) commands. <code>wc</code>, when used with the <code>-l</code> options, allows to
count the number of lines in its input.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-1<span class="w"> </span>~/Downloads<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">wc</span><span class="w"> </span>-l
<span class="m">34</span>
</code></pre></div>
<p>Now, let’s say I only want to count the number of pdf files in my
<code>Downloads</code> folder, not just all of them. No problem, <code>grep</code> to the
rescue! <code>grep</code> allows to filer its input on a given pattern (more on
<code>grep</code> in the next chapter). By using <code>grep pdf</code>, we filter the output
of <code>ls -1</code> to only the filenames containing “pdf”, and then count how
many filenames were filtered using <code>wc -l</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-1<span class="w"> </span>~/Downloads<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">grep</span><span class="w"> </span>pdf<span class="w"> </span><span class="k">|</span><span class="w"> </span><span class="nb">wc</span><span class="w"> </span>-l
<span class="m">22</span>
</code></pre></div>
<h3 id="going-further-redirecting-output-to-both-the-console-and-a-file">Going further: redirecting output to both the console and a file</h3>
<p>The <code>tee</code> command allows you to write a command’s <code>stdout</code> to a file
while still displaying it into the console. This can be very useful if
you want to store the output of a command in a file, but still be able
to see what it’s doing in real-time.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">ls</span><span class="w"> </span>-1<span class="w"> </span><span class="k">|</span><span class="w"> </span>tee<span class="w"> </span>output.txt
Android
code
...
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>output.txt
Android
code
...
</code></pre></div>
<p><img alt="tee is named after the T-splitter used in plumbing." decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/terminal/tee.png">
<span class=imgcaption><code>tee</code> is named after the T-splitter used in plumbing.</span></p>
<p><a id="escaping-from-bad-situations"></a></p>
<h2 id="escaping-from-bad-situations">Escaping from bad situations</h2>
<h3 id="mistyped-command-missing-arguments">Mistyped command, missing arguments</h3>
<p>If you mistype a command, or forget to add arguments, you can find
yourself in a situation where your shell hangs, and nothing happens. For
example, type any of the following commands.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="err">'</span>hello<span class="w"> </span>world
</code></pre></div>
<p>The first command hangs because it is waiting for input on its <code>stdin</code>
stream, as no argument file was provided. In the case of the second
command, it is missing a matching single quote. In both cases, you get
can out of this situation by hitting <kbd>Ctrl</kbd> - <kbd>C</kbd>
which kills the command by sending it a interruption signal.</p>
<div class="Note">
<p>If your shell is stuck on receiving input (like in the <code>cat</code> example),
you can also cleanly exit it by hitting <kbd>Ctrl</kbd> - <kbd>D</kbd>
which will send a special EOF (“end of file”) character, indicating to
the command that its input is now closed.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span>
hello
hello
world
world
<span class="c1"># Ctrl-D</span>
$
</code></pre></div>
</div>
<h3 id="escaping-characters">Escaping characters</h3>
<p>Imagine for a second that you had a file on disk named <code>my file</code>, and
you wanted to display its content using <code>cat</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>my<span class="w"> </span>file
cat:<span class="w"> </span>my:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
cat:<span class="w"> </span>file:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
</code></pre></div>
<p>In the previous example, the <code>cat</code> command was given 2 arguments <code>my</code>
and <code>file</code>, none of which corresponded to any existing file. We have 2
solutions to make this work: quoting the file name, or using an <span
class="gls" key="escapechar">escape character</span>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span><span class="s1">'my file'</span>
That<span class="w"> </span>file<span class="w"> </span>has<span class="w"> </span>spaces<span class="w"> </span><span class="k">in</span><span class="w"> </span>it...
$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span><span class="s2">"my file"</span>
That<span class="w"> </span>file<span class="w"> </span>has<span class="w"> </span>spaces<span class="w"> </span><span class="k">in</span><span class="w"> </span>it...
</code></pre></div>
<p>By putting quotes around the file name, you are telling your shell that
whatever is between the quotes is a single argument.</p>
<p>Like previously mentioned, we could also use the backslash escape
character, which indicates that the following character doesn’t
have any special meaning.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cat</span><span class="w"> </span>my<span class="se">\ </span>file
That<span class="w"> </span>file<span class="w"> </span>has<span class="w"> </span>spaces<span class="w"> </span><span class="k">in</span><span class="w"> </span>it...
</code></pre></div>
<p>By using <code>\</code> (a backslash character followed by a space), we indicate to
the shell that the space is simply a space, and should not be
interpreted as a separator between 2 arguments.</p>
<p><a id="summary"></a></p>
<h2 id="summary">Summary</h2>
<p>In that chapter, we’ve discovered what a terminal is: an application in
which you can type text commands to have them executed by a program
called a shell.</p>
<p>Facing the terminal can be intimidating at first because you might not
always know what command to type. Learning your way around the terminal
is however part of the journey of becoming a software engineer. Like any
other powerful tool, it can be hard to learn but will also make you
immensely more productive once you get more accustomed to it.</p>
<p>The fundamental philosophy of working in a terminal is being free to
compose different tools in a way that might not have been initially
foreseen by the tools’ developers, by using pipes and stream
redirections. Instead of using a single tool that was only designed to
perform a finite set of tasks, you are free to assemble a patchwork of
unrelated commands, that can all work together by joining their input
and output streams.</p>
<p>In the next chapter, we will dig into text processing commands, which
can be immensely powerful when chained together with pipes.</p>
<p><a id="going-further"></a></p>
<h2 id="going-further">Going further</h2>
<p><strong>1.1</strong>: Look into the <code>ls</code> manual and research what the <code>-a</code> option is
doing. Run <code>ls -a ~/</code>. What are the <code>.</code> and <code>..</code> directories? What are
the files starting with a <code>.</code> ?</p>
<p><strong>1.2</strong>: Run a command and redirect its output into a file, but display
any errors in the terminal.</p>
<p><strong>1.3</strong>: Run a command and redirect its output into a file, and any
errors into a different file.</p>
<p><strong>1.4</strong>: Run a command and redirect both its output and errors into the
same file, while also displaying them all on screen at the same time.</p>
<p><strong>1.5</strong>: Use a heredoc redirection to create a new file with text in it.</p>
<p><strong>1.6</strong>: Given an <code>echoes</code> file, what is the difference between
<code>wc -l echoes</code>, <code>cat echoes | wc -l</code> and <code>wc -l < echoes</code> ?</p>
<footer>
<p>
<em>Essential Tools and Practices for the Aspiring Software Developer</em> is a self-published book project by Balthazar Rouberol and <a href=https://etnbrd.com>Etienne Brodu</a>, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!
</p>
<p>The book will help you set up a productive development environment and get acquainted with tools and practices that, along with your programming languages of choice, will go a long way in helping you grow as a software developer.
It will cover subjects such as mastering the terminal, configuring and getting productive in a shell, the basics of code versioning with <code>git</code>, SQL basics, tools such as <code>Make</code>, <code>jq</code> and regular expressions, networking basics as well as software engineering and collaboration best practices.
</p>
<p>
If you are interested in the project, we invite you to join the <a href=https://balthazar-rouberol.us4.list-manage.com/subscribe?u=1f6080d496af07a836270ff1d&id=81ebd36adb>mailing list</a>!
</p>
</footer>How to setup a personal wireguard VPN2019-12-11T00:00:00+01:002019-12-11T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2019-12-11:/how-to-setup-a-personal-wireguard-vpn<p>This article will provide guidance about how to setup a Wireguard VPN between a server and your phone, allowing you to avoid being snooped on while you travel.</p><p>My work takes me to the United-States multiple times a year, and I've never been comfortable using the hotel Wi-Fi, or even my company VPN for that matter, when I'm there. I want to be assured that what I do online is my business and my business alone.</p>
<p>I had heard about <a href="https://wireguard.com">Wireguard</a> multiple times, how performant and simple it was compared to OpenVPN (I'd like to have a talk with whomever came up with the OpenVPN config file...). I decided to jump in and give it a try. The idea was to setup a VPN access point on my VPS, hosted in Paris, to which I could connect when I travel.</p>
<hr>
<h2 id="installing-wireguard">Installing wireguard</h2>
<p>I followed Wireguard's <a href="https://www.wireguard.com/install">official install instructions</a>. However, I also needed to install the headers files for the kernel I was running so that <code>dkms</code> could compile the <code>wiregard</code> kernel module.</p>
<div class="highlight"><pre><span></span><code><span class="gp">% </span>apt-get<span class="w"> </span>install<span class="w"> </span>linux-headers-<span class="k">$(</span>uname<span class="w"> </span>-r<span class="k">)</span>
<span class="gp">% </span>add-apt-repository<span class="w"> </span>ppa:wireguard/wireguard
<span class="gp">% </span>apt-get<span class="w"> </span>update
<span class="gp">% </span>apt-get<span class="w"> </span>install<span class="w"> </span>wireguard
</code></pre></div>
<p>If everything is going according to plan, you should see the <code>wireguard</code> kernel module being compiled by <code>dkms</code> at install time:</p>
<div class="highlight"><pre><span></span><code><span class="go">...</span>
<span class="go">DKMS: build completed.wireguard.ko:</span>
<span class="go">Running module version sanity check.</span>
<span class="go"> - Original module</span>
<span class="go"> - No original module exists within this kernel</span>
<span class="go"> - Installation</span>
<span class="go"> - Installing to /lib/modules/X.Y.Z-ABC-generic/updates/dkms/</span>
<span class="go">...</span>
</code></pre></div>
<p>At that point, you should be able to see the module in the <code>lsmod</code> output and load it.</p>
<div class="highlight"><pre><span></span><code><span class="gp">% </span>lsmod<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>wireguard
<span class="go">wireguard 204800 0</span>
<span class="go">ip6_udp_tunnel 16384 1 wireguard</span>
<span class="go">udp_tunnel 16384 1 wireguard</span>
<span class="gp">% </span>modprobe<span class="w"> </span>wireguard
</code></pre></div>
<h2 id="configuring-the-server-peer">Configuring the server peer</h2>
<p>First off, we create the server <code>wireguard</code> peer's public and private keys.</p>
<div class="highlight"><pre><span></span><code><span class="gp">% </span><span class="nb">cd</span><span class="w"> </span>/etc/wireguard
<span class="gp">% </span><span class="nb">umask</span><span class="w"> </span><span class="m">077</span><span class="w"> </span><span class="c1"># disable public access</span>
<span class="gp">% </span>wg<span class="w"> </span>genkey<span class="w"> </span><span class="p">|</span><span class="w"> </span>tee<span class="w"> </span>privatekey<span class="w"> </span><span class="p">|</span><span class="w"> </span>wg<span class="w"> </span>pubkey<span class="w"> </span>><span class="w"> </span>publickey
</code></pre></div>
<p>We now configure the server peer, assuming that the VPS public network interface is <code>ens2</code>. We'll use the <code>192.168.2.0/24</code> subnet for all <code>wireguard</code>-related addresses, and assign <code>192.168.2.1</code> IP to the server peer.</p>
<div class="highlight"><pre><span></span><code><span class="gp">% </span>cat<span class="w"> </span><<EOF<span class="w"> </span>><span class="w"> </span>/etc/wireguard/wg0.conf
<span class="go">[Interface]</span>
<span class="gp"># </span>The<span class="w"> </span>IP<span class="w"> </span>assigned<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>wg0<span class="w"> </span>interface
<span class="go">Address = 192.168.2.1/24</span>
<span class="gp"># </span>The<span class="w"> </span>port<span class="w"> </span>wireguard<span class="w"> </span>will<span class="w"> </span>listen<span class="w"> </span>on
<span class="go">ListenPort = <public port></span>
<span class="gp"># </span>The<span class="w"> </span>private<span class="w"> </span>key<span class="w"> </span>used<span class="w"> </span>by<span class="w"> </span>the<span class="w"> </span><span class="nb">local</span><span class="w"> </span>peer
<span class="go">PrivateKey = $(cat /etc/wireguard/privatekey)</span>
<span class="gp"># </span>Accept<span class="w"> </span>traffic<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>wg0<span class="w"> </span>interface<span class="w"> </span>and<span class="w"> </span>allow<span class="w"> </span>NATing<span class="w"> </span>traffic<span class="w"> </span>from<span class="w"> </span>ens2<span class="w"> </span>to<span class="w"> </span>wg0
<span class="go">PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o ens2 -j MASQUERADE</span>
<span class="go">PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o ens2 -j MASQUERADE</span>
<span class="go">EOF</span>
<span class="gp">% </span>rm<span class="w"> </span>/etc/wireguard/privatekey
</code></pre></div>
<p>We also need to authorize UDP traffic on the <code><public port></code> port.</p>
<div class="highlight"><pre><span></span><code><span class="gp">% </span>iptables<span class="w"> </span>-i<span class="w"> </span>ens2<span class="w"> </span>-p<span class="w"> </span>udp<span class="w"> </span>--dport<span class="w"> </span><public<span class="w"> </span>port><span class="w"> </span>-j<span class="w"> </span>ACCEPT
</code></pre></div>
<p>Once that's done, we're now able to use <code>wg-quick</code> to setup the <code>wg0</code> network interface, as well as the <code>MASQUERADE</code> iptables rules that will NAT the traffic between the public <code>ens2</code> interface to <code>wg0</code>. We can actually use systemd for that, as we're assured that the <code>wg0</code> interface is re-created in case of a reboot.</p>
<div class="highlight"><pre><span></span><code><span class="gp">% </span>systemctl<span class="w"> </span>start<span class="w"> </span>wg-quick@wg0
<span class="go">[#] ip link add wg0 type wireguard</span>
<span class="go">[#] wg setconf wg0 /dev/fd/63</span>
<span class="go">[#] ip -4 address add 192.168.2.1/24 dev wg0</span>
<span class="go">[#] ip link set mtu 1420 up dev wg0</span>
<span class="go">[#] iptables -A FORWARD -i wg0 -j ACCEPT; iptables -A FORWARD -o wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens2 -j MASQUERADE</span>
<span class="gp">% </span>systemctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>wg-quick@wg0
<span class="go">Created symlink from /etc/systemd/system/multi-user.target.wants/wg-quick@wg0.service to /lib/systemd/system/wg-quick@.service.</span>
</code></pre></div>
<h2 id="configuring-the-phone-peer">Configuring the phone peer</h2>
<p>I use the Wireguard <a href="https://play.google.com/store/apps/details?id=com.wireguard.android">Android app</a>, and assign the <code>192.168.2.2/32</code> address to my phone, as well as add the server peer details (as Wireguard is a point-to-point VPN without a client/server architecture).</p>
<p>The server peer public key is set to the content of the remote <code>/etc/wireguard/publickey</code> file, on my VPS. As I want to route all my phone traffic through <code>wireguard</code>, I set the <code>Allowed IPs</code> field to <code>0.0.0.0/0</code>, and the peer endpoint to <code><server public ens2 IP>:<public port></code>.</p>
<p><img alt="screenshot" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/wireguard/android-wg.jpg"></p>
<h2 id="authorizing-the-phone-peer">Authorizing the phone peer</h2>
<p>After having generated a public key for the phone peer, we also need to authorize it on the server peer and restart <code>wireguard</code>.</p>
<div class="highlight"><pre><span></span><code><span class="gp">% </span>cat<span class="w"> </span><<EOF<span class="w"> </span>>><span class="w"> </span>/etc/wireguard/wg0.conf
<span class="go">[Peer]</span>
<span class="gp"># </span>Phone<span class="w"> </span>peer
<span class="go">PublicKey = <phone peer public key generated in app></span>
<span class="go">AllowedIPs = 192.168.2.2/32</span>
<span class="go">EOF</span>
<span class="gp">% </span>systemctl<span class="w"> </span>restart<span class="w"> </span>wg-quick@wg0
</code></pre></div>
<h2 id="testing-the-whole-thing">Testing the whole thing</h2>
<p>My phone disconnected from the server <code>wireguard</code> peer, I'm now able to inspect the state of the <code>wg0</code> server network interface:</p>
<div class="highlight"><pre><span></span><code><span class="gp">% </span>ifconfig<span class="w"> </span>wg0
<span class="go">wg0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00</span>
<span class="go"> inet addr:192.168.2.1 P-t-P:192.168.2.1 Mask:255.255.255.0</span>
<span class="go"> UP POINTOPOINT RUNNING NOARP MTU:1420 Metric:1</span>
<span class="go"> RX packets:0 errors:0 dropped:0 overruns:0 frame:0</span>
<span class="go"> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0</span>
<span class="go"> collisions:0 txqueuelen:1</span>
<span class="go"> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)</span>
</code></pre></div>
<p>I then connect my phone to the server peer, open a random webpage, and <em>voila</em>, we can see traffic going through the server <code>wg0</code> interface.</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>ifconfig<span class="w"> </span>wg0
<span class="go">wg0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00</span>
<span class="go"> inet addr:192.168.2.1 P-t-P:192.168.2.1 Mask:255.255.255.0</span>
<span class="go"> UP POINTOPOINT RUNNING NOARP MTU:1420 Metric:1</span>
<span class="go"> RX packets:4084 errors:0 dropped:132 overruns:0 frame:0</span>
<span class="go"> TX packets:4895 errors:0 dropped:0 overruns:0 carrier:0</span>
<span class="go"> collisions:0 txqueuelen:1</span>
<span class="go"> RX bytes:452436 (452.4 KB) TX bytes:2954188 (2.9 MB)</span>
</code></pre></div>
<p>A quick <code>tcpdump</code> shows that the data flowing to <code>wg0</code> is indeed encrypted.</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>tcpdump<span class="w"> </span>-i<span class="w"> </span>wg0<span class="w"> </span>-vv<span class="w"> </span>-c<span class="w"> </span><span class="m">100</span><span class="w"> </span>-X
<span class="go">tcpdump: listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes</span>
<span class="go">15:14:05.096356 IP (tos 0x0, ttl 105, id 47301, offset 0, flags [none], proto TCP (6), length 332)</span>
<span class="go"> wq-in-f188.1e100.net.5228 > 192.168.2.2.46641: Flags [P.], cksum 0x8308 (correct), seq 1867855144:1867855424, ack 229885280, win 253, options [nop,nop,TS val 426814177 ecr 2017832], length 280</span>
<span class="go"> 0x0000: 4500 014c b8c5 0000 6906 fe02 4a7d 8cbc E..L....i...J}..</span>
<span class="go"> 0x0010: c0a8 0202 146c b631 6f55 3528 0db3 c560 .....l.1oU5(...`</span>
<span class="go"> 0x0020: 8018 00fd 8308 0000 0101 080a 1970 aae1 .............p..</span>
<span class="go"> 0x0030: 001e ca28 1703 0301 13e7 c1f4 5089 ed04 ...(........P...</span>
<span class="go"> 0x0040: aba6 ef67 2cbe a7b3 f0cc 02d0 caaa d675 ...g,..........u</span>
<span class="go">...</span>
</code></pre></div>
<p>I now have have a personal VPN I can use whenever I travel abroad.</p>
<hr>
<p>Thanks to Thomas for being patient with me while answering networking questions at 11pm, and for proof-reading this article. Any remaining mistake is my own.</p>My DIY proposal2019-11-30T00:00:00+01:002019-11-30T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2019-11-30:/my-diy-proposal<p><img alt="My DIY chest" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/chest-3d.png"> I proposed to my girlfriend by building her a Zelda chest and developing an Android application for the occasion.</p><p>You know how everyone wants their proposal to be special, thoughtful, original, and above all, wants to avoid being cliché? Well, I wanted all that. I also wanted it to be DIY and geeky. With all that in mind, and because my SO is such a Zelda fan, I decided to propose to her by having her open an Ocarina of time themed treasure chest, which would light up from the inside and play the famous music when it opens.</p>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/69AyYUJUBTg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
<p>I started googling for instructions for a DIY Zelda treasure chest and found this perfect <a href="https://www.instructables.com/id/Legend-of-Zelda-Treasure-chest-with-sound/">tutorial</a>. Now, being perfectly honest, I have to admit that I'm not a very gifted craftsman. The idea of making a chest myself from wood got me a little worried, as I felt that I really needed to be precise and prepared, whereas I tend to lean to the more yolo side of things.</p>
<p>To that end, I started by creating a <a href="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/Chest.FCStd">3D model</a> of the chest itself, using <a href="https://www.freecadweb.org/">FreeCAD</a>, which I had to learn from scratch, by using the dimensions (in imperial freedom units) from the <a href="https://www.instructables.com/id/Legend-of-Zelda-Treasure-chest-with-sound/">instructables tutorial</a>.</p>
<p><img alt="3dModel" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/chest-3d.png"></p>
<p>I finally ended up with something looking pretty good, that I could export to a <a href="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/chest.pdf">plan</a> with precise dimensions reported in mm.</p>
<p><img alt="chest-plan" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/chest-plan.png"></p>
<p>The easy part was now done, and I needed to get to the actual building phase. I found a <a href="https://fabmanager.astech-fablab.fr">fablab</a> in my area that looked pretty nice, but I couldn't find enough time to sneak around there and actually get to it. Time passed, and I one day noticed a big stack of cardboard lying around in our flat. I finally decided to make the chest out of cardboard instead, as it'd be easier to work and iterate with.</p>
<p>I followed the plan as best as I could (and improvised a fair amount), and ended up with something looking quite good!</p>
<p><img alt="chest4" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/chest-4.jpeg">
<img alt="chest2" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/chest-2.jpeg">
<img alt="chest3" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/chest-3.jpeg"></p>
<p>At that point, I had a nice looking chest, but I also wanted music and light to beam out of it when it was being opened. I investigated a setup based on an Arduino with external speakers, a lever switch and an external LED for a while, but I soon realised that my phone was all I needed to make it work! All I needed to do was write an app that would play an mp3 and light up the phone's LED when it detected an ambient brightness increase. As the brightness sensor needed to face upwards, that'd mean that the LED would face downwards and beam the light towards the bottom of the chest. I decided to put a small mirror in the chest, and go with that.</p>
<p>I taught myself <a href="https://kotlinlang.org/">Kotlin</a> and Android development on <a href="https://udemy.com">Udemy</a> (thanks to Datadog for providing employees with an account!), by following <a href="https://datadog.udemy.com/course/devslopes-android-kotlin/learn/lecture/7866294">Kotlin for Android: Beginner to Advanced</a>, by <a href="https://www.youtube.com/channel/UClLXKYEEM8OBBx85DOa6-cg/featured">Devslopes</a>, which I can't recommend enough. I ended up with that small <a href="https://github.com/brouberol/OpenChest">application</a> installed on my phone.</p>
<p>One issue that I encountered was that Android does not give you any API to ask the brightness sensor for the <em>current</em> brightness value. All you can do is be notified when that value changes.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// Event handler executed when the light sensor detects a change</span>
<span class="kd">override</span><span class="w"> </span><span class="kd">fun</span><span class="w"> </span><span class="nf">onSensorChanged</span><span class="p">(</span><span class="n">event</span><span class="p">:</span><span class="w"> </span><span class="n">SensorEvent</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">lux</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">event</span><span class="p">.</span><span class="na">values</span><span class="o">[</span><span class="m">0</span><span class="o">]</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="s">"Lux: </span><span class="si">${</span><span class="n">lux</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">lux</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="n">activationLux</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="o">!</span><span class="n">mPlayer</span><span class="p">.</span><span class="na">isPlaying</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">turnLightsOn</span><span class="p">()</span>
<span class="w"> </span><span class="n">playSound</span><span class="p">()</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>This is unwieldy as all the app can do is detect if the ambient brightness crosses an absolute threshold, which itself depends on the time of the day, the ambient light of the room, etc. I investigate whether I could light the LED up for a couple of seconds, then turn it off, to force the sensor to pick up some changes, to approximate the current brightness inside the chest. In the end, it was easier to just cover the inside with black foam to make sure the phone was in pitch black darkness.</p>
<p>I was <em>finally</em> all set.</p>
<div class="vid-container">
<video class="video" controls>
<source
src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/chest-opening.webm"
type="video/webm">
<source
src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/proposal/chest-opening.mp4"
type="video/mp4">
</video>
</div>
<p>The ring was a ruby, obviously.</p>
<p>Oh and she said yes!</p>On letting go2019-11-11T00:00:00+01:002019-11-11T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2019-11-11:/on-letting-go<p>I have been feeling more and more burdened in the recent months. It first wasn't clear to me why I was feeling that way: I'm in the most happy and fulfilling relationship I've ever been in, we just moved to a beautiful apartment we both fell in love with, I'm …</p><p>I have been feeling more and more burdened in the recent months. It first wasn't clear to me why I was feeling that way: I'm in the most happy and fulfilling relationship I've ever been in, we just moved to a beautiful apartment we both fell in love with, I'm honoured and lucky to have great friends, and a fantastic job I'm enjoying myself in (which also turns out to pay greatly), alongside some of the greatest engineers out there. Why on Earth was I therefore sometimes feeling like all I felt like doing was sit on the couch, turn on Netflix, and wait for the day to end?</p>
<p>I started to get a better understanding of what was happening when my girlfriend confronted me on the subject of money, which turned out to be my greatest insecurity. At the time, we were planning for a trip to Canada mid-October, during which we were going camping in one of the national parks. Having almost no camping gear, we borrowed some and went out to buy the rest. What should have been an uneventful event turned into a fight, due to the anxiety that increasing pile of items in our cart was somehow causing me. Almost all I could think about was how much that gear was going to cost, and the space they would take at home when we got back. When we finally got to the point of choosing sleeping bags (something that you don't want to be cheap about when sleeping in the cold Canadian wilderness), I'd become so anxious that I was almost incapable of adding them to the cart and snapped back when my partner asked me why I was hesitating, given that I could afford them.</p>
<p>That whole conversation felt like <em>dejà vu</em>, as we went through the exact same process when planning for the renovation of our kitchen and bathroom, in the new apartment. During the process of choosing what we liked and imagining what we'd want, I couldn't mute that little voice in my head keeping track of what it would cost.</p>
<p>Why was I anxious at the idea of buying something that I needed or wanted, that would bring me comfort, when I could afford it in the first place?</p>
<p>As a side note, I don't think that I'm a cheap person. I have no problem buying someone else a present, lending or giving money, but when it comes to <em>me</em>, I can behave like Balthazar Scrooge (how appropriate).</p>
<p>It finally clicked as I was looking for old pictures and un-used apps to delete from my phone. I was doing that to maintain it as clutter-free as possible, to free myself from the weight of the things I wasn't using. It felt <em>good</em> to get rid of that dead digital weight, to make space for myself. And as I was doing so, I finally realised that the anguish didn't come from <em>spending</em> the money, but from the fact that buying new things for home would thus decrease my living space. These objects would then add to the clutter I was trying to clear from my phone, and weight me down.</p>
<p>I needed to apply to the physical life what my digital self instinctively knew: that letting things go is okay.</p>
<p>By choosing to only surround myself with what I love and find beautiful, I can create an environment that lifts me up and "recharges my batteries", so to speak. Anything not actively contributing to a shared sense of aesthetics or not making me feel good can be given away (as long as my partner and I agree). I now realise that <em>less is more</em> is key to my happiness and peace of mind. Keep what you love, enjoy what you have, give away the rest.</p>
<p>I should probably avoid starting a new project and practice what I already know I love but haven't worked on for a while.</p>
<p><img alt="How long has bit been since I last baked?" decoding="async" loading="lazy" src="https://lh3.googleusercontent.com/60efHOnhKHQ6Bzjlh8a0wK4Icf4Rdiq_t37i0Gb7xKQ8RQNbfW7RjQJOnQ3L_gx_Q2Vg39eeyPyMZ8jswF4sFfKiNg9teqPUKICH4H6VYXVKOM3IMPblX_F970AgVlXNm7FWsSCFdvKRF9jXUxG810UdWYx3EWXHbbGJ6SWclsTuFQxcdl1-HZa2i1Tp4UxB8AeHkWLpf6Xg3d4bKoV3P7b5aRK3-mZiNB4ZJkVx3f56pD4bQwZFnAH82vj_WOTu6RfSvis9gC9DBccPvbUy7hMMYXHVjqcudVMKQW8Q9S13QF_GyjGGDTKqSRuDwmpwA_AAxj8iuHyvjZwpyDYYp2GQ6XtVsVHukfIuA9afPGXwuHX2gIA7bbc4Xg8NAZOx1_tI0D2ERQcvDBRFRi5qNCcXxZYGvPtBYcOpMBeO911oxTwbwQgx6xwFZoh1ZHqi3tBmRDZpCWnOGoOi-bFGh_zAZ0FOx6SyY-r16pSt3Psh31zKW5u0csTaoO9qWgPjT6v5AvThTzI1dP989vaLhYvpVNgetkk4OeXI1oOjEAkweLuc_iabn99iD9mSeI6pK5Lwj6chBy7Vv6lBHjStEmv4H2HstAIdZQrSnb6YQL0dcHjW_L2N4B9E6K6tnoZkQzjTivPAuqpVZzQSaEsnnEB-mdjYA7VdvdsJCr3lw6mNTGFS_xAnXrQ4D5258TrMFeAuX1zBkqDkVSdNyT3mPeRmYqcaSQYAjQwbXv6ERnUmPH9N=w1758-h989-no"></p>
<p>In <a href="http://www.calnewport.com/books/digital-minimalism/">Digital Minimalism</a>, Cal Newport points out that we oftentimes find ourselves installing a new digital tool (whether it's a new app, a new social media account, ...) without actively considering the benefit we're getting out of it. By doing so, we're inviting new notifications and distractions into our lives, which will eat away at our personal time and attention. As time and space are related under the Einsteinien theory of relativity, maybe our personal time and space are related as well? They both should be protected as much as possible from whatever is unaesthetic, distracting and that which weighs us down.</p>
<p><img alt="photos" decoding="async" loading="lazy" src="https://lh3.googleusercontent.com/9D0eDX_i-m3lILpmMHqpQRJugCGu2MvF3m2063c8CBLzbdAUkN2kcyR-9pbWLjetHJ0civXr8Xgo0Lqsg7SpehZ6g0vOuYqIm_OCtDapqpVauUf2UZY8HBO_0jWzZZ5yeCso7dm7dEw_plCLuDiB5vdjb12hWFeniPx3LRKr8MY_gQTUnN3ZqjP8Jnpy971mWAGcT11WnCIVSwY-9J2bKmuc1gQGbrNFvLll5PIVLPKE2VUdNDpZYd_qT8LDbeZmhseqTX_NCiLIbZ4OpqXLQczKyxZINMoUWdpsb-sLKDpMkzJIA1outFjxRE0xynsXnI6dU2lO9YOpgX-LhohYoXpf804zy3KqmnaO1GAu2hDJ_V87sY02IyT3oMLNLLScol6RkTRf4nvqHuKRieL6lsQQr6GvkSjqHqLJr6br5WKFkN_uLO54VVexjKbe27mf59gBm-dTb5BcVdwELR4at8JYSrTxQAcEdJPq4Tw5RuxnY6jkr52rw8aUHpaOXimTgfmhNWMUdxYu1sOe-KonNPkyyb0UTgncmDjW-jVb99h1weefa9DqDvu8Vh56x9C74PdMd2OKmT-8Csdzu9QRIUqZwMuYgqbeLbbHGopBmdP79mgDdizguc9q0NQDgnQJVjs29XLCmYCFr3agevqD5izwRcQMrnTHnzHs7M9UsHw6K1p6IdDV9d8eRPtwn43GHsOUG50gQbWwayRb-1gWZyD3W5x1ZV19dwTd5gjOnjla8Elp=w596-h1059-no"></p>
<p>I've thus cleared out my Reddit and Twitter accounts from everything work-related (and almost abandoned Twitter altogether), have deactivated almost all phone notifications (except for my partner's messages), and am in the process of applying the same principles in the physical world.</p>
<p>Keep what you love, enjoy what you have, give away the rest.</p>
<p>It feels fantastic.</p>Managing my infra like it's 20192019-07-22T00:00:00+02:002019-07-22T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2019-07-22:/managing-my-infra-like-its-2019<p>I recently realized that I was routinely managing thousands of servers and petabytes of data in my daily job, but was still managing my own personal infrastructure like I was living in 1999.</p>
<p><img alt="my-infra" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/managing-infra/infra.png"></p>
<hr>
<p>With the advent of configuration management tools such as <a href="https://docs.ansible.com/">Ansible</a>, <a href="https://www.chef.io/">Chef</a>, and the like, it became easier …</p><p>I recently realized that I was routinely managing thousands of servers and petabytes of data in my daily job, but was still managing my own personal infrastructure like I was living in 1999.</p>
<p><img alt="my-infra" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/managing-infra/infra.png"></p>
<hr>
<p>With the advent of configuration management tools such as <a href="https://docs.ansible.com/">Ansible</a>, <a href="https://www.chef.io/">Chef</a>, and the like, it became easier to configure instances in a reproducible manner by defining said configuration as code. <a href="http://terraform.io/">Terraform</a> made it easier to codify and provision cloud resources: instances, but also security groups, permissions, storage, load balancers, etc.</p>
<p>It's easy to simply think of a cloud infrastructure as a pool of compute resource. It is however often so much more than that. When executed right, The Cloud is a set of meshed services, interacting and communicating with each other (possibly with compute resources sitting in the middle). That applies for vast and complex infrastructures such as the one I work on at <a href="https://datadoghq.com">Datadog</a>, but it also applies to my ridiculously tiny personal one. Realizing this got me thinking. Why wasn't I using the same tools and techniques to manage my small infrastructure than the ones I'm using daily?</p>
<h2 id="my-infrastructure">My infrastructure</h2>
<p>My personal infrastructure consists of (drumrolls...) 3 servers:</p>
<ul>
<li>a VPS running in Scaleway, hosting my personal services (personal website, blog, git repositories, <a href="https://radicale.org/documentation/">CalDAV server</a>, <a href="https://usefathom.com/">traffic analytics</a>, <a href="https://thelounge.chat/">IRC client</a>, <a href="https://www.wallabag.org/en">Read-it-later service</a>, etc)</li>
<li>a VPS running in OVH, hosting my mother's website</li>
<li>a Raspberry Pi, running in my living room, hosting private services (<a href="https://kresus.org/en/index.html">Kresus</a>)</li>
</ul>
<p>Until now, each of these servers were managed in an <em>ad-hoc</em> fashion, sometimes with scripts, sometimes without. All the cloud resources on which my services (S3 buckets, DNS zones, etc) were managed manually, using the cloud provider web console.</p>
<p>I manage my DNS zones with OVH, I use the AWS S3 bucket free tier for the blog images, and Datadog for monitoring.</p>
<p><img alt="ssl-expiry-monitoring" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/managing-infra/datadog-monitors.png"></p>
<h2 id="improving-the-setup">Improving the setup</h2>
<p>I had several objectives in mind to improve the current setup:</p>
<ul>
<li>define all instances configuration and state in <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks.html">ansible playbooks</a></li>
<li>re-use and share instances configuration by leveraging <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html">ansible roles</a></li>
<li>define and manage all cloud resources using <a href="https://terraform.io">terraform</a> to never have to log into a cloud web console again</li>
<li>secure all web-services with an automatically renewed SSL certificate provided by Let's Encrypt</li>
<li>run all services behind a reverse-proxy, using a docker container or a <a href="https://www.brendanlong.com/systemd-user-services-are-amazing.html">userland systemd service</a> with minimal permissions and privileges</li>
<li>monitor the hosts and services using <a href="https://datadoghq.com">Datadog</a> (free for 5 hosts or less) , with monitors define in terraform</li>
<li>secure the SSH connections of the internet-facing hosts via <a href="https://duo.com/">Duo</a> (free for 10 users or less)</li>
<li>be able to SSH into all hosts from my personal and work laptop, as well as from my <a href="https://play.google.com/store/apps/details?id=org.connectbot&hl=en_US">phone</a></li>
<li>monitor my daily backups</li>
</ul>
<h2 id="show-me-the-code">Show me the code</h2>
<p>You can have a look at the code <a href="https://github.com/brouberol/infrastructure">here</a>. I've purposefully omitted the <code>terraform/global_vars/main.tf</code> file, credentials are obviously encrypted, API keys are defined in my home directory, but everything else is readable openly. My hope is that that readers might either learn something or point out where I'm doing something silly or insecure.</p>
<h2 id="what-now">What now?</h2>
<p>I'm now confident that I can open some of these services to friends, if they want to. I measure and monitor my own SLIs, the expiry of the SSL certificates, and can intervene from anywhere if something breaks.</p>
<p><img alt="ssl-expiry-monitoring" decoding="async" loading="lazy" src="https://balthazar-rouberol-blog.s3.eu-west-3.amazonaws.com/managing-infra/ssl-expiry-monitoring.png"></p>
<p>My infrastructure is now more secure, and has been audited by fellow peers <sup id="fnref:review"><a class="footnote-ref" href="#fn:review">1</a></sup>. I'm now confident I can restore the services in the face of an instance loss (which is very important for my mother, as her website has a fair amount of traffic and brings her regular new customers).</p>
<p>I'm also dogfooding Datadog features, which got to me suggest a couple of improvements to the Datadog <a href="https://www.terraform.io/docs/providers/datadog/index.html">terraform provider</a> which will be worked on next quarter.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:review">
<p>Thanks to Mehdi and Thomas for the thorough playbook review. Any remaining mistake or silliness is my own. <a class="footnote-backref" href="#fnref:review" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>My no-knead bread recipe2019-05-18T00:00:00+02:002019-05-18T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2019-05-18:/my-no-knead-bread-recipe-english.html<p><img alt="bread" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0032.JPG"> I've started baking again, and I think I'm really getting nice results. On top of it, they are reproducible! In that post, I'll walk you through my favorite recipe step-by-step.</p><p>I've started baking again, and I think I'm really getting nice results. On top of it, they are reproducible! In that post, I'll walk you through my favorite recipe step-by-step.</p>
<p>I'm baking a 65% hydrated loaf using a no-knead technique, cooked in a cooking pot.</p>
<h2 id="ingredients">Ingredients</h2>
<p>As the <a href="https://www.amazon.fr/Flour-Water-Salt-Yeast-Fundamentals/dp/160774273X">FWSY book</a> said, a regular contains 4 ingredients: flour, water, salt and yeast.</p>
<p>My recipe contains:</p>
<ul>
<li>500g T65 flour</li>
<li>325mL of lukewarm water (the hydration percentage is calculated based on the flour mass, so 500 * 0.65 = 325g)</li>
<li>9g of coarse sea salt (the french yeast syndicate <a href="https://www.chambresyndicalelevure.com/18-g-de-sel/">has settled</a> on 18g of salt per kg of flour, so who am I so say otherwise?)</li>
<li>8g of fresh baker's yeast</li>
</ul>
<h2 id="initial-mix">Initial mix</h2>
<p>First, mix the flour, the salt, and half of the water in a bowl, and the fresh baker's yeast with the other half of the water. Wait for a good half-hour.</p>
<p><img alt="mix" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0014.JPG"></p>
<p>Once the yeast is active and is well mixed into the water, incorporate it into the bowl and mix.</p>
<p><img alt="mix2" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0033.JPG"></p>
<h2 id="stretch-and-folds">Stretch and folds</h2>
<p>As this is a no-knead recipe, we'll use the <a href="https://www.youtube.com/watch?v=3_5x70h14VQ">stretch and fold</a> technique to <a href="https://cooking.stackexchange.com/questions/21675/why-stretch-and-fold-vs-traditional-kneading-of-bread-dough">stretch and reinforce the gluten strands</a>, which will then make sure the dough is elastic and can expand and rise nicely at cooking time.</p>
<p><strong>Note</strong>: the dough will stick to the fingers during the first 2 stretch-and-folds. It's ok. Try to refrain from adding too much flour.</p>
<p>First, flour the table a little, and get the dough out of the bowl (preferably using a <a href="https://www.amazon.fr/Buyer-4858-00N-Raclette-Corne-Blanche/dp/B000ECUDVK/ref=sr_1_4?__mk_fr_FR=%C3%85M%C3%85%C5%BD%C3%95%C3%91&crid=3NQH6UB4FQIMI&keywords=corne+de+boulanger&qid=1558194084&s=gateway&sprefix=corne+de+boul%2Caps%2C219&sr=8-4">scraper</a>, to avoid tearing the dough while you get it out). Stretch it slowly and as much as possible. The first times, it's possible some holes will form. If so, try to patch them and don't overstretch.</p>
<p><img alt="stretched" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0017.JPG"></p>
<p>Then, fold the dough <a href="https://youtu.be/j0o4asEGW78?t=42">4 times on itself</a> and shape the dough into a boule. Let it rest in a bowl, under a slightly damp towel for between a half-hour an an hour. In my experience, the more you wait, the more the yeast will activate and the more bubbles you'll get in the end.</p>
<p>After that waiting period, the dough should have expanded a bit, and feel more elastic, as well as less sticky.</p>
<p><img alt="after waiting" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0019.JPG"></p>
<p>Repeat these steps 4 to 5 times until the dough passes the <a href="https://www.youtube.com/watch?v=jK0jN6xaf1o">finger dent test</a>. The more you wait, the better.</p>
<p><img alt="stretch3" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0024.JPG">
<span class="image-caption">The dough after 3 stretch and folds</span></p>
<p><img alt="stretch4" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0026.JPG">
<span class="image-caption">The dough after 4 stretch and folds. Look at that puffy boi!</span></p>
<p><img alt="stretch5" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0028.JPG">
<span class="image-caption">The dough after 5 stretch and folds</span></p>
<h2 id="proofing">Proofing</h2>
<p>Shape the dough into a boule, and let it proof for 2 hours under a slightly damp towel. Go watch Netflix or something. After the 2 hours, shape it into a boule again, to re-tighten the dough.</p>
<p><img alt="post-proof" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0029.JPG">
<span class="image-caption">The dough after 2 hours of proofing and a tightening</span></p>
<h2 id="pre-heating-and-scoring">Pre-heating and scoring</h2>
<p>Pre-heat your oven at 250°C (482°F) with the cooking pot (lid included) inside. Once the oven is hot enough, place the proofed dough on cooking paper, flour it, then <a href="https://www.machineapain.org/comment-bien-faire-le-grignage-de-votre-pain/">score it</a>.</p>
<p><img alt="post-flouring" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0030.JPG">
<span class="image-caption">The floured dough</span></p>
<p><img alt="post-scoring" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0031.JPG">
<span class="image-caption">A deep scoring pattern allows the gas to dissipate. Shallow ones are just for show</span></p>
<h2 id="cooking">Cooking</h2>
<p>Get the cooking pot from the oven, and place the loaf inside, still on the cooking paper.
Let it cook lid closed for 30 minutes, to make sure the water contained in the bread evaporates in the pot, which will help the crust develop. Remove the lid and lower the oven temperature to 235°C (455°F). Get the bread out of the oven after 15 to 20 minutes, when you feel it's cooked enough and you like the color.</p>
<h2 id="resting">Resting</h2>
<p>Place your hot loaf on a grille, and let it rest and cool down for a couple of hours. Enjoy the cracking sounds.</p>
<p><img alt="all done" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0032.JPG">
<span class="image-caption">All done!</span></p>
<h2 id="eating">Eating</h2>
<p>You know what to do.</p>
<p><img alt="crumb shot" decoding="async" loading="lazy" src="https://s3.eu-west-3.amazonaws.com/balthazar-rouberol-blog/no-knead-bread-recipe/DSC_0015.JPG">
<span class="image-caption">Final crumb shot</span></p>Allocating unbounded resources to a kubernetes pod2018-09-29T00:00:00+02:002018-09-29T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2018-09-29:/allocating-unbounded-resources-to-a-kubernetes-pod<p>Note: this article assumes that the reader is familiar with <a href="https://kubernetes.io">Kubernetes</a> and Linux <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01">cgroups</a>.</p>
<hr>
<p>When deploying a pod in a Kubernetes cluster, you normally have 2 choices when it comes to <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container">resources</a> allotment:</p>
<ul>
<li>defining CPU/memory resource requests and limits <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container">at the pod level</a></li>
<li>defining default CPU/memory requests and …</li></ul><p>Note: this article assumes that the reader is familiar with <a href="https://kubernetes.io">Kubernetes</a> and Linux <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01">cgroups</a>.</p>
<hr>
<p>When deploying a pod in a Kubernetes cluster, you normally have 2 choices when it comes to <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container">resources</a> allotment:</p>
<ul>
<li>defining CPU/memory resource requests and limits <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container">at the pod level</a></li>
<li>defining default CPU/memory requests and limits at the <a href="https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/">namespace level</a> using a <code>LimitRange</code></li>
</ul>
<p>However, what if circumstances allowed you to allocate unbounded resources to your pod? While that would go against the idea of bin-packing pods by using resource bounded cgroups, it could still useful if you ran no other pods that the unbounded one on your node. In that case, wouldn't be interested in protecting your pod against any noisy neighbour, and you'd want it to be able to use all the available node resources.</p>
<p>This (while not strictly documented) can be accomplished by using the following resource limits and requests:</p>
<div class="highlight"><pre><span></span><code><span class="n">resources</span><span class="o">:</span>
<span class="w"> </span><span class="n">limits</span><span class="o">:</span>
<span class="w"> </span><span class="n">cpu</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">memory</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">requests</span><span class="o">:</span>
<span class="w"> </span><span class="n">cpu</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">memory</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span>
</code></pre></div>
<p>In our case, we also have a defined <code>LimitRange</code> in our namespace, so we want to make sure that our request for unbounded resources does not get overridden by the default values.</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="n">kubectl</span><span class="w"> </span><span class="n">describe</span><span class="w"> </span><span class="n">limitrange</span><span class="w"> </span><span class="n">my</span><span class="o">-</span><span class="n">limit</span><span class="o">-</span><span class="n">range</span>
<span class="n">Name</span><span class="p">:</span><span class="w"> </span><span class="n">my</span><span class="o">-</span><span class="n">limit</span><span class="o">-</span><span class="n">range</span>
<span class="n">Namespace</span><span class="p">:</span><span class="w"> </span><span class="k">default</span>
<span class="n">Type</span><span class="w"> </span><span class="n">Resource</span><span class="w"> </span><span class="n">Min</span><span class="w"> </span><span class="n">Max</span><span class="w"> </span><span class="k">Default</span><span class="w"> </span><span class="n">Request</span><span class="w"> </span><span class="k">Default</span><span class="w"> </span><span class="n">Limit</span>
<span class="o">----</span><span class="w"> </span><span class="o">--------</span><span class="w"> </span><span class="o">---</span><span class="w"> </span><span class="o">---</span><span class="w"> </span><span class="o">---------------</span><span class="w"> </span><span class="o">-------------</span>
<span class="n">Container</span><span class="w"> </span><span class="n">memory</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">512</span><span class="n">Mi</span><span class="w"> </span><span class="mi">1</span><span class="n">Gi</span>
<span class="n">Container</span><span class="w"> </span><span class="n">cpu</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">500</span><span class="n">m</span><span class="w"> </span><span class="mi">1</span>
<span class="err">$</span><span class="w"> </span><span class="n">kubectl</span><span class="w"> </span><span class="k">get</span><span class="w"> </span><span class="n">pod</span><span class="w"> </span><span class="n">my</span><span class="o">-</span><span class="n">pod</span><span class="w"> </span><span class="o">-</span><span class="n">o</span><span class="w"> </span><span class="n">jsonpath</span><span class="o">=</span><span class="c">'{.spec.containers[0].resources}'</span>
<span class="n">map</span><span class="o">[</span><span class="n">limits</span><span class="p">:</span><span class="n">map</span><span class="o">[</span><span class="n">cpu</span><span class="p">:</span><span class="mi">0</span><span class="w"> </span><span class="n">memory</span><span class="p">:</span><span class="mi">0</span><span class="o">]</span><span class="w"> </span><span class="n">requests</span><span class="p">:</span><span class="n">map</span><span class="o">[</span><span class="n">cpu</span><span class="p">:</span><span class="mi">0</span><span class="w"> </span><span class="n">memory</span><span class="p">:</span><span class="mi">0</span><span class="o">]]</span>
</code></pre></div>
<p>It seems that the <code>LimitRange</code> has not overridden our request. However, we see a different picture when we inspect the node running our pod:</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="n">kubectl</span><span class="w"> </span><span class="k">get</span><span class="w"> </span><span class="n">pod</span><span class="w"> </span><span class="n">my</span><span class="o">-</span><span class="n">pod</span><span class="w"> </span><span class="o">-</span><span class="n">o</span><span class="w"> </span><span class="n">jsonpath</span><span class="o">=</span><span class="c">'{.spec.nodeName}'</span>
<span class="n">my</span><span class="o">-</span><span class="n">node</span>
<span class="err">$</span><span class="w"> </span><span class="n">kubectl</span><span class="w"> </span><span class="n">describe</span><span class="w"> </span><span class="n">node</span><span class="w"> </span><span class="n">my</span><span class="o">-</span><span class="n">node</span>
<span class="p">...</span>
<span class="n">Non</span><span class="o">-</span><span class="n">terminated</span><span class="w"> </span><span class="n">Pods</span><span class="p">:</span><span class="w"> </span><span class="p">(</span><span class="mi">6</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">total</span><span class="p">)</span>
<span class="w"> </span><span class="k">Namespace</span><span class="w"> </span><span class="nn">Name</span><span class="w"> </span><span class="n">CPU</span><span class="w"> </span><span class="n">Requests</span><span class="w"> </span><span class="n">CPU</span><span class="w"> </span><span class="n">Limits</span><span class="w"> </span><span class="n">Memory</span><span class="w"> </span><span class="n">Requests</span><span class="w"> </span><span class="n">Memory</span><span class="w"> </span><span class="n">Limits</span>
<span class="w"> </span><span class="o">---------</span><span class="w"> </span><span class="o">----</span><span class="w"> </span><span class="o">------------</span><span class="w"> </span><span class="o">----------</span><span class="w"> </span><span class="o">---------------</span><span class="w"> </span><span class="o">-------------</span>
<span class="w"> </span><span class="n">datadog</span><span class="w"> </span><span class="n">my</span><span class="o">-</span><span class="n">pod</span><span class="w"> </span><span class="mi">500</span><span class="n">m</span><span class="w"> </span><span class="p">(</span><span class="mi">13</span><span class="err">%</span><span class="p">)</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">(</span><span class="mi">26</span><span class="err">%</span><span class="p">)</span><span class="w"> </span><span class="mi">512</span><span class="n">Mi</span><span class="w"> </span><span class="p">(</span><span class="mi">2</span><span class="err">%</span><span class="p">)</span><span class="w"> </span><span class="mi">1</span><span class="n">Gi</span><span class="w"> </span><span class="p">(</span><span class="mi">4</span><span class="err">%</span><span class="p">)</span>
<span class="p">...</span>
</code></pre></div>
<p>Who should we believe? When different parts of the control plane disagree on the resource allotment, there's really one place to get the truth from: the container cgroup itself.</p>
<p>To do so, we need to exec into the pod, and inspect the CPU quota and memory limit values.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kubectl<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>-it<span class="w"> </span>my-pod
user@my-pod:/$<span class="w"> </span>cat<span class="w"> </span>/sys/fs/cgroup/cpu/cpu.cfs_quota_us
-1
</code></pre></div>
<p>As detailed on the <a href="https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt">Linux kernel documentation</a>, or the Red Hat <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-cpu">documentation portal</a></p>
<blockquote>
<p>A value of -1 for <code>cpu.cfs_quota_us</code> indicates that the group does not have any
bandwidth restriction in place, such a group is described as an unconstrained
bandwidth group. This represents the traditional work-conserving behavior for
CFS.</p>
</blockquote>
<p>Now, the memory.</p>
<div class="highlight"><pre><span></span><code><span class="k">user</span><span class="nv">@my</span><span class="o">-</span><span class="nl">pod</span><span class="p">:</span><span class="o">/</span><span class="err">$</span><span class="w"> </span><span class="n">cat</span><span class="w"> </span><span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">fs</span><span class="o">/</span><span class="n">cgroup</span><span class="o">/</span><span class="n">memory</span><span class="o">/</span><span class="n">memory</span><span class="p">.</span><span class="n">limit_in_bytes</span>
<span class="mi">9223372036854771712</span>
</code></pre></div>
<p>That looks odd. This would indicate that the process has a limit of ... 8191TB of memory!</p>
<p>Digging <a href="https://unix.stackexchange.com/questions/420906/what-is-the-value-for-the-cgroups-limit-in-bytes-if-the-memory-is-not-restricte">a bit further</a>, we learn that <code>9223372036854771712</code> is a kind of "magic" number in the memory management layer of the kernel, meaning that the process gets unbounded memory.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Looking at the cgroup itself showed that a value of <code>0</code> for cpu/memory requests/limits is not intercepted by the <code>LimitRange</code> in place, and is translated to an unbounded cgroup in the end. It also showed that the pod resource requests and limits reported at the node level are inaccurate.</p>On meritocracy, identity and context2018-09-21T00:00:00+02:002018-09-21T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2018-09-21:/on-meritocracy-identity-and-context<p><strong>Before reading</strong></p>
<p>This is a deeply personal article, that hasn't been easy to write, especially with all the tension currently occurring in the tech industry, around inclusiveness, gender, code of conducts, etc. I've done my best to explain my thoughts on the matter, while being as respectful as possible. If …</p><p><strong>Before reading</strong></p>
<p>This is a deeply personal article, that hasn't been easy to write, especially with all the tension currently occurring in the tech industry, around inclusiveness, gender, code of conducts, etc. I've done my best to explain my thoughts on the matter, while being as respectful as possible. If you feel that you disagree with me, I'd be happy to debate with you, as long as the discussion stays civil and respectful.</p>
<hr>
<h3 id="the-lay-of-the-land">The lay of the land</h3>
<p>I have been working as a professional engineer in the tech industry for the last 7 years or so. My first contact with the subject of under-representation of minorities in the industry came during EuroPython 2012, when a tasteless tweet was posted the very night after which <a href="http://www.roguelynn.com/">Lynn Root</a> talked about <a href="https://www.youtube.com/watch?v=l2PnVKQJg0I">"Increasing women engagement in the Python community"</a> (these events are best summarized by <a href="http://www.roguelynn.com/words/a-memorable-europython-for-the-better/">Lynn herself</a>). And these events kept <a href="https://en.wiktionary.org/wiki/Donglegate">happening</a>, and <a href="https://www.dailydot.com/debug/sexist-tech-conference-slide/">happening</a>, and <a href="https://en.wikipedia.org/wiki/Sexism_in_the_technology_industry#Incidents">happening</a>. My personal view has since then been that each of these highly publicized events were caused by an appalling lack of tact, thoughtfulness, empathy and respect, and that conference attendees should make sure to behave appropriately or should suffer the consequences.</p>
<p><a href="http://confcodeofconduct.com/">Code of Conducts</a> started to be <a href="https://ep2018.europython.eu/en/coc/">defined</a> for <a href="https://www.dotconferences.com/codeofconduct">conferences</a> and <a href="https://www.djangoproject.com/conduct/">online projects</a>, thanks to the initiative of groups and individuals pushing for more respectful and inclusive communities. I have always thought that these were essential and useful, because they seemed to be making some conference attendees or project members feel safer (and not making myself feel less so), and would probably help keeping jerk-like behavior at bay.</p>
<p><a href="https://djangogirls.org/pyconuk/">Safe spaces</a> were organized in conferences (I even helped on a few myself, as a tutor), and I thought it was a wonderful idea. I did not take anything away from the most represented types of conference attendees, and allowed less represented people to be able to take their marks in a safe environment.</p>
<p>However, I feel something changed for me when I read the proposal to replace the <code>master/slave</code> terminology by <code>leader/follower</code> in the <a href="https://github.com/django/django/pull/2692">Django framework</a>. The PR starts with the following stance:</p>
<blockquote>
<p>The docs and some tests contain references to a master/slave db configuration.
While this terminology has been used for a long time, those terms may carry racially charged meanings to users.</p>
</blockquote>
<p>My view at the time was <em>"I mean, it does not really change anything for me, and if it can help people feel better..."</em>. Looking back, I'm pretty sure I felt a bit of unease reading the PR, but I (subconsciously or not) shrugged it off.</p>
<p>That debate recently resurfaced when the same thing happened in both the <a href="https://github.com/antirez/redis/issues/5335">redis</a> and the <a href="https://bugs.python.org/issue34605">CPython</a> codebases.
Reading what antirez (the redis creator) <a href="http://antirez.com/news/122">had to say on the subject</a> was a real moment of clarity for me.</p>
<blockquote>
<p>Today it happened again. A developer, that we’ll call Mark to avoid exposing his real name, read the Redis 5.0 RC5 change log, and was disappointed to see that Redis still uses the “master” and “slave” terminology in order to identify different roles in Redis replication.</p>
<p>I said that I was sorry he was disappointed about that, but at the same time, I don’t believe that terminology out of context is offensive, so if I use master-slave in the context of databases, and I’m not referring in any way to slavery. I originally copied the terms from MySQL, and now they are the way we call things in Redis, and since I do not believe in this battle (I’ll tell you later why), to change the documentation, deprecate the API and add a new one, change the INFO fields, just to make a subset of people that care about those things more happy, do not make sense to me.</p>
<p>After it was clear that I was not interested in his argument, Mark accused me of being fascist.</p>
</blockquote>
<p>At this point, I realized the landscape had dramatically changed, and that the inclusiveness debate had morphed into a more politicized and (according to me) confused and sterile version of itself.</p>
<p>Case in point, someone suggested the <a href="https://www.python.org/dev/peps/pep-0020/">Zen of Python</a> should be <a href="https://mail.python.org/pipermail/python-ideas/2018-September/053365.html">modified</a> because the sentence <em>Beautiful is better than ugly</em> could be interpreted as a support for body-shaming behaviors. Words cannot express how wrong this feels to me. That suggestion shows both a profound lack of contextual thinking, and a will to advance a pro political correctness agenda.</p>
<p>People have been talking about <a href="https://www.amazon.com/Beautiful-Code-Leading-Programmers-Practice/dp/0596510047">Beautiful Code</a> and <a href="http://uglycode.com/">Ugly Code</a> for a <strong>long time</strong>. Long enough to write books about it. Long enough so that I could have late night discussions about it with my father (who's also a computer scientist). To me, suggesting that <em>Beautiful is better than ugly</em> encourages body shaming feels alien, because it's completely <strong>out of context</strong>. Words have certain meanings in certain contexts. That's how we get away with synonyms. In the context of the <a href="https://www.python.org/dev/peps/pep-0020/">Zen of Python</a>, the word <em>Beautiful</em> clearly characterizes code, not people. The <a href="https://en.wikipedia.org/wiki/Dwarf_star">Dwarf Star</a> term defines a certain type of star, with given astrophysical properties. Should the entire astrophysics community rename it just because some people feel it's an offensive way of calling <a href="https://fr.wikipedia.org/wiki/Peter_Dinklage">Peter Dinklage</a>? Similar humorous (or not?) <a href="https://bugs.python.org/msg324816">counter-arguments</a> were offered during the CPython master/slave debate.</p>
<p>It seems all we read about now (especially after Linus Torvalds' <a href="https://lkml.org/lkml/2018/9/16/167">temporary stepdown</a>) is either written by <a href="https://medium.com/culture-null/how-sjws-infiltrated-the-open-source-community-21001e7059ef">strong meritocracy</a> <a href="https://lkml.org/lkml/2018/9/16/198">partisans</a>, <a href="https://www.reddit.com/r/linux/comments/9ghrrj/linuxs_new_coc_is_a_piece_of_shit/e64h04h/">conspiracy theorists</a> or by <a href="https://archive.is/dgilk">strong inclusiveness defenders</a> (I've decided not to use the term SJW, as I understand it's a <a href="https://en.wikipedia.org/wiki/Social_justice_warrior">mocking and pejorative</a> term).</p>
<p>It was even <a href="https://mail.python.org/pipermail/python-ideas/2018-September/053369.html">suggested</a> and <a href="https://mail.python.org/pipermail/python-ideas/2018-September/053375.html">debated</a> whether that this suggestion was made by a troll. The fact that, troll or not, that discussion lingered for several days is a very serious issue to me. It shows how polarized the debate now is, and how easily a strong community can be derailed.</p>
<h3 id="about-inclusivity-diversity-and-context">About inclusivity, diversity and context</h3>
<p>The core of the debate is focused on inclusivity and diversity (see <a href="https://bugs.python.org/issue34605">this example</a>), which got me thinking. It's clear to me <em>why</em> we want to push for diversity:</p>
<ul>
<li>a body of similar minds will likely producer similar solutions to a problem, causing the final adopted solution to be <a href="https://www.dailymail.co.uk/sciencetech/article-4800234/Is-soap-dispenser-RACIST.html">more narrowed</a></li>
<li>a person could (subconsciously or not) avoid a given career path / community because she/he might not feel represented enough, and thus feel excluded or as though he/she does not belong</li>
</ul>
<p>I want to focus on the second point, because I'm of the opinion that this is where the heated debates stem from.</p>
<p>If you read the <a href="https://www.contributor-covenant.org/version/1/4/code-of-conduct">Code of Conduct Covenant</a>, which is a code of conduct most of the current conferences and community use or are based on, the text starts with:</p>
<blockquote>
<p>In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.</p>
</blockquote>
<p>I naturally tend to agree with this. We should all strive for inclusiveness and diversity, and should make sure everyone is treated gently and is given a friendly, open hand, whomever they are.
However, if I were fostering malicious intent, I could point out that this list does not cover diets. I myself am a flexitarian (I've cut out all fish and meat from my daily diet, but will eat some without issue if there's no other option). I could somehow feel unrepresented or even excluded from a given community if its CoC does not state that my personal diet should be respected.</p>
<p>Although that example could seem frivolous or ridiculous, it points out something I feel is interesting. That whole paragraph attempts at listing all the way people could differ, to make sure everyone is explicitly included. I would personally have phrased it a more open-ended way:</p>
<blockquote>
<p>In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of who they are and how they identify.</p>
</blockquote>
<p>as I think some issues stem from the fact we have attempted to list what constitutes "diversity". If a tech conference decides to impose quotas on speakers, these quotas will focus on certain attributes (eg sex and skin color) while missing others (eg age, education), which might help some people to better identify to the speakers, but might not help others. This <em>inventaire à la Prévert</em> certainly looks like inclusiveness, but I think it misses the point.</p>
<p>How we identify is both subjective and subject to context. I might identify as an SRE, an engineer or a Python developer in the context of work or a tech-related event, a social extrovert in the context of a party, an leftist heterosexual male in the context of my personal and private life, etc.</p>
<p>How we identify depends on context, and yet, we seem intent on mixing personal identities and non-personal contexts, the same way accusing <em>Beautiful is better than ugly</em> to promote body shaming mixes human and technological contexts. I recognize that some situations are trickier than others (eg conferences, workplaces), because they can mix personal and professional contexts, thus blurring the lines.</p>
<p>If diversity is defined as having multiple identities present, then diversity must be subjective and subject to context too. To follow in that tech conference example, I feel diversity in the technological content should reside in education background, level of experience and field of interest of the speaker, while diversity in the social events tied to the conference could have a totally different definition.</p>
<p>These criterion are my pick, but I suggest you clearly and openly define which ones matter to you if you're ever in the position of selecting speakers or employees.</p>
<h3 id="closing-words">Closing words</h3>
<p>In my view, the tech industry as a whole has been guilty of resistance to change by kicking around the old meritocracy horse for too long. We need to talk about the lack of women, the rampant misogynist attitudes, the male/women pay gap. We need to fix these issues by acknowledging them first, and debating them transparently, in a less polarized way. Not just as an industry, but as a society.</p>
<p>However, as I don't buy in the "show me the code or GTFO" attitude, I don't believe in politically correctness before everything else. If some people lack the ability to recognize that <em>Beautiful is better than ugly</em> in the <a href="https://www.python.org/dev/peps/pep-0020/">Zen of Python</a> does not body shame people, then maybe we shouldn't let them define what our core values are.</p>Solution to Advent of Code "Day 3: Spiral Memory"2017-12-31T00:00:00+01:002017-12-31T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2017-12-31:/solution-to-advent-of-code-day-3-spiral-memory<p>After an unsuccessful attempt at learning Rust earlier this year (I mainly read through the documentation without applying it in any project), I recently started to tackle the <a href="https://adventofcode.com/2017/">2017 edition of Advent of Code</a>, in order to practice Rust for real.</p>
<p>The 3rd challenge, <a href="https://adventofcode.com/2017/day/3"><em>Spiral Memory</em></a> is interesting because you …</p><p>After an unsuccessful attempt at learning Rust earlier this year (I mainly read through the documentation without applying it in any project), I recently started to tackle the <a href="https://adventofcode.com/2017/">2017 edition of Advent of Code</a>, in order to practice Rust for real.</p>
<p>The 3rd challenge, <a href="https://adventofcode.com/2017/day/3"><em>Spiral Memory</em></a> is interesting because you can <a href="https://gist.github.com/pawlos/0cefa9d753bd6416e6cc9a456ed787f7">bruteforce</a> it, or solve it with math. I ended up doing the latter, even though math is really not my strong suit.</p>
<p>We're asked to calculate the <a href="https://en.wikipedia.org/wiki/Taxicab_geometry">Manhattan distance</a> between a given point and the center, in a spiral reference. The problem amounts to finding the coordinates of any point $P$ in this spiral reference, as once we have the point coordinates, calculating the Manhattan distance is easy:</p>
<p>\begin{align<em>}
D_P &= |X_P - X_0| + |Y_P - Y_0| \
&= |X_P| + |Y_P|
\end{align</em>}</p>
<h2 id="nested-shells">Nested shells</h2>
<p>My approach was the following: a spiral has nested "shells", all centered around the center. In this image, the first shell is outlined in grey, and the second one in purple. Each of these spirals has a first value, called $S_i$, where $i$ is the index of the spiral.</p>
<p><img alt="spiral" decoding="async" loading="lazy" src="images/memory-spiral.jpg"></p>
<p>For any point $(X_P, Y_P)$ of value $V$, we know that it is located somewhere on the shell located right before the first shell with start value $S$ such as $S > V$. For example, if the input value was 23, we know that it's located on the second shell as $S_2 ≤ 23 < S_3$.</p>
<p><img alt="spiral" decoding="async" loading="lazy" src="images/spiral-shells.jpg"></p>
<p>We need to know the number of elements a shell of index $i$ is composed of, noted $Δ_i$ On this representation, the first shell is a square of side of length 3, the second shell is a square of side of length 5. We can generalize this to $L = 2i + 1$, where $i$ is the index of the shell. For any index $i$, the shell is composed of the following number of elements</p>
<p>\begin{align<em>}
Δ_i &= (2i + 1)^2 - (2(i -1) + 1)^2 \
&= 4i^2 + 4i + 1 - 4i^2 +4i - 1 \
&= 8i
\end{align</em>}</p>
<h2 id="coordinates-of-the-first-element-of-a-shell">Coordinates of the first element of a shell</h2>
<p>Once we know on which shell a given point $P$ is located, we need to know the coordinates of the first point $S_i$ of this shell, so we can infer $P$'s coordinates. This first point will always be located after the center point, and all points composing the previous shells. We can thus infer</p>
<p>\begin{equation<em>}
V_{S_{{}<em x="1">i}} = 2 + \sum</em>^{i-1}Δ_i
\end{equation</em>}</p>
<p>We now need to get the coordinates of any given first shell point. By simply looking at the spiral itself, we can deduce that</p>
<p>\begin{equation<em>}
(X_{S_{{}<em S__="S_{{">i}}, Y</em>_i}}) = (n, -n + 1)
\end{equation</em>}</p>
<h2 id="navigating-the-spiral">Navigating the spiral</h2>
<p>The final piece of the puzzle is to infer the coordinates of the point $P$ given the coordinates of the start point $S_i$ of the shell it belongs to. To do that, we need to look at how the coordinates evolve along a shell.</p>
<p><img alt="spiral" decoding="async" loading="lazy" src="images/shell-coordinates.jpg"></p>
<p>We can see that:</p>
<ul>
<li>on the first quarter of the shell, $Y$ coordinates increase by 1 for each increasing value</li>
<li>on the second quarter of the shell, $X$ coordinates decrease by 1 for each increasing value</li>
<li>on the third quarter of the shell, $Y$ coordinates decrease by 1 for each increasing value</li>
<li>on the fourth quarter of the shell, $X$ coordinates increase by 1 for each increasing value</li>
</ul>
<p>To calculate the coordinates of the point $P$, we just need to locate it on the shell, start from $(X_{S_{{}<em S__="S_{{">i}}, Y</em>_i}})$ and increase/decrease the $X$ and $Y$ coordinates until we reach the target value.</p>
<h2 id="the-implementation">The implementation</h2>
<p>The strategy is:</p>
<ul>
<li>calculate the values of the first shell points until we find a value greater than our target point</li>
<li>backtrack to the previous shell</li>
<li>compute the coordinates of the first point of the shell we backtracked to</li>
<li>increase/decrease the $X$ and $Y$ coordinates until we reach the target value</li>
<li>calculate the Manhattan distance using these coordinates</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="c1">// advent_day03.rs</span>
<span class="k">fn</span> <span class="nf">nb_elements_in_outer_level</span><span class="p">(</span><span class="n">level</span>: <span class="kt">i32</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">i32</span><span class="p">{</span>
<span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">start_element</span><span class="p">(</span><span class="n">level</span>: <span class="kt">i32</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">i32</span> <span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">level</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="mi">1</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">out</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">1</span><span class="o">..</span><span class="n">level</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">out</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">nb_elements_in_outer_level</span><span class="p">(</span><span class="n">i</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">out</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">first_element_coordinates</span><span class="p">(</span><span class="n">level</span>: <span class="kt">i32</span><span class="p">)</span><span class="w"> </span>-> <span class="p">(</span><span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="kt">i32</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="p">(</span><span class="n">level</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="n">level</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">number_coordinates</span><span class="p">(</span><span class="n">number</span>: <span class="kt">i32</span><span class="p">)</span><span class="w"> </span>-> <span class="p">(</span><span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="kt">i32</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">level</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">start</span>: <span class="kt">i32</span><span class="p">;</span>
<span class="w"> </span><span class="c1">// Increase level until we found a starting value greater than</span>
<span class="w"> </span><span class="c1">// input value. When such a value is found, backtrack a step.</span>
<span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">start_element</span><span class="p">(</span><span class="n">level</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="n">number</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">level</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{:?} is found on level {:?} of the spiral"</span><span class="p">,</span><span class="w"> </span><span class="n">number</span><span class="p">,</span><span class="w"> </span><span class="n">level</span><span class="p">);</span>
<span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">start_element</span><span class="p">(</span><span class="n">level</span><span class="p">);</span>
<span class="w"> </span><span class="k">break</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">level</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="c1">// At this point, we've found the starting point of the spiral</span>
<span class="w"> </span><span class="c1">// level we number belongs to.</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">delta</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">number</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">start</span><span class="p">;</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">first_element_coordinates</span><span class="p">(</span><span class="n">level</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">delta</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">delta</span><span class="p">;</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">delta</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="n">delta</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="p">(</span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">delta</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="n">delta</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="p">(</span><span class="mi">4</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">delta</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="p">(</span><span class="mi">6</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">level</span><span class="p">);</span>
<span class="w"> </span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">manhattan_distance</span><span class="p">(</span><span class="n">x</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="n">y</span>: <span class="kt">i32</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">i32</span><span class="p">{</span>
<span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="n">abs</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">y</span><span class="p">.</span><span class="n">abs</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">number</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">312051</span><span class="p">;</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">number_coordinates</span><span class="p">(</span><span class="n">number</span><span class="p">);</span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{:?} has coordinates {:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">number</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">));</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">distance</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">manhattan_distance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">);</span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{:?} is at a distance of {:?} from the center"</span><span class="p">,</span><span class="w"> </span><span class="n">number</span><span class="p">,</span><span class="w"> </span><span class="n">distance</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>
<h2 id="the-solution">The solution</h2>
<div class="highlight"><pre><span></span><code><span class="mi">312051</span> <span class="k">is</span> <span class="n">found</span> <span class="n">on</span> <span class="nb">level</span> <span class="mi">279</span> <span class="nb">of</span> <span class="n">the</span> <span class="n">spiral</span>
<span class="mi">312051</span> <span class="k">has</span> <span class="n">coordinates</span> (-<span class="mi">152</span>, -<span class="mi">278</span>)
<span class="mi">312051</span> <span class="k">is</span> <span class="nb">at</span> <span class="n">a</span> <span class="n">distance</span> <span class="nb">of</span> <span class="mi">430</span> <span class="nb">from</span> <span class="n">the</span> <span class="n">center</span>
</code></pre></div>On working from home while remaining sane2017-10-29T00:00:00+02:002017-10-29T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2017-10-29:/on-working-from-home-while-remaining-sane<p>Since I started working at <a href="https://datadoghq.com">Datadog</a>, I've had the opportunity of working from home full-time (for the second time in my career). Although I consider this to be a real privilege, it comes with its own set of challenges that I'd like to pinpoint and address in light of my …</p><p>Since I started working at <a href="https://datadoghq.com">Datadog</a>, I've had the opportunity of working from home full-time (for the second time in my career). Although I consider this to be a real privilege, it comes with its own set of challenges that I'd like to pinpoint and address in light of my personal experiences.</p>
<p>I hope this article will be useful for anyone willing to try out (or struggling with) remote work.</p>
<h2 id="productivity-vs-isolation">Productivity VS isolation</h2>
<p>First, why would you even want to work from home in the first place? To me, it's both about flexibility and productivity. I can focus on complex tasks for long periods of time without being <a href="http://heeris.id.au/2013/this-is-why-you-shouldnt-interrupt-a-programmer/">interrupted</a>, while still being able to keep a flexible timetable. I can also work from anywhere, as long as I can have a good enough internet connection.</p>
<p>However, this flexibility and freedom is paid with isolation, which can then lead to demotivation or burn-out down the road. Remote work is, by definition, solitary, which can quickly become an issue, because humans are social animals and (for most of us) crave for social and physical interaction. This make be believe that remote workers are more exposed to burn-out.</p>
<h2 id="the-burn-out-cycle">The burn-out cycle</h2>
<p>In my experience, the easiest path to demotivation or burn-out (whether you're working remotely or not) is being over-enthusiast and working long hours. When doing so, it's easy to develop some kind of <em>hero complex</em>, a belief that you're indispensable and that things will break down if you take a break, or leave on holidays. The more hours you pull, the less sleep you get, the more stressed and tired you become. Because you're stressed, you then feel you need to work harder, until you just can't take it anymore, and you burn-out.</p>
<p>Ideally, this cycle can be prevented or broken with proper management and supervision. If your manager realises you've started to walk this slippery slope, she/he should take action, and incite/force you to take a break. This can be enforced by regular 1-1 meetings, to keep track on how remote workers are doing.</p>
<p>This brings me to an important point: <strong>remote work can dangerous if it's not in the company culture, and you should keep away</strong>.</p>
<h2 id="remote-as-a-culture">Remote as a culture</h2>
<p>To enable sane remote work, a company does must include remote workers in all events, when physically possible. All brown-bags, talks, all-hands, etc, should be streamed live, or at least recorded. If being out of the office means you have access to less information, it means that remote workers are seen as second-rate employees.</p>
<p>All communication must be asynchronous, to include remote workers, especially if teams are working across timezones. Wether it's slack, email, Google Docs or something else, anyone should be able to catch up with any conversation or topic. Any significant direct discussion should be made available one way or another to remote workers.</p>
<p>Finally, it should be easy to go meet your team in person. I'd go even further and recommend you do it on a regular basis. I personally chose to go to our Paris office a week every month.</p>
<h2 id="work-hygiene">Work hygiene</h2>
<p>Now, if your company has remote in its blood and culture, good! Now, all is left to figure out is <em>your</em> organisation and work hygiene. The following advice come from my personal experience and should not be considered as absolute truths backed by science. Take them if they make sense to you.</p>
<h3 id="containerisation-of-private-and-personal-life">Containerisation of private and personal life</h3>
<p>The first thing I find absolutely essential is containerisation (no, not Docker) of your private and professional life. You need to have a dedicated office room, with a door, that is not your living room. The idea is that, when you open that door, you're at work, and when you close it, you're out. I find it to be especially important during the first weeks of remote work. I now find myself work more and more from my living room, but I know that if I need isolation for some reason, I still have this room I can go to.</p>
<p>For the same reason (along with a bazillion security reasons), never work from your personal machine. You want to make it a conscious effort to switch from watching Netflix to reading your work email.</p>
<h3 id="routine">Routine</h3>
<p>To me, routine is key to avoid getting tired. Try to wake up, start working, eat, stop working and go to sleep at regular hours. Ban any night work, especially when you're not on-call.</p>
<p>Exercise is also very important. It's easy to keep extremely static during your remote work-day, which can take a toll on your health. Also, one of the things I miss the most is my daily bike commute. I replaced it with 45 minutes of gym in the morning, 3 times a week. This has the nice advantage of making me feel like I accomplished something very early in the day, and gives me energy to keep it going.</p>
<p>Also, take regular breaks and take a 15 minute walk at some point in the day.</p>
<h3 id="share">Share</h3>
<p>Talk with other remote workers about how <em>they</em> make it. Share tips, stories, do-s and don't-s, to build collective wisdom.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Working from home can be liberating and an amazing productivity booster, but you need to stay alert and conscious of the challenges and constraints it entails. I'd urge you to show a fair amount of self-discipline and organisation in order to avoid falling into the burn-out spiral.</p>
<p>Have fun!</p>
<h2 id="edit">Edit</h2>
<p>I found this very interesting resource from Trello, called <a href="https://info.trello.com/hubfs/Trello-Embrace-Remote-Work-Ultimate-Guide.pdf">How to embrace remote work</a>.</p>
<p>The main takeways I get from it are:</p>
<ul>
<li>pace yourself: work isn't going anyhwere. Do not forget to take breaks.</li>
<li>use the right tool to convey the right information (do not rely on instant messaging for crucial information!)</li>
<li>don't forget to use passive communication (eg: status messages)</li>
<li>intent can be lost over text communication. Assume <strong>positive</strong> intent.</li>
</ul>The story of the 20°C cronjob2017-05-25T00:00:00+02:002017-05-25T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2017-05-25:/the-story-of-the-20degc-cronjob<p>For the last month or so, the lifespan of my beloved Thinkpad X1 Carbon battery had been getting down the drains, from 5-6 hours to less than 3. Following <a href="https://twitter.com/padenot">@padenot</a>'s advice, I installed <code>powertop</code> and started investigating what was draining this good'ol battery of mine.</p>
<p>Looking at the <code>powertop …</code></p><p>For the last month or so, the lifespan of my beloved Thinkpad X1 Carbon battery had been getting down the drains, from 5-6 hours to less than 3. Following <a href="https://twitter.com/padenot">@padenot</a>'s advice, I installed <code>powertop</code> and started investigating what was draining this good'ol battery of mine.</p>
<p>Looking at the <code>powertop</code> output, I immediatly realized that something fishy was happening on this laptop:</p>
<div class="highlight"><pre><span></span><code>The battery reports a discharge rate of 4.95 W
The estimated remaining time is 2 hours, 6 minutes
Summary: 1111.7 wakeups/second, 7.9 GPU ops/seconds, 0.0 VFS ops/sec and 23.0% CPU use
Usage Events/s Category Description
264.4 ms/s 3656.7 Process /bin/bash /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f br
114.3 ms/s 626.2 Process /usr/lib64/firefox/firefox
20.7 ms/s 95.5 Process /opt/sublime_text_3/plugin_host 3272
...
</code></pre></div>
<p>Why was <code>sendmail</code> so busy, and why in the hell was it running anyway? <code>strace</code> showed me that the process was indeed very busy, and <code>mailq</code> showed that I had more than 15000 outgoing emails in the system mail queue!</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mailq
...
mail<span class="w"> </span><span class="k">in</span><span class="w"> </span>dir<span class="w"> </span>/home/br/.esmtp_queue/TSRueRJD:
<span class="w"> </span>From:<span class="w"> </span><span class="s2">"(Cron Daemon)"</span><span class="w"> </span><br><span class="w"> </span>To:<span class="w"> </span>br
mail<span class="w"> </span><span class="k">in</span><span class="w"> </span>dir<span class="w"> </span>/home/br/.esmtp_queue/ZI1LtzhT:
<span class="w"> </span>From:<span class="w"> </span><span class="s2">"(Cron Daemon)"</span><span class="w"> </span><br><span class="w"> </span>To:<span class="w"> </span>br
<span class="m">15653</span><span class="w"> </span>mails<span class="w"> </span>to<span class="w"> </span>deliver
</code></pre></div>
<p>Ok, so all these mails were being sent by <code>cron</code>. My user crontab only had one job, and it was <code>* * * * * rm $HOME/crash_dump.erl</code>. Indeed, I had been experimenting with <a href="http://elixir-lang.org/">Elixir</a> recently, and when I crashed the Erlang VM, this file would pop-up in my home directory. At some point, I added this cronjob to make it go away and forgot about it. As the job's <code>stdout</code> was not redirected to <code>/dev/null</code>, each time the file was not found, the cron job would fail and a mail would be added to the queue.</p>
<p>After removing this job, purging the mail queue, and adding <code>MAILTO=""</code> at <a href="https://www.cyberciti.biz/faq/disable-the-mail-alert-by-crontab-command/">the beginning of my crontab</a> (to avoid repeating this investigation down the road), <code>sendmail</code> went quiet, my battery life went back to ~6 hours, and the laptop average temperature went down 20°C.</p>Preparing the SRE interview2017-04-20T00:00:00+02:002017-04-20T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2017-04-20:/preparing-the-sre-interview<p>I recently interviewed for an <abbr title="Site Reliability Engineer">SRE</abbr> position. I spent a full week learning (or refreshing my memory) on the subjects and topics that could be covered in such an interview. I'll try and lay down the list of topics I covered and resources I used.</p>
<h2 id="what-is-an-sre">What is an SRE?</h2>
<p>Having …</p><p>I recently interviewed for an <abbr title="Site Reliability Engineer">SRE</abbr> position. I spent a full week learning (or refreshing my memory) on the subjects and topics that could be covered in such an interview. I'll try and lay down the list of topics I covered and resources I used.</p>
<h2 id="what-is-an-sre">What is an SRE?</h2>
<p>Having spent the last 2 years employed as a DevOps, I've often felt that DevOps and SRE were two slightly differing implementations of the same ideas. The first one felt like a set of general principles, when the second one is a clear and detailed model (pre-dating DevOps), with a set of rules and guidelines. Google developed the SRE model and explained it in the <a href="https://landing.google.com/sre/book.html">SRE book</a>. The underlying ideas are simple, but powerful:</p>
<ul>
<li>Develop tools and systems reducing toil and repetitive work from engineers</li>
<li>Automate everything, or as much as possible (deployments, maintenances, tests, scaling, mitigation)</li>
<li>Monitor everything</li>
<li>Think scalable from the start</li>
<li>Build <strong>resilient-enough</strong> architectures</li>
<li>Handle change and risk through <abbr title="Service Level Agreement">SLA</abbr>s, <abbr title="Service Level Objective">SLO</abbr>s and <abbr title="Service Level Indicator">SLI</abbr>s</li>
<li>Learn from outages</li>
</ul>
<p>If you haven't yet read the <a href="https://landing.google.com/sre/book.html">SRE book</a>, I strongly urge you to do so. There's even a <a href="https://landing.google.com/sre/book/index.html">free online version</a> available. If you do not have the time, then maybe have a look at this Ben Treynor (Google VP Engineering) <a href="https://landing.google.com/sre/interview/ben-treynor.html">What is 'Site Reliability Engineering'?</a> interview, for a general introduction.</p>
<p>According to the SRE book, an SRE should spend half of its time on "ops" work, and the other half doing development.</p>
<blockquote>
<p>Google places a 50% cap on the aggregate "ops" work for all SREs—tickets, on-call, manual tasks, etc. [...] An SRE team must spend the remaining 50% of its time actually doing development.
<a href="https://landing.google.com/sre/book/chapters/introduction.html">Source</a></p>
</blockquote>
<p>Some skills are thus paramount to an SRE:</p>
<ul>
<li>coding / software development</li>
<li>system administration and automation</li>
<li>scalable system design</li>
<li>system troubleshooting</li>
</ul>
<p>Consequently, each of these areas of expertise can be (and often are) the subject of an interview.</p>
<h2 id="coding-software-development-interview">Coding / Software development interview</h2>
<p>I've found that the reference resource to prepare a coding interview, especially when targeting companies like Amazon, Google, Microsoft, Yahoo, etc, is <a href="https://www.amazon.com/Cracking-Coding-Interview-Programming-Questions/dp/0984782850/ref=sr_1_1?ie=UTF8&qid=1492689425&sr=8-1&keywords=cracking+the+coding+interview">Cracking the Coding Interview</a>, by <a href="https://www.amazon.com/Gayle-Laakmann-McDowell/e/B004BI1ZUQ/ref=dp_byline_cont_book_1">Gayle Laakmann McDowell</a>. This book is a real trove of advice (technical or not) and example exercises (with the associated solutions).</p>
<p>Even though it is targeted to <em>software developer</em> interviews, I still covered the following topics listed in the <em>Must Know</em> section of the book:</p>
<p><strong>Data structures</strong>:</p>
<ul>
<li>Linked list</li>
<li>Stack</li>
<li>Queue</li>
<li>Heap</li>
<li>Hash table</li>
<li>Binary tree</li>
<li>associated Big-O <a href="http://bigocheatsheet.com/">time and memory complexity</a> for common operations (Search, insert, delete, etc).</li>
</ul>
<p>I found <a href="https://www.amazon.com/Data-Structures-Algorithms-Using-Python/dp/1590282337">Data structures and Algorithms using Python and C++</a> to be useful (albeit a bit lengthy) when dealing with these data structures for the first time. This <a href="http://www.columbia.edu/~jxz2101/#">presentation</a> gives a short but to-the-point, no-nonsense introduction of these data structures.</p>
<p><strong>Algorithms</strong></p>
<ul>
<li>Mergesort</li>
<li>Quicksort</li>
<li>Binary search</li>
</ul>
<p>I also had a look at <a href="https://github.com/adicu/interview_help/">https://github.com/adicu/interview_help</a> to practice on some real-life interview questions, and at <a href="https://github.com/nryoung/algorithms">https://github.com/nryoung/algorithms</a> to read Python implementations of common data structures and algorithms.</p>
<h2 id="scalable-system-design-interview">Scalable system design interview</h2>
<p>This was my favorite subject to work on, as an apparently simple question such as "Design the bit.ly service" hides unexpected depths of complexity. Being able to design a scalable system implies knowing about:</p>
<ul>
<li>DNS</li>
<li>load balancing</li>
<li>micro-service architecture</li>
<li>CAP theorem</li>
<li>consistency patterns</li>
<li>availability patterns</li>
<li>databases</li>
<li>caching</li>
<li>asynchronism patterns</li>
<li>etc</li>
</ul>
<p>The main idea is to be able to identify the architecture bottlenecks, and to dimension the architecture with an appropriate number of machines, with some "back-of-the-envelope" calculations, whilst being robust and failure tolerant.</p>
<p>The most useful resources I found to prepare were:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=-W9F__D3oY4">Scalability lecture</a> given at Harvard</li>
<li><a href="http://norvig.com/21-days.html#answers">Latency Numbers Every Programmer Should Know</a></li>
<li><a href="https://github.com/donnemartin/system-design-primer">The System Design Primer</a> (I suggest you follow the links after each section for an in-depth follow-up)</li>
<li>this great <a href="https://www.hiredintech.com/classrooms/system-design/lesson/52">step-by-step walkthrough</a> on design questions, by HiredInTech</li>
<li><a href="https://www.youtube.com/watch?v=vg5onp8TU6Q">Scaling up to your first 10 million users</a>, talk given by Joel Williams of AWS</li>
<li><a href="http://www.puncsky.com/blog/2016/02/14/crack-the-system-design-interview/">Crack the design interview</a></li>
<li><a href="https://docs.microsoft.com/en-us/azure/architecture/guide/technology-choices/data-store-overview">When to use NoSQL vs SQL</a></li>
</ul>
<h2 id="system-troubleshooting-interview">System troubleshooting interview</h2>
<p>To be able to automate the administration of a system, one should first know the said system in depth, which, in a lot of cases, will be GNU/Linux. If you have time, I strongly suggest reading <a href="https://www.amazon.com/Linux-Programming-Interface-System-Handbook/dp/1593272200/ref=sr_1_1?ie=UTF8&qid=1492692882&sr=8-1&keywords=linux+programming+interface">The Linux Programming Interface</a>. Note that this is a <strong>large</strong> book (my version has 1556 pages) focusing on an old version of the Linux kernel (2.6.x). Fear not! You'll still gain a vast knowledge about how a GNU/Linux system operates. For a quicker tour, you could have a look at the <a href="http://learnlinuxconcepts.blogspot.fr/2014/10/this-blog-is-to-help-those-students-and.html">Linux Kernel Internals</a> blog. You'll also find interesting SRE interview questions/answers in this <a href="https://syedali.net/engineer-interview-questions/">SRE interview questions</a> blogpost.</p>
<p><a href="https://jvns.ca/">Julia Evans</a>, also known as <a href="https://twitter.com/b0rk">b0rk</a> has written some absolutely <strong>fantastic</strong> beginner-friendly resources about troubleshooting and networking.
I strongly recommend having a look at:</p>
<ul>
<li><a href="http://jvns.ca/debugging-zine.pdf">the debugging zine</a></li>
<li><a href="https://jvns.ca/networking-zine.pdf">networking! ACK!</a></li>
<li><a href="http://jvns.ca/strace-zine-v2.pdf">How to spy on your programs with <code>strace</code></a></li>
</ul>
<p>Mastering the mentioned tools (<code>strace</code>, <code>tcpdump</code>, <code>netstat</code>, <code>lsof</code>, <code>ngrep</code>, etc) gave me some good debugging chops I have applied in production many times.</p>
<p>Netflix has also written a very nice and thorough blogpost on performance troubleshooting: <a href="http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html">Linux Performance Analysis in 60,000 Milliseconds</a>, detailing what to check in case of a performance issue.</p>
<h2 id="wait-theres-more">Wait, there's more</h2>
<p>Technical knowledge is one thing, but SRE being a relatively new activity, I also wanted to get real-life feedbacks from real-life SREs. To that end, I watched the following (great) talks:</p>
<ul>
<li><a href="https://www.usenix.org/conference/srecon15/program/presentation/limoncelli">Case Study: Adopting SRE Principles at StackOverflow</a>, by Tom Limoncelli of Stack Exchange</li>
<li><a href="https://www.youtube.com/watch?v=fsTpRx8Pt-k">Love DevOps? Wait until you meet SRE</a>, by Nick Wright, from Atlassian</li>
<li><a href="https://www.usenix.org/conference/srecon17americas/program/presentation/training-new-sres">Panel: training new SREs</a>, with Katie Ballinger (CircleCI), Saravanan Loganathan (Yahoo), Rita Lu (Google), Craig Sebenik (Matterport), Andrew Widdowson (Google)</li>
</ul>
<h2 id="oh-and-one-last-thing">Oh and one last thing...</h2>
<blockquote class="twitter-tweet" data-lang="fr"><p lang="en" dir="ltr">I'm super excited to announce I'm joining <a href="https://twitter.com/datadoghq">@datadoghq</a> as an SRE ! <a href="https://t.co/Ji1JJQLJ4x">pic.twitter.com/Ji1JJQLJ4x</a></p>— Balthazar Rouberol (@brouberol) <a href="https://twitter.com/brouberol/status/854620051307196417">19 avril 2017</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>Découverte de la command line Docker2016-10-13T00:00:00+02:002016-10-13T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2016-10-13:/decouverte-de-la-command-line-docker<p>Connectez vous sur la machine virtuelle installée en première partie de la journée.</p>
<p>1 - Installez docker</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>su
$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>docker.io
</code></pre></div>
<p>2 - Assurez vous que docker est correctement installé. Si c'est le cas, vous devriez avoir une sortie similaire à celle-ci:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>info
Containers:<span class="w"> </span><span class="m">20</span>
<span class="w"> </span>Running:<span class="w"> </span><span class="m">2</span>
<span class="w"> </span>Paused:<span class="w"> </span><span class="m">0 …</span></code></pre></div><p>Connectez vous sur la machine virtuelle installée en première partie de la journée.</p>
<p>1 - Installez docker</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>su
$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>docker.io
</code></pre></div>
<p>2 - Assurez vous que docker est correctement installé. Si c'est le cas, vous devriez avoir une sortie similaire à celle-ci:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>info
Containers:<span class="w"> </span><span class="m">20</span>
<span class="w"> </span>Running:<span class="w"> </span><span class="m">2</span>
<span class="w"> </span>Paused:<span class="w"> </span><span class="m">0</span>
<span class="w"> </span>Stopped:<span class="w"> </span><span class="m">18</span>
Images:<span class="w"> </span><span class="m">70</span>
Server<span class="w"> </span>Version:<span class="w"> </span><span class="m">1</span>.12.1
Storage<span class="w"> </span>Driver:<span class="w"> </span>aufs
<span class="w"> </span>Root<span class="w"> </span>Dir:<span class="w"> </span>/var/lib/docker/aufs
<span class="w"> </span>Backing<span class="w"> </span>Filesystem:<span class="w"> </span>extfs
<span class="w"> </span>Dirs:<span class="w"> </span><span class="m">153</span>
<span class="w"> </span>Dirperm1<span class="w"> </span>Supported:<span class="w"> </span><span class="nb">true</span>
Logging<span class="w"> </span>Driver:<span class="w"> </span>json-file
Cgroup<span class="w"> </span>Driver:<span class="w"> </span>cgroupfs
Plugins:
<span class="w"> </span>Volume:<span class="w"> </span><span class="nb">local</span>
<span class="w"> </span>Network:<span class="w"> </span>bridge<span class="w"> </span>host<span class="w"> </span>null<span class="w"> </span>overlay
Swarm:<span class="w"> </span>inactive
Runtimes:<span class="w"> </span>runc
Default<span class="w"> </span>Runtime:<span class="w"> </span>runc
Security<span class="w"> </span>Options:
Kernel<span class="w"> </span>Version:<span class="w"> </span><span class="m">3</span>.16.0-4-amd64
Operating<span class="w"> </span>System:<span class="w"> </span>Debian<span class="w"> </span>GNU/Linux<span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="o">(</span>jessie<span class="o">)</span>
OSType:<span class="w"> </span>linux
Architecture:<span class="w"> </span>x86_64
CPUs:<span class="w"> </span><span class="m">1</span>
Total<span class="w"> </span>Memory:<span class="w"> </span><span class="m">3</span>.779<span class="w"> </span>GiB
Name:<span class="w"> </span>gallifrey
ID:<span class="w"> </span>RYUC:5OT6:3JFQ:APQG:QEW7:V7KP:ZDYY:WBBF:LZL5:PSFY:WFEH:MF6V
Docker<span class="w"> </span>Root<span class="w"> </span>Dir:<span class="w"> </span>/var/lib/docker
Debug<span class="w"> </span>Mode<span class="w"> </span><span class="o">(</span>client<span class="o">)</span>:<span class="w"> </span><span class="nb">false</span>
Debug<span class="w"> </span>Mode<span class="w"> </span><span class="o">(</span>server<span class="o">)</span>:<span class="w"> </span><span class="nb">false</span>
Registry:<span class="w"> </span>https://index.docker.io/v1/
WARNING:<span class="w"> </span>No<span class="w"> </span>memory<span class="w"> </span>limit<span class="w"> </span>support
WARNING:<span class="w"> </span>No<span class="w"> </span>swap<span class="w"> </span>limit<span class="w"> </span>support
WARNING:<span class="w"> </span>No<span class="w"> </span>kernel<span class="w"> </span>memory<span class="w"> </span>limit<span class="w"> </span>support
WARNING:<span class="w"> </span>No<span class="w"> </span>oom<span class="w"> </span><span class="nb">kill</span><span class="w"> </span>disable<span class="w"> </span>support
WARNING:<span class="w"> </span>No<span class="w"> </span>cpu<span class="w"> </span>cfs<span class="w"> </span>quota<span class="w"> </span>support
WARNING:<span class="w"> </span>No<span class="w"> </span>cpu<span class="w"> </span>cfs<span class="w"> </span>period<span class="w"> </span>support
Insecure<span class="w"> </span>Registries:
<span class="w"> </span><span class="m">127</span>.0.0.0/8
</code></pre></div>
<p>3 - Listez les commandes disponibles:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker
</code></pre></div>
<p>4 - Listez les images disponibles localement</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>images
</code></pre></div>
<p>5 - Pullez l'image <code>ubuntu</code></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>pull<span class="w"> </span>ubuntu
</code></pre></div>
<p>6 - Listez les images disponibles localement</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>images
</code></pre></div>
<p>7 - Lancez un <code>echo "hello world!"</code> dans votre image <code>ubuntu</code></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>ubuntu<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"hello world"</span>
</code></pre></div>
<p>8 - Listez <em>tous</em> les conteneurs. Pourquoi le conteneur est-il en status <code>Exited</code>?</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>ps<span class="w"> </span>-a
</code></pre></div>
<p>9 - Pullez l'image <code>brouberol/nginx</code></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>pull<span class="w"> </span>brouberol/nginx
</code></pre></div>
<p>10 - Lancez l'image <code>brouberol/nginx</code> avec la commande suivante</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>-it<span class="w"> </span>--rm<span class="w"> </span>--name<span class="o">=</span>nginx-1<span class="w"> </span>-p<span class="w"> </span><span class="m">80</span>:80<span class="w"> </span>brouberol/nginx
</code></pre></div>
<p>11 - Lancez un <code>docker run --help</code> et tentez de comprendre les options passées dans la commande précédente</p>
<p>12 - Ouvrez un second terminal et ouvrez une session ssh dans votre VM</p>
<p>13 - Inspectez les conteneurs:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>ps
</code></pre></div>
<p>Vous devriez avoir une sortie telle que</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>ps
CONTAINER<span class="w"> </span>ID<span class="w"> </span>IMAGE<span class="w"> </span>COMMAND<span class="w"> </span>CREATED<span class="w"> </span>STATUS<span class="w"> </span>PORTS<span class="w"> </span>NAMES
76ee1a3f4ce2<span class="w"> </span>brouberol/nginx<span class="w"> </span><span class="s2">"./entrypoint.sh"</span><span class="w"> </span><span class="m">7</span><span class="w"> </span>seconds<span class="w"> </span>ago<span class="w"> </span>Up<span class="w"> </span><span class="m">6</span><span class="w"> </span>seconds<span class="w"> </span><span class="m">0</span>.0.0.0:80->80/tcp<span class="w"> </span>nginx-1
</code></pre></div>
<p>À quoi servait l'option <code>-p 80:80</code> de la commande <code>docker run</code>?</p>
<p>14 - Listez les ports hosts/conteneurs liés a votre conteneur via <code>docker port <id></code> (dans l'exemple précédent, <code><id></code> vallait <code>76ee1a3f4ce2</code>).</p>
<p>15 - Interrogez votre conteneur nginx via <code>curl localhost</code></p>
<p>16 - Recuperez l'IP publique de votre VM via <code>ip -o -4 addr show eth0 | awk '{print $4}' | cut -d'/' -f 1</code></p>
<p>17 - Ouvrez votre navigateur et visitez <code>http://<public_ip></code></p>
<p>18 - Inspectez votre conteneur via la commande <code>docker inspect</code>.</p>
<p>19 - Lancez la commande <code>docker history --no-trunc brouberol/nginx</code> et tentez de comprendre la différence entre l'image <code>nginx</code> de base et l'image <code>brouberol/nginx</code>.</p>
<p>20 - Inspectez le Dockerfile et l'entrypoint utilisés via les commandes suivantes:</p>
<ul>
<li><code>docker exec nginx-1 cat Dockerfile</code></li>
<li><code>docker exec nginx-1 cat entrypoint.sh</code></li>
</ul>
<p>21 - À quoi sert <code>docker exec</code>?</p>
<p>22 - Selon vous, pourquoi utilisons nous un <code>exec</code> dans l'entrypoint? Indice: la commande <code>docker stop</code> envoie un SIGTERM au process PID 1 du conteneur.</p>
<p>23 - Visitez <a href="https://hub.docker.com/explore/">https://hub.docker.com/explore/</a>, et tentez de deployer une autre application officiellement supportée par Docker!</p>Celery best practices2015-12-29T00:00:00+01:002015-12-29T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2015-12-29:/celery-best-practices<p>I've been programming with <a href="http://celery.readthedocs.org/">celery</a> for the last three years, and <a href="https://denibertovic.com/pages/about-me/">Deni Bertović</a>'s article about <a href="https://denibertovic.com/posts/celery-best-practices/">Celery best practices</a> has truly been invaluable to me. In time, I've also come up with my set of best practices, and I guess this blog is as good a place as any to …</p><p>I've been programming with <a href="http://celery.readthedocs.org/">celery</a> for the last three years, and <a href="https://denibertovic.com/pages/about-me/">Deni Bertović</a>'s article about <a href="https://denibertovic.com/posts/celery-best-practices/">Celery best practices</a> has truly been invaluable to me. In time, I've also come up with my set of best practices, and I guess this blog is as good a place as any to write them down.</p>
<h2 id="write-short-tasks">Write short tasks</h2>
<p>I think that a task should be as concise as possible, in order to be able to understand what it does and how it handles corner cases as quickly as possible. I personally try to follow these rules:</p>
<ul>
<li>wrap the main task logic in an object method or a function</li>
<li>make this method/function raise identified exceptions for identified corner cases and decide what is the logic for each of them</li>
<li>implement a retry mechanism only where appropriate</li>
</ul>
<p>Let's illustrate these rules with a simple example: sending an email using a 3rd party API (eg: <a href="https://mailgun.com">Mailgun</a>, <a href="https://en.mailjet.com/">Mailjet</a>, etc). Anyone having spent enough time using third party infrastructure and systems knows they should never totally rely on them: the network can fail, they can be unavailable, etc. We thus need to handle some expectable error cases and have a fallback strategy in case of an unexpected error.</p>
<p>Let's say that we have a function <code>api_send_mail</code> that does the actual API call, raising a <code>myapp.exceptions.InvalidUserInput</code> exception, in case of an HTTP client error. This exception constitutes our set of expectable exceptions, that we need to plan for. Any other exception (connection error, server HTTP error, etc) will be sent to some crash report backend, like <a href="http://getsentry.com">Sentry</a> and trigger a retry.</p>
<p>My task implementation would look something like this:</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">myproject.tasks</span> <span class="kn">import</span> <span class="n">app</span> <span class="c1"># app is your celery application</span>
<span class="kn">from</span> <span class="nn">myproject.exceptions</span> <span class="kn">import</span> <span class="n">InvalidUserInput</span>
<span class="kn">from</span> <span class="nn">utils.mail</span> <span class="kn">import</span> <span class="n">api_send_mail</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">task</span><span class="p">(</span><span class="n">bind</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">max_retries</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">send_mail</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">recipients</span><span class="p">,</span> <span class="n">sender_email</span><span class="p">,</span> <span class="n">subject</span><span class="p">,</span> <span class="n">body</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Send a plaintext email with argument subject, sender and body to a list of recipients."""</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">api_send_mail</span><span class="p">(</span><span class="n">recipients</span><span class="p">,</span> <span class="n">sender_email</span><span class="p">,</span> <span class="n">subject</span><span class="p">,</span> <span class="n">body</span><span class="p">)</span>
<span class="k">except</span> <span class="n">InvalidUserInput</span><span class="p">:</span>
<span class="c1"># No need to retry as the user provided an invalid input</span>
<span class="k">raise</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
<span class="c1"># Any other exception. Log the exception to sentry and retry in 10s.</span>
<span class="n">sentrycli</span><span class="o">.</span><span class="n">captureException</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">retry</span><span class="p">(</span><span class="n">countdown</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">exc</span><span class="o">=</span><span class="n">exc</span><span class="p">)</span>
<span class="k">return</span> <span class="n">data</span>
</code></pre></div>
<p>What the task actually does is abstracted one layer down, and almost all the rest of the task body is handling errors. I feel that it's easier to grasp the bigger picture, and that the task is easier to maintain this way.</p>
<h2 id="retry-gracefully">Retry gracefully</h2>
<p>Setting fixed countdowns for retries may not be what you want. I tend to prefer using a backoff increasing with the number of retries. This means the more a task fails, the more we have to wait until the next retry. I think this has a couple of interesting consequences:</p>
<ul>
<li>we don't hammer the external service in case of an outage,</li>
<li>it gives more time to the service to go back to normal,</li>
<li>and thus increases our overall chance of success</li>
</ul>
<p>A simple (but effective anyhow) implementation could look something like this:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">backoff</span><span class="p">(</span><span class="n">attempts</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return a backoff delay, in seconds, given a number of attempts.</span>
<span class="sd"> The delay increases very rapidly with the number of attemps:</span>
<span class="sd"> 1, 2, 4, 8, 16, 32, ...</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="mi">2</span> <span class="o">**</span> <span class="n">attempts</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">task</span><span class="p">(</span><span class="n">bind</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">max_retries</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">send_mail</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">recipients</span><span class="p">,</span> <span class="n">sender_email</span><span class="p">,</span> <span class="n">subject</span><span class="p">,</span> <span class="n">body</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Send a plaintext email with argument subject, sender and body to a list of recipients."""</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">api_send_mail</span><span class="p">(</span><span class="n">recipients</span><span class="p">,</span> <span class="n">sender_email</span><span class="p">,</span> <span class="n">subject</span><span class="p">,</span> <span class="n">body</span><span class="p">)</span>
<span class="k">except</span> <span class="n">InvalidUserInput</span><span class="p">:</span>
<span class="k">raise</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
<span class="n">sentrycli</span><span class="o">.</span><span class="n">captureException</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">retry</span><span class="p">(</span><span class="n">countdown</span><span class="o">=</span><span class="n">backoff</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">retries</span><span class="p">),</span> <span class="n">exc</span><span class="o">=</span><span class="n">exc</span><span class="p">)</span>
<span class="o">...</span>
</code></pre></div>
<h2 id="fail-fast-and-dont-block-forever">Fail fast and don't block forever</h2>
<p>One thing to remember is to <strong>always</strong> specify a timeout on I/O operations, or at least on the celery task itself. If you don't, it's possible all your tasks could block indefinitely, which would then prevent any additional task to start. In the context of the <code>send_mail</code> task, I could probably do something like this, as an API call should probably not take more than 5 seconds:</p>
<div class="highlight"><pre><span></span><code><span class="nd">@app</span><span class="o">.</span><span class="n">task</span><span class="p">(</span>
<span class="n">bind</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">max_retries</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
<span class="n">soft_time_limit</span><span class="o">=</span><span class="mi">5</span> <span class="c1"># time limit is in seconds.</span>
<span class="p">)</span>
<span class="k">def</span> <span class="nf">send_mail</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">recipients</span><span class="p">,</span> <span class="n">sender_email</span><span class="p">,</span> <span class="n">subject</span><span class="p">,</span> <span class="n">body</span><span class="p">):</span>
<span class="o">...</span>
</code></pre></div>
<p>If the task takes more than 5 seconds to complete, the <code>celery.exceptions.SoftTimeLimitExceeded</code> exception would get raised and logged to Sentry.</p>
<p>I also tend to set the <a href="https://celery.readthedocs.org/en/latest/configuration.html?highlight=eager#celeryd-task-soft-time-limit"><code>CELERYD_TASK_SOFT_TIME_LIMIT</code></a> configuration option with a default value of 300 (5 minutes). This will act as a failsafe if I forget to set an appropriate <code>soft_time_limit</code> option on a task.</p>
<h2 id="share-common-behavior-among-tasks">Share common behavior among tasks</h2>
<p>All that is pretty dandy, but I don't want to re-implement the exception catching for every task. I should be able to specify a basic behavior shared between all my tasks. Turns out you can, using an <a href="https://celery.readthedocs.org/en/latest/userguide/tasks.html?highlight=context#abstract-classes">abstract task class</a>.</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">myproject.tasks</span> <span class="kn">import</span> <span class="n">app</span>
<span class="k">class</span> <span class="nc">BaseTask</span><span class="p">(</span><span class="n">app</span><span class="o">.</span><span class="n">Task</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Abstract base class for all tasks in my app."""</span>
<span class="n">abstract</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">def</span> <span class="nf">on_retry</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc</span><span class="p">,</span> <span class="n">task_id</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">,</span> <span class="n">einfo</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Log the exceptions to sentry at retry."""</span>
<span class="n">sentrycli</span><span class="o">.</span><span class="n">captureException</span><span class="p">(</span><span class="n">exc</span><span class="p">)</span>
<span class="nb">super</span><span class="p">(</span><span class="n">BaseTask</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">on_retry</span><span class="p">(</span><span class="n">exc</span><span class="p">,</span> <span class="n">task_id</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">,</span> <span class="n">einfo</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">on_failure</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc</span><span class="p">,</span> <span class="n">task_id</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">,</span> <span class="n">einfo</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Log the exceptions to sentry."""</span>
<span class="n">sentrycli</span><span class="o">.</span><span class="n">captureException</span><span class="p">(</span><span class="n">exc</span><span class="p">)</span>
<span class="nb">super</span><span class="p">(</span><span class="n">BaseTask</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">on_failure</span><span class="p">(</span><span class="n">exc</span><span class="p">,</span> <span class="n">task_id</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">,</span> <span class="n">einfo</span><span class="p">)</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">task</span><span class="p">(</span>
<span class="n">bind</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">max_retries</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
<span class="n">soft_time_limit</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span>
<span class="n">base</span><span class="o">=</span><span class="n">BaseTask</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">send_mail</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">recipients</span><span class="p">,</span> <span class="n">sender_email</span><span class="p">,</span> <span class="n">subject</span><span class="p">,</span> <span class="n">body</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Send a plaintext email with argument subject, sender and body to a list of recipients."""</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">api_send_mail</span><span class="p">(</span><span class="n">recipients</span><span class="p">,</span> <span class="n">sender_email</span><span class="p">,</span> <span class="n">subject</span><span class="p">,</span> <span class="n">body</span><span class="p">)</span>
<span class="k">except</span> <span class="n">InvalidUserInput</span><span class="p">:</span>
<span class="k">raise</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">retry</span><span class="p">(</span><span class="n">countdown</span><span class="o">=</span><span class="n">backoff</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">retries</span><span class="p">),</span> <span class="n">exc</span><span class="o">=</span><span class="n">exc</span><span class="p">)</span>
<span class="k">return</span> <span class="n">data</span>
</code></pre></div>
<p>You can see that the <code>send_mail</code> task implementation only deals with email sending and expected error handling. Everything else is handled by the abstract base class. If the common behavior is more complex, this trick can <em>drastically</em> reduce the size of each task body and the amount of duplicated code in your tasks.</p>
<p><strong>Note</strong>: this example is only here to demonstrate how to share behavior between tasks. To properly integrate Sentry with Celery, have a look at <a href="https://docs.getsentry.com/hosted/clients/python/integrations/celery/">this page</a>.</p>
<p><strong>Tip</strong>: have a look at the list of <a href="https://celery.readthedocs.org/en/latest/userguide/tasks.html?highlight=context#handlers">available handlers</a>, to get an idea of what behavior can be shared between tasks.</p>
<h2 id="write-large-tasks-as-classes">Write large tasks as classes</h2>
<p>So far, I've only implemented tasks as functions. However, it's also possible to define <a href="https://celery.readthedocs.org/en/latest/userguide/tasks.html#custom-task-classes">class tasks</a>.</p>
<p>I think one of the scenarii where class tasks really shine are when you'd like to split a large task function into several well-defined and testable methods. As you can see <a href="https://celery.readthedocs.org/en/latest/userguide/tasks.html#custom-task-classes">here</a>, the <code>celery.task</code> decorator will generate a task class and inject the decorated function as the class <code>run</code> method.
Defining a class task amounts to defining a class inheriting from <code>app.Task</code> with a <code>run</code> method.</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">handle_event</span><span class="p">(</span><span class="n">BaseTask</span><span class="p">):</span> <span class="c1"># BaseTask inherits from app.Task</span>
<span class="k">def</span> <span class="nf">validate_input</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">get_or_create_model</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">stream_event</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">validate_intput</span><span class="p">(</span><span class="n">event</span><span class="p">):</span>
<span class="k">raise</span> <span class="n">InvalidInput</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">model</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_or_create_model</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">call_hooks</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">persist_model</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">retry</span><span class="p">(</span><span class="n">countdown</span><span class="o">=</span><span class="n">backoff</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">retries</span><span class="p">),</span> <span class="n">exc</span><span class="o">=</span><span class="n">exc</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">stream_event</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
</code></pre></div>
<p>By doing this, the task logic is clear and easy to follow (the <code>run</code> method stays concise even if the methods body are large), and each of these method can then be unit-tested independently.</p>
<p>Another advantage of using class tasks is using multiple inheritance to specialize a task with multiple abstract base classes.
For example, I'd like to use the <a href="https://github.com/TrackMaven/celery-once/">celery_once</a> <code>QueueOnce</code> abstract class to introduce some locking mechanism, while still using the <code>BaseTask</code> for sentry logging. This way, each abstract task class is used as a mixin, adding some behaviour to the task.</p>
<h2 id="unit-test-your-tasks">Unit-test your tasks</h2>
<p>Unit testing a project involving celery has always been a pickle for me. I tried to deploy a broker and a test celery worker in the CI environment, but it felt like killing a fly with a bazooka. The answer turns out to be quite simple, thanks to Nicolas Le Manchet for figuring this one out! When the <a href="https://celery.readthedocs.org/en/latest/configuration.html#celery-always-eager"><code>CELERY_ALWAYS_EAGER</code></a> option is activated, all tasks called using their <code>apply_async</code> or <code>delay</code> method are called <em>directly</em>, without requiring any broker or celery worker. Easy as pie.</p>Installing Guitar Pro 6 on Fedora 22+2015-12-06T00:00:00+01:002015-12-06T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2015-12-06:/installing-guitar-pro-6-on-fedora-22<p>I've been playing guitar for the last 10 years now, but I spent the last 4 years only playing and singing alone. I decided to improve my technique, and to treat me with <a href="http://www.guitar-pro.com/en/index.php?pg=guitar-pro-6">Guitar Pro 6</a>. I was happy to see they even supported Linux natively! Sadly, they only provide …</p><p>I've been playing guitar for the last 10 years now, but I spent the last 4 years only playing and singing alone. I decided to improve my technique, and to treat me with <a href="http://www.guitar-pro.com/en/index.php?pg=guitar-pro-6">Guitar Pro 6</a>. I was happy to see they even supported Linux natively! Sadly, they only provide a deb file, and no rpm. I'll thus describe here how I managed to install it on Fedora 22, ♬ <em>with a little help from my friends</em> ♫.</p>
<p><img alt="GuitarPro screenshot" decoding="async" loading="lazy" src="https://i.ytimg.com/vi/9TRyKjXpfZo/maxresdefault.jpg?time=1495551778029"></p>
<h3 id="installing-the-dependencies">Installing the dependencies</h3>
<p>First, download the Guitar Pro deb file. Mine was called <code>gp6-full-linux-r11686.deb</code>
Extract the archive called <code>data.tar.gz</code> from the deb, and then de-archive it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/tmp
$<span class="w"> </span>mv<span class="w"> </span>~/Downloads/gp6-full-linux-r11686.deb<span class="w"> </span>.
$<span class="w"> </span>ar<span class="w"> </span>vx<span class="w"> </span>gp6-full-linux-r11686.deb
$<span class="w"> </span>tar<span class="w"> </span>-xvf<span class="w"> </span>data.tar.gz
</code></pre></div>
<p>Create the installation directory for Guitar Pro.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>mkdir<span class="w"> </span>-p<span class="w"> </span>/opt/GuitarPro6
</code></pre></div>
<p>Move the GuitarPro files to the installation directory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>mv<span class="w"> </span>./opt/GuitarPro6/<span class="w"> </span>/opt/GuitarPro6/
</code></pre></div>
<p>We now need to install Guitar Pro's dependencies, and of course, they're 32 bit...</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>dnf<span class="w"> </span>install<span class="w"> </span>libICE.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>libSM.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>libssh.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>libxml2.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>libxslt.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>libpng12.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>libvorbis.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>alsa-lib.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>portaudio.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>pulseaudio-libs.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>qt-x11.i686<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>qtwebkit.i668
</code></pre></div>
<p>You might have to download other packages as well, as Guitar Pro was not my first 32 bit program I had to install.</p>
<p><em>Note</em>: The required packages will be listed when you execute the <code>/opt/GuitarPro6/launcher.sh</code> script, and you can use the <code>dnf whatprovides</code> command to find the package that provides each required library.</p>
<p>Sadly, that's not it yet. GuitarPro also depends on both <code>libcrypto</code> and <code>libssl</code> 0.9.8, and they're not packaged anymore in Fedora 22.
We'll use a very cool trick: by a great stroke of luck, the libraries contained in the <code>openssl</code> Ubuntu deb package are usable as a drop-in replacement!</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>wget<span class="w"> </span>-q<span class="w"> </span>http://security.ubuntu.com/ubuntu/pool/universe/o/openssl098/libssl0.9.8_0.9.8o-7ubuntu3.2.14.04.1_i386.deb<span class="w"> </span><span class="m">1</span>>/dev/null
$<span class="w"> </span>ar<span class="w"> </span>x<span class="w"> </span>libssl0.9.8_0.9.8o-7ubuntu3.2.14.04.1_i386.deb<span class="w"> </span>data.tar.xz
$<span class="w"> </span>tar<span class="w"> </span>--strip-components<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-xf<span class="w"> </span>data.tar.xz<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>./lib/i386-linux-gnu/libcrypto.so.0.9.8<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>./lib/i386-linux-gnu/libssl.so.0.9.8
$<span class="w"> </span>chmod<span class="w"> </span><span class="m">755</span><span class="w"> </span>libssl.so.0.9.8<span class="w"> </span>libcrypto.so.0.9.8
$<span class="w"> </span>mv<span class="w"> </span>libssl.so.0.9.8<span class="w"> </span>libcrypto.so.0.9.8<span class="w"> </span>/opt/GuitarPro6
</code></pre></div>
<p>Seriously, how cool is that <sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>?</p>
<p><em>Note</em>: I moved both shared libraries to the <code>/opt/GuitarPro6</code> directory, because it also contained numerous other ones. My guess, which turned out to be correct, was that the executable uses will look for shared objects in its directory. This way, I didn't have to fiddle with <code>LD_LIBRARY_PATH</code> and <code>ldconfig</code>, or install these libraries into my <code>/usr/lib</code> folder.</p>
<h3 id="installing-the-sound-banks">Installing the sound banks</h3>
<p>We now need to install the sound banks. First, download them from the official website. Then, install them via the <code>/opt/GuitarPro6/GPBankInstaller</code> script:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>mv<span class="w"> </span>Banks-r370.gpbank<span class="w"> </span>/opt/GuitarPro6
$<span class="w"> </span>sudo<span class="w"> </span>/opt/GuitarPro6/GPBankInstaller<span class="w"> </span>/opt/GuitarPro6/Soundbanks.gpbank<span class="w"> </span>/opt/GuitarPro6/Data/Soundbanks/
</code></pre></div>
<h3 id="make-guitar-pro-the-default-program-for-tab-files">Make Guitar Pro the default program for tab files</h3>
<p>We then install the desktop and icon file that were packaged in the Guitar Pro deb, so that it can be executed from the app launcher.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>cp<span class="w"> </span>./usr/share/applications/GuitarPro6.desktop<span class="w"> </span>/usr/share/applications/GuitarPro6.desktop
$<span class="w"> </span>sudo<span class="w"> </span>cp<span class="w"> </span>./usr/share/pixmaps/guitarpro6.png<span class="w"> </span>/usr/share/pixmaps/
</code></pre></div>
<p>The last and final step is the cherry on the cake: we're going to make Guitar Pro open automatically when opening the tab files. We first define the <code>application/x-guitar-pro</code> MIME-type and then associate it with the <code>gp3</code>, <code>gp4</code>, <code>gp5</code> and <code>gp6</code> extensions.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span><span class="s"><<EOF > guitarpro-mime.xml</span>
<span class="s"><?xml version="1.0"?></span>
<span class="s"> <mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info"></span>
<span class="s"> <mime-type type="application/x-guitar-pro"></span>
<span class="s"> <glob pattern="*.gp3"/></span>
<span class="s"> <glob pattern="*.gp4"/></span>
<span class="s"> <glob pattern="*.gp5"/></span>
<span class="s"> <glob pattern="*.gp6"/></span>
<span class="s"> </mime-type></span>
<span class="s"></mime-info></span>
<span class="s">EOF</span>
$<span class="w"> </span>sudo<span class="w"> </span>xdg-mime<span class="w"> </span>install<span class="w"> </span>guitarpro-mime.xml
$<span class="w"> </span>xdg-mime<span class="w"> </span>default<span class="w"> </span>GuitarPro6.desktop<span class="w"> </span>application/x-guitar-pro
</code></pre></div>
<p>Done. You should now be able to click on a tab file, and enjoy!</p>
<h3 id="conclusion">Conclusion</h3>
<p>I managed to make everything work, with both some help and luck. I would however have prefered if the Guitar Pro binary had been compiled statically, to ease the installation process.</p>
<p>Also, when you advertise Linux compatibility, please, PLEASE, at least mention the package format (deb, rpm, other), and also mention the distributions you support natively.</p>
<p>Finally, when you want to support Linux, do not <strong>ever</strong> redirect Linux users to the Windows installation guide, by stating that both processes are <a href="https://support.guitar-pro.com/hc/fr/articles/201863332-GP6-Je-viens-d-acheter-Guitar-Pro-6-que-dois-je-faire-">"substantially similar"</a>. They are not.</p>
<p>On that note (get it?), keep on rocking in a free world!</p>
<h3 id="sources">Sources</h3>
<ul>
<li><a href="https://www.linux.com/community/blogs/128-desktops/494464">https://www.linux.com/community/blogs/128-desktops/494464</a></li>
<li><a href="https://github.com/dpurgin/guitarpro6-rpm/">https://github.com/dpurgin/guitarpro6-rpm/</a></li>
<li><a href="https://stackoverflow.com/questions/30931/register-file-extensions-mime-types-in-linux">https://stackoverflow.com/questions/30931/register-file-extensions-mime-types-in-linux</a></li>
</ul>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>We can of course try to include them from other <a href="http://www.rpmfind.net/linux/rpm2html/search.php?query=openssl-devel+0.9.8&submit=Search+...&system=&arch=">rpm-based distributions</a>, but I have to admit I found it cooler this way. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Pain au levain, deuxième essai2015-11-14T00:00:00+01:002015-11-14T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2015-11-14:/pain-au-levain-deuxieme-essai<p>Nouvel essai, nouvelle réussite! Cette fois, j'ai testé deux nouvelles techniques (l'autolyse et le pétrissage <em>stretch and fold</em>) et j'ai augmenté la proportion de farine complète.</p>
<p><img alt="Hmm" decoding="async" loading="lazy" src="images/2015-11-14 01.36.43_preview.jpeg"></p>
<h2 id="ingredients">Ingrédients</h2>
<ul>
<li>200g de levain</li>
<li>360ml d'eau</li>
<li>600g de farines (430g de T55, 50g de T65 et 120g de T110)</li>
<li>10g de sel</li>
</ul>
<p>Rien de …</p><p>Nouvel essai, nouvelle réussite! Cette fois, j'ai testé deux nouvelles techniques (l'autolyse et le pétrissage <em>stretch and fold</em>) et j'ai augmenté la proportion de farine complète.</p>
<p><img alt="Hmm" decoding="async" loading="lazy" src="images/2015-11-14 01.36.43_preview.jpeg"></p>
<h2 id="ingredients">Ingrédients</h2>
<ul>
<li>200g de levain</li>
<li>360ml d'eau</li>
<li>600g de farines (430g de T55, 50g de T65 et 120g de T110)</li>
<li>10g de sel</li>
</ul>
<p>Rien de révolutionnaire ici. J'ai légèrement baissé le volume d'eau par rapport au <a href="/pain-au-levain-premier-essai">premier essai</a> et augmenté le poids en farine semi-complète.</p>
<h2 id="recette">Recette</h2>
<h3 id="autolyse">Autolyse</h3>
<p>C'est ici que j'ai innové. Au lieu de simplement mélanger les ingrédients et de pétrir, je suis passé par deux phases d'<a href="http://www.abreadaday.com/the-autolyse-method/">autolyse</a>. J'ai d'abord mélangé l'eau et la farine, et attendu 20 minutes. J'ai ensuite rajouté le levain, et attendu 20 minutes. Cette technique favorise la formation de chaînes de gluten (qui donnent à la pâte son élasticité) sans trop oxyder la pâte, ce qui évite de se retrouver avec un pain fade en sortie. Les partisans de l'autolyse insistent sur le fait que ça a une énorme influence sur le goût final.</p>
<h3 id="stretch-and-fold">Stretch and fold</h3>
<p>Depuis que j'ai commencé à m'amuser à faire du pain, ma technique de pétrissage a toujours consisté à travailler la pâte avec la paume de la main en la pliant sur elle même. J'ai souvent trouvé que mes pains n'étaient pas assez aérés. J'ai récemment entendu parler d'une technique radicalement différente, mise au point par <a href="https://www.youtube.com/watch?v=kXV8mayG3W0">Richard Bertinet</a>, permettant de travailler des pâtes à fort taux d'hydratation (plus de 80%!) tout en conservant son aération.</p>
<p>L'idée est simple (comme montré dans cette <a href="https://www.youtube.com/watch?v=VrcTHcLQ_GM">vidéo</a>): on étire la pâte, plus la replie en 3, encore en deux. On forme ensuite une boule, et on <strong>attend</strong> pendant 50 minutes, avant de recommencer. Pendant ces 50 minutes, la pâte va travailler pour nous en formant les fameuses chaînes de gluten. Le résultat est assez miraculeux: la pâte est bien plus ferme après la première opération, et parfaitement travaillable après la deuxième, sans <strong>aucune</strong> phase de pétrissage!</p>
<p>L'opération est à répéter 2 ou 3 fois selon les sources. J'ai tenté 2 fois (stretch and fold, 50 minutes d'attente, stretch and fold, 50 minutes d'attente).</p>
<h3 id="pointage">Pointage</h3>
<p>Donner à la pâte la forme voulue, et laisser reposer 2 heures. Une fois ce temps écoulé, fariner le pâton et entailler la pâte avec un couteau tranchant, ce qui aidera la pâte à gonfler à la cuisson.</p>
<h3 id="cuisson">Cuisson</h3>
<p>Même technique que <a href="/pain-au-levain-premier-essai">la dernière fois</a>: préchauffer le four à 240°C, enfourner quand le four est bien chaud pendant 22 minutes, et ensuite 25 minutes à 220°C. Régulièrement verser de l'eau au fond du four, afin de créer de la vapeur, qui favorisera la formation de croûte.</p>
<h3 id="repos">Repos</h3>
<p>Laisser le pain refroidir sur une grille pour que l'humidité interne disparaisse.</p>
<h2 id="resultat">Résultat</h2>
<p><img alt="Hmm" decoding="async" loading="lazy" src="images/2015-11-14 09.43.35_preview.jpeg"></p>
<p>Le résultat n'est pas aussi aéré que ce que j'espérais, mais c'est déjà beaucoup mieux! Et c'est délicieux!</p>Pain au levain, premier essai2015-11-06T00:00:00+01:002015-11-06T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2015-11-06:/pain-au-levain-premier-essai<p>Mon premier essai de pain au levain étant, en toute modestie, particulièrement réussi, je consigne ici la recette.</p>
<p><img alt="Hmm" decoding="async" loading="lazy" src="images/2015-11-09 22.32.01_preview.jpeg"></p>
<h2 id="le-levain">Le levain</h2>
<p>Tout d'abord, le levain. J'ai suivi cette <a href="http://www.chefnini.com/levain-naturel-maison/">recette</a>, ainsi que <a href="http://www.marmiton.org/magazine/tendances-gourmandes_l-aventure-du-levain-naturel-maison_4.aspx">celle-ci</a>. Au bout de presque 2 semaines, mon levain (Gérard) est prêt à être intégré dans du pain: il est …</p><p>Mon premier essai de pain au levain étant, en toute modestie, particulièrement réussi, je consigne ici la recette.</p>
<p><img alt="Hmm" decoding="async" loading="lazy" src="images/2015-11-09 22.32.01_preview.jpeg"></p>
<h2 id="le-levain">Le levain</h2>
<p>Tout d'abord, le levain. J'ai suivi cette <a href="http://www.chefnini.com/levain-naturel-maison/">recette</a>, ainsi que <a href="http://www.marmiton.org/magazine/tendances-gourmandes_l-aventure-du-levain-naturel-maison_4.aspx">celle-ci</a>. Au bout de presque 2 semaines, mon levain (Gérard) est prêt à être intégré dans du pain: il est vif, crisse sous le doigt et a une bonne (et douce) odeur de fermentation. Notez que j'ai fait mon levain à base de farine T85 (de la bonne grosse farine dégrossie), avec de temps en temps une goûte de miel bio.</p>
<h2 id="le-pain">Le pain</h2>
<p>J'ai suivi ici les conseils de Clément (déjà pas le dernier dans le domaine): une mesure de levain, deux mesures d'eau et trois mesures de farines. Notez le "s" à "farines": j'ai mis 5/6e de farine T55 et le 1/6e restant de farine semi-complète T110.</p>
<h3 id="recette-et-ingredients">Recette et ingrédients</h3>
<ul>
<li>200g de levain</li>
<li>400g d'eau (compter 350-360g d'eau si on pétrit à la main. Mes voisins m'ont entendu gueuler parce que la pâte était trop collante, ce qui m'a forcé à rajouter beaucoup de farine au pétrissage).</li>
<li>600g de farines (500g de T55 et 100g de T110)</li>
</ul>
<p>Pétrir pendant une bonne quinzaine de minutes. Afin d'incorporer le maximum d'air possible, plier la pâte sur elle même, et éviter de trop la faire dégazer en appuyant dessus comme une brute. Laisser reposer la pâte pendant une heure ou deux. Une fois reposée, la pâte aura gonflé. Certaines recettes vous diront que le pâton aura double voire triplé de taille, mais ça va dépendre de la quantité de farine complète incorporée. En effet, elle est beaucoup plus lourde (parce que comportant plus de son, l'écorce du blé) que la farine blanche, donc lève moins.</p>
<p>La plupart des recettes vous conseillent d'attendre entre 6 et 8h à ce stade là. Pour mon premier essai, vu qu'il était déjà 22h, j'ai décidé de m'en balancer et d'y aller.</p>
<p>Je fait préchauffer mon four à 240°C, avec une tasse d'eau au fond du four. Une fois la pâte levée, je la façonne, afin de lui donner la forme finale générale. Je farine d'abord ma plaque de cuisson. J'aime beaucoup les boules de campagne, du coup je ramène un peu la pâte sous elle même, afin d'avoir une belle boule sur mon plaque. Je farine le dessus de la boule, et incise la pâte au couteau d'une croix, d'environ un cm de profondeur. Au four!</p>
<p>Au bout de 10-15 minutes, je verse le contenu de la tasse au fond du four, afin de libérer de la vapeur d'eau, qui va aider à la formation de la croûte. Au bout de 20-25 minutes de cuisson à 240°C, je baisse le four à 220°C pour encore 20 minutes de cuisson. Une fois le pain sorti du four, je le laisse reposer plusieurs heures sur une grille, afin que l'humidité s'échappe bien. Je suis un grand adepte de manger le pain à la sortie du four, mais avec de telles quantités, c'est l'effet caoutchouteux garanti...</p>
<h3 id="le-resultat-le-lendemain">Le résultat le lendemain</h3>
<p><img alt="Hmm" decoding="async" loading="lazy" src="images/2015-11-10 07.35.34_preview.jpeg">
Le pain est très bon, avec un léger goût acide donné par le levain. J'aimerais avoir des bulles un peu plus grosses la prochaine fois, mais je crois qu'il faut que je retravaille ma technique de pétrissage, qui n'incorpore pas assez d'air, et surtout que je laisse la pâte gonfler bien plus longtemps (voire toute la nuit).</p>
<p>Peut-être aussi mettre un quart ou un tiers de farine semi-complète, pour lui donner un goût plus brut, auquel cas il faudra faire gaffe à ce que la pâte réussisse quand même à gonfler (augmenter l'hydratation?).</p>My n-step plan to become a better programmer2015-05-24T00:00:00+02:002015-05-24T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2015-05-24:/my-n-step-plan-to-become-a-better-programmer<p>One of the main selling points of Python are its multi-paradigm philosophy. You can code in imperative, object-oriented or aspect-oriented style, use meta-programming techniques, etc. It also has an immense amount of libraries available. Finally, it's both a simple language to pick up for beginners, and a powerful language for …</p><p>One of the main selling points of Python are its multi-paradigm philosophy. You can code in imperative, object-oriented or aspect-oriented style, use meta-programming techniques, etc. It also has an immense amount of libraries available. Finally, it's both a simple language to pick up for beginners, and a powerful language for more experienced programmers.</p>
<p>I've been programming for the last six or seven years, and I feel that my main strength is also my main weakness: I've been mainly coding in Python since the beginning. It means that I can now use Python's features and standard library pretty well, but it also means that I tend to think of every problem in terms of Python features and libraries (standard or not).</p>
<p>A proverb programmers are taught quite early is</p>
<blockquote>
<p>If all you have is a hammer, everything looks like a nail.
(<a href="https://en.wiktionary.org/wiki/if_all_you_have_is_a_hammer,_everything_looks_like_a_nail">Source</a>)</p>
</blockquote>
<p>It means that if you're only comfortable with a single tool, then you'll try to use it in every situation, even in one where it's not appropriate. I strongly feel that to become a better programmer, I now need to learn other programming languages and even other paradigms. I was initially thinking of functional languages, like <a href="https://www.haskell.org/">Haskell</a> or <a href="http://ocaml.org/">oCaml</a>, but then I remembered something <a href="http://blaag.haard.se/">Fredrik</a> told me a while ago, at a EuroPython after-party: reading "Structure and Interpretation of Computer Programs" immediately made him a better programmer. I remember being curious as to why.</p>
<p>It so happens that the books is written under a Creative Common license, and can be downloaded <a href="https://github.com/ieure/sicp/downloads">here</a>, AND uses <a href="https://en.wikipedia.org/wiki/Scheme_%28programming_language%29">Scheme</a> as a teaching language. It thus combines three things I strive for: a new language, a new programming paradigm and more insight into the art of programming itself.</p>
<p>I'm thus laying out my n-step plan to become a better programmer:</p>
<ol>
<li>Read the book thoroughly</li>
<li>Solve the exercises</li>
<li>Stop conceiving every solution in Python</li>
</ol>
<p>Behold, one of my first Scheme programs, a pavement in the road of my improvement.</p>
<div class="highlight"><pre><span></span><code><span class="c1">; Implementation of cubic root Newton approximation technique in Scheme</span>
<span class="p">(</span><span class="k">define</span><span class="w"> </span><span class="p">(</span><span class="nf">square</span><span class="w"> </span><span class="nv">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nb">*</span><span class="w"> </span><span class="nv">x</span><span class="w"> </span><span class="nv">x</span><span class="p">))</span>
<span class="p">(</span><span class="k">define</span><span class="w"> </span><span class="p">(</span><span class="nf">cubic-root</span><span class="w"> </span><span class="nv">x</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="k">define</span><span class="w"> </span><span class="p">(</span><span class="nf">improve</span><span class="w"> </span><span class="nv">guess</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nb">/</span><span class="w"> </span><span class="p">(</span><span class="nb">+</span><span class="w"> </span><span class="p">(</span><span class="nb">/</span><span class="w"> </span><span class="nv">x</span><span class="w"> </span><span class="p">(</span><span class="nf">square</span><span class="w"> </span><span class="nv">guess</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="nb">*</span><span class="w"> </span><span class="nv">guess</span><span class="w"> </span><span class="mi">2</span><span class="p">))</span><span class="w"> </span><span class="mi">3</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="k">define</span><span class="w"> </span><span class="p">(</span><span class="nf">good-enough?</span><span class="w"> </span><span class="nv">new-guess</span><span class="w"> </span><span class="nv">old-guess</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nb"><</span><span class="w"> </span><span class="p">(</span><span class="nb">abs</span><span class="w"> </span><span class="p">(</span><span class="nb">/</span><span class="w"> </span><span class="p">(</span><span class="nb">-</span><span class="w"> </span><span class="nv">new-guess</span><span class="w"> </span><span class="nv">old-guess</span><span class="p">)</span><span class="w"> </span><span class="nv">old-guess</span><span class="p">))</span><span class="w"> </span><span class="mf">0.001</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="k">define</span><span class="w"> </span><span class="p">(</span><span class="nf">try</span><span class="w"> </span><span class="nv">new-guess</span><span class="w"> </span><span class="nv">old-guess</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nf">good-enough?</span><span class="w"> </span><span class="nv">new-guess</span><span class="w"> </span><span class="nv">old-guess</span><span class="p">)</span>
<span class="w"> </span><span class="nv">new-guess</span>
<span class="w"> </span><span class="p">(</span><span class="nf">try</span><span class="w"> </span><span class="p">(</span><span class="nf">improve</span><span class="w"> </span><span class="nv">new-guess</span><span class="p">)</span><span class="w"> </span><span class="nv">new-guess</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="nf">try</span><span class="w"> </span><span class="mf">1.0</span><span class="w"> </span><span class="nv">x</span><span class="p">)</span>
<span class="p">)</span>
<span class="p">(</span><span class="nf">cubic-root</span><span class="w"> </span><span class="mi">9</span><span class="p">)</span>
<span class="c1">; => 2.0800838232385224</span>
</code></pre></div>
<p>Note: If you want to experiment with various languages (Scheme included) without having to install them on your machine, have a look at <a href="http://repl.it/languages">repl.it</a>.</p>So long and thank you for all the (deep fried) fish!2015-01-24T00:00:00+01:002015-01-24T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2015-01-24:/so-long-and-thank-you-for-all-the-deep-fried-fish<p>J'ai délaissé ce blog depuis longtemps. Je pourrais vous dire que j'avais plein d'autres trucs à faire, mais en vérité, j'avais bien le temps d'écrire une ligne ou deux. Mon séjour se termine, plus tôt que prévu, pour deux raisons majeures: mon boulot est devenu chiant à mourir, et ma …</p><p>J'ai délaissé ce blog depuis longtemps. Je pourrais vous dire que j'avais plein d'autres trucs à faire, mais en vérité, j'avais bien le temps d'écrire une ligne ou deux. Mon séjour se termine, plus tôt que prévu, pour deux raisons majeures: mon boulot est devenu chiant à mourir, et ma vie lyonnaise me manque.</p>
<p>Felix et Thais sont venus me voir la semaine dernière, et j'ai passé le week-end dans les rues, à leur faire découvrir la ville, et tous les recoins que les touristes ne voient jamais. Je me suis rendu compte que malgré mon départ précipité, je commence à <em>vraiment</em> bien là connaître, cette ville! Si je devais vous la décrire en une ligne, je pense que je dirais que c'est une ville avec des chauffeurs de bus, un château et des gens qui baladent leur chien.</p>
<p>Par beaucoup d'aspects, c'est un vrai dépaysement: les gens se déplacent des quatre coins de la ville pour aller faire des courses dans le seul (<em>minuscule</em>) marché de la ville, alors qu'à Lyon, on peut trouver des immenses marchés dans chaque quartier. Avouer ne pas supporter les chiens est passible de la peine capitale, et une grande majorité des locaux disent ne pas avoir goûté le haggis (qui les dégoûte), et ne pas oser prendre un <a href="https://en.wikipedia.org/wiki/Deep-fried_Mars_bar"><em>deep fried mars bar</em></a>. Ils ont vraiment tort à propos du haggis, mais pas pour le coup du mars. C'est un coup à ce que votre infarctus se tape un arrêt cardiaque! Sachez que le modèle <em>deep fried</em> se décline aussi en demi-pizza, en pain à l'ail et autres douceurs et légèretés. Ce qui est assez incroyable, c'est que le nombre d'obèses soit environ égal à 0. Ça doit être le coup du froid, c'est pas possible...</p>
<p>Je confirme aussi que l'Écosse est toujours un grand pays pour le whisky. J'ai appris à vraiment aimer cette boisson, qui est vraiment incroyable en terme de variété des saveurs, des odeurs et des couleurs. Bien plus que le vin je trouve. Je comprends enfin le plaisir qu'on les gens à parler pendant des heures de ce qu'ils ont (ou ont eu) dans leur verre. C'est juste que j'étais ignare en fait: pour parler, il faut déjà avoir du vocabulaire. J'ai vraiment découvert ça avec le whisky.</p>
<p>Je me suis vraiment senti chez moi partout, sauf chez moi. C'est principalement dû au fait que mon appart est un couloir, et ma voisine du dessus (ma propriétaire) se croit tout permis. Du coup, j'ai passé mon temps à être le plus silencieux possible, histoire qu'elle me lâche la grappe, ou alors à sortir (histoire qu'elle me lâche la grappe). Je peux vous dire qu'entre elle, et l'équipe que j'ai du diriger (composée d'une folle clinique, d'un superactif probablement drogué et d'un inutile amorphe), je reviens avec un cuir bien épais. Annonce générale à tous ceux qui auraient dans leurs plans de m'emmerder: <em>"don't"</em>.</p>
<p>Je vais rajouter des photos dans la <a href="http://images.brouberol.imap.cc/edinburgh/">galerie</a>, histoire que vous puissiez vous mettre un peu à ma place. Attention, pas trop longtemps, vous risqueriez de chopper froid.</p>
<p>Je vous embrasse, et on se reverra sur le sol gaulois.</p>
<p>B.</p>Jour J et pantalon moule fesses2014-12-08T00:00:00+01:002014-12-08T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2014-12-08:/jour-j-et-pantalon-moule-fesses<p>Grande excitation au boulot: le jour J est finalement arrivé. Personne n'est mort, et on a eu environ 500% de trafic en plus que prévu. Du coup, on est plutôt contents. Ça veut aussi dire que d'ici les vacances de Noël, on met un dernier coup de collier, et ensuite …</p><p>Grande excitation au boulot: le jour J est finalement arrivé. Personne n'est mort, et on a eu environ 500% de trafic en plus que prévu. Du coup, on est plutôt contents. Ça veut aussi dire que d'ici les vacances de Noël, on met un dernier coup de collier, et ensuite, Fredrik et moi sommes de support pendant 3 mois. Autant dire que ça va pas être la même ambiance: DONDE ESTA LA PLAYA? Au bureau.</p>
<p>J'ai souvent regardé de haut ceux qui avaient du matos super pointu pour faire un truc de tous les jours. Comme courir par exemple. La réflexion était la suivante: "Pff, ça fait 20000 ans qu'on court dans tous les sens, mais soudainement, il faut absolument un pantalon qui moule les fesses et un T-shirt bleu électrique à manche longues". Autant le dédain peut être justifiable sous nos latitudes clémentes, mais allez essayer de courir 45 minutes à 9h du soir en décembre en Écosse, et vous m'en direz des nouvelles. Tout d'un coup, les gens équipés me paraissent moins ridicules. Sensés même! Alors voilà, comme quoi on change en voyage. On s'achète des pantalons moule fesses, des T-shirts bleu électrique à manches longues, des chaussures avec des orteils, et on va courir la nuit en forêt en espérant ne pas tomber sur un sanglier.</p>
<p>Avis à tous ceux qui pensent que je passe mes journées à boire des coups, je vous embrasse!</p>Julie2014-11-30T00:00:00+01:002014-11-30T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2014-11-30:/julie<p>Julie est venue me voir ce weekend. Elle.</p>
<p>Sur ce chantage affectif quasi assumé, je vous embrasse, et je vous vois à Noël!</p>Rock, Roll et thé à la menthe2014-11-23T00:00:00+01:002014-11-23T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2014-11-23:/rock-roll-et-the-a-la-menthe<p>Désolé pour le long silence, j'ai été pas mal occupé ces temps-ci. J'ai bien essayé d'embaucher des plumes, comme Alexandre Dumas, mais figurez vous qu'ils ne parlent pas la langue ces cons là. Ça en deviendrait limite frustrant à la fin.</p>
<p>Premièrement, j'ai mis quelques photos du coin <a href="http://images.brouberol.imap.cc/edinburgh/">ici</a>. C'est …</p><p>Désolé pour le long silence, j'ai été pas mal occupé ces temps-ci. J'ai bien essayé d'embaucher des plumes, comme Alexandre Dumas, mais figurez vous qu'ils ne parlent pas la langue ces cons là. Ça en deviendrait limite frustrant à la fin.</p>
<p>Premièrement, j'ai mis quelques photos du coin <a href="http://images.brouberol.imap.cc/edinburgh/">ici</a>. C'est un petit vrac, mais c'est trié du plus ancien au plus récent. Je confirme mes premières impressions: la ville est à la fois accueillante et et très vivante. Et j'ai eu la bonne idée de ne pas y habiter. Au lieu de ça, j'ai trouvé un studio perdu près d'une forêt, de champs et d'une rivière. Et vous savez quoi? C'est vraiment chouette. J'ai pu me remettre à la course, de jour comme de nuit (lampe torche oblige) et je suis relativement tranquille (sauf quand ma logeuse essaye de me soutirer du pognon). En fait, je me suis organisé une petite vie tranquillou. Quand je ne bosse pas, on sort manger un bout, on se balade, on regarde des Doctor Who en buvant du thé ou alors je vais courir. C'est pas aussi rock and roll que j'avais imaginé au début, mais c'est finalement très chouette.</p>
<p>Bon, je ne vais pas vous mentir, je ne suis pas devenu moine pour autant: les bières sont très bonnes, et le haggis aussi hein! Faudrait pas non plus déconner.</p>
<p>Je rentrerais à Noël finalement, du coup et d'ici là, je vous embrasse.</p>Ovfbhf Cncn2014-11-03T00:00:00+01:002014-11-03T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2014-11-03:/ovfbhf-cncn<p>Je pense qu'une des plus belles surprises de ce voyage, en dehors du hamburger camembert/oignons confits, c'est de m'être rendu compte que je suis bon dans ce que je fais. Loin de moi l'idée de me faire mousser en famille, mais c'est quand même une belle prise de conscience …</p><p>Je pense qu'une des plus belles surprises de ce voyage, en dehors du hamburger camembert/oignons confits, c'est de m'être rendu compte que je suis bon dans ce que je fais. Loin de moi l'idée de me faire mousser en famille, mais c'est quand même une belle prise de conscience! Ça m'a fait me demander comment j'en suis arrivé à faire de l'informatique mon métier.</p>
<p>Petit retour en arrière <a href="http://img2.wikia.nocookie.net/__cb20100221013911/stargate/images/6/65/Wormhole_Moving.gif">à travers l'espace-temps</a>.</p>
<p>L'informatique, c'est d'abord un truc de famille. Des bouquins techniques qui traînent sur le piano de la maison de Béthisy, des carcasses de PC entassées dans le grenier, mon premier PC que j'ai monté avec papa, des jeux que j'ai cracké avec Félix... L'informatique, c'est une curiosité de gosse qui ne m'a jamais vraiment quitté en fait. J'ai eu beau jurer, à l'apothéose de ma délicieuse adolescence, que je ne ferais jamais comme papa, ni comme maman, je me retrouve avec le métier de l'un, à faire du Yoga et du Tai-Chi, tout en ayant un lombricomposteur à la cave. Ironie du sort, quand tu nous tiens.</p>
<p>Bref, l'informatique, je crois que ça me vient tout d'abord de la volonté de comprendre des trucs obscurs, des trucs de grands. Déjà, maîtriser un jargon, à défaut d'aider avec les filles, ça donne l'impression d'en savoir plus que les autres. Et puis, ça donne aussi l'impression d'en savoir plus que la veille. Si on n'a pas cette envie de constamment se rendre compte qu'on ne sait pas grand chose, et d'y remédier, je crois que ça n'est pas la peine de se lancer dans cette voie.</p>
<p>Un jour ou l'autre, on se rend compte aussi qu'on peut rendre service, à soi comme aux autres. On peut installer un système pour l'un, conseiller l'autre dans l'achat d'un PC, développer un programme qui s'occupe de me chercher un appart, auto-héberger son blog, organiser une conférence, parler à une autre, et plein d'autres trucs rigolos. J'insiste sur le mot <em>rigolo</em>. Mes journées passent généralement assez vite, parce que je m'amuse. Tous les passionnés d'informatique que j'ai pu rencontrer s'amusent, à un moment de leur journée. Pas plus tard que dimanche soir, j'ai décidé de tester une nouvelle techno, pour me rendre compte tout d'un coup qu'il était 4h du matin, et qu'il serait sans doute bon de fermer l'oeil. Sans l'amusement, on serait juste des gens pâles devant un écran. Mais comme on trouve que tout ça c'est quand même chouette, parce que visiblement, ça rend service à plein de gens, on développe des trucs comme Youtube, OpenStreetMap (et Mapado :), et on file des coups de main, pour le simple plaisir d'aider. Un peu comme dans le vrai monde en fait.</p>
<p>Ça faisait plusieurs années que je m'amusais dans mon coin ou avec des amis, et puis tout d'un coup, on m'a proposé de me donner de l'argent pour continuer. ALLO. Encore quelques années après, on m'a dit que comme j'étais expert dans tel truc, ça serait choupi de déménager dans la semaine et dans un autre pays, pour que le pays en question puisse réclamer ses taxes foncières sans trop de problème. Alors bon, être expert, c'est qu'une question de perspective, mais quand je repense au Balthazar d'il y a 15 ans, je me dis que le chemin parcouru ne l'a sans doute pas été pour rien.</p>
<p>Alors voilà, je suis parti sur un coup de tête, pour aller faire un truc qui m'amuse, pour des gens qui ont l'air de penser que ça leur rend service, et qui sont même prêts à m'aider à rembourser mon appart pour ça. Je sais pas vous, mais je trouve qu'il y a pire comme passion.</p>
<p>Allez, il est presque minuit, je me souhaite un bon anniversaire un peu en avance, et j'éteins mon PC. Il parait que ça empêche de dormir. Des conneries tout ça.</p>Camembert et oignons confits2014-11-01T00:00:00+01:002014-11-01T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2014-11-01:/camembert-et-oignons-confits<p>Ça fait un petit moment que je vous ai négligé. Sachez que l'alcool n'a rien a voir là dedans. On pourrait même dire que le travail et le changement d'heure en sont la cause première. Voyez vous, je travaille. Beaucoup. Le weekend aussi, de temps en temps. Du coup, quand …</p><p>Ça fait un petit moment que je vous ai négligé. Sachez que l'alcool n'a rien a voir là dedans. On pourrait même dire que le travail et le changement d'heure en sont la cause première. Voyez vous, je travaille. Beaucoup. Le weekend aussi, de temps en temps. Du coup, quand je sors du travail, il fait nuit, et les photos que je pourrais prendre d'Edimbourg seraient assez monochromes.</p>
<p>Sachez aussi que tout va vraiment bien. C'est fou comme je me sens bien dans ce pays et dans cette ville. Depuis mon dernier article, j'ai déménagé <a href="https://www.google.fr/maps/place/71+Daiches+Braes,+Edinburgh,+City+of+Edinburgh+EH15+2RD,+UK/@55.9420016,-3.0972016,14z/data=!4m2!3m1!1s0x4887b9a75012a7bb:0xb24314d3de3705a8?hl=en">ici</a>, dans un appartement ma foi bien sympathique, un peu loin de tout. Le calme est assez royal. J'ai une petite rivière qui passe en face de chez moi, que je peux suivre jusqu'à la mer, en passant à travers un parc et un terrain de golf. On atteint pas le charme de Villeurbanne, mais c'est somme toute très correct, je vous rassure. En longeant la plage, on tombe même sur un pub qui sert des burgers au camembert et oignons confits, absolument affolants!</p>
<p>Côté boulot, je ne peux pas vous dire grand chose, simplement parce que je n'ai pas le droit. Mais tout va très bien aussi. C'est un projet pour le gouvernement écossais, d'une complexité assez folle et dans laquelle on essaie de mettre un peu d'ordre. On a une date butoir en décembre, qui met bien la pression à tout le monde! C'est assez fréquent de croiser des collègues le dimanche au bureau.</p>
<p>Je vous embrasse <em>mates</em>.</p>Pizzas et compte bancaire2014-10-23T00:00:00+02:002014-10-23T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2014-10-23:/pizzas-et-compte-bancaire<p>Si on y réfléchit bien, l'Angleterre est un autre pays. C'est à dire que ce n'est pas la France en fait. Du coup, il y a pas mal de choses qui changent, voire même qui dépayseraient le premier gaulois descendant du bateau.</p>
<p>Quand j'ai voulu ouvrir un compte bancaire local …</p><p>Si on y réfléchit bien, l'Angleterre est un autre pays. C'est à dire que ce n'est pas la France en fait. Du coup, il y a pas mal de choses qui changent, voire même qui dépayseraient le premier gaulois descendant du bateau.</p>
<p>Quand j'ai voulu ouvrir un compte bancaire local, on m'a demandé mon passport, et c'est tout. Mon charmant banquier français nous demande 3 fiches de salaire, une attestation de domicile (et limite un test urinaire) pour qu'on puisse ouvrir un compte commun avec Julie. Le compte ne coute rien (du tout), la carte non plus, et seulement les agios font vaguement mal. C'est comme si les banques anglaises étaient reconnaissantes qu'on mette de l'argent chez elles. C'est pas facile à imaginer hein?</p>
<p>Autre exemple. Les anglais <strong>adorent</strong> les journaux de caniveau. On en trouve partout (sauf dans les caniveaux d'ailleurs, vu que tout est très propre). Les gros titres d'ajourd'hui: "Mon infection des reins était en fait un bébé", "Le footballeur violeur demande une seconde chance".</p>
<p>Les policiers sourient et disent bonjour. On est poli dans le bus et on dit bonjour et au revoir au chauffeur.</p>
<p>Alors evidemment, tout n'est pas rose (c'est même assez gris au niveau du ciel), mais vraiment, c'est dépaysant.</p>
<p>Au passage.</p>
<p><img alt="service" decoding="async" loading="lazy" src="./images/2014-10-23-13.31.09.jpg"></p>
<p>C'est tout.</p>Deep fried mars bars2014-10-22T00:00:00+02:002014-10-22T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2014-10-22:/deep-fried-mars-bars<p>Je n'avais pas dans l'idée de faire un blog en partant en Écosse. Disons que j'envisageais plus de boire des pintes à la place. Cela dit, comme ma maman m'a demandé d'en faire un, je m'exécute. Je suis très bien élevé voyez vous.</p>
<p>Du coup, résumons un peu. J'ai été …</p><p>Je n'avais pas dans l'idée de faire un blog en partant en Écosse. Disons que j'envisageais plus de boire des pintes à la place. Cela dit, comme ma maman m'a demandé d'en faire un, je m'exécute. Je suis très bien élevé voyez vous.</p>
<p>Du coup, résumons un peu. J'ai été contacté par Fredrik, un ami suédois, qui m'a proposé de venir travailler à Édimbourg pendant quelques mois. J'avais dans l'idée de changer de travail, et le plan consistant à partir à 7h de Lyon, prendre deux avions, un bus, un taxi pour aller directement au bureau ne manquant pas de rock'n roll, j'ai décidé d'accepter. J'y serais jusqu'à mars 2015, en théorie.</p>
<p>Je suis parti avec quelques à priori. Par exemple, j'avais dans l'idée que tous les écossais ressemblaient à <a href="http://historicalhistrionics.files.wordpress.com/2011/07/braveheart-crazy-face.jpg">ça</a>. Et bien figurez vous que c'est faux. On m'avait aussi dit que j'allais me peler les marrons. Et bien figurez vous que c'est en bonne voie. On m'avait aussi dit que l'accent écossais est complètement incompréhensible. Je vous propose de vérifier par vous même avec une <a href="https://www.youtube.com/watch?v=JHugYi8GU5s">petite vidéo <strong>facile</strong></a>.</p>
<p>Comme j'ai déjà pu le dire à certains d'entre vous, la ville est magnifique. Vous allez devoir me croire sur parole, parce que j'ai pas (encore) de photos sous la main, mais entre les châteaux, les tours médiévales, les donjons et les parcs, c'est vraiment un plaisir de s'y balader. Je n'ai toujours pas eu un seul jour de pluie depuis mon arrivée, ce qui doit bien relever du miracle. En ce qui concerne la nourriture locale, je n'ai toujours pas essayé les <a href="https://fr.wikipedia.org/wiki/Mars_frit"><em>deep fried mars bars</em></a> (parce que ça va arrêter mon coeur), mais contre toute attente, j'ai découvert qu'on mange très bien. Le <a href="https://fr.wikipedia.org/wiki/Haggis">haggis</a> est bien évidemment excellent, mais on peut trouver de la bonne nourriture de pub un peu partout, pour pas grand chose.</p>
<p>Je vous embrasse <em>lads</em>.</p>Crawl a website with scrapy2012-04-23T00:00:00+02:002012-04-23T00:00:00+02:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2012-04-23:/crawl-a-website-with-scrapy<p>In this article, we are going to see how to scrape information from a website, in particular, from all pages with a common URL pattern. We will see how to do that with <a href="http://scrapy.org/">Scrapy</a>, a very powerful, and yet simple, scraping and web-crawling framework.</p>
<p>For example, you might be interested …</p><p>In this article, we are going to see how to scrape information from a website, in particular, from all pages with a common URL pattern. We will see how to do that with <a href="http://scrapy.org/">Scrapy</a>, a very powerful, and yet simple, scraping and web-crawling framework.</p>
<p>For example, you might be interested in scraping information about each article of a blog, and store it information in a database. To achieve such a thing, we will see how to implement a simple <a href="https://en.wikipedia.org/wiki/Web_crawler">spider</a> using <a href="http://scrapy.org/">Scrapy</a>, which will crawl the blog and store the extracted data into a <a href="http://www.mongodb.org/">MongoDB</a> database.</p>
<p>We will consider that you have a <a href="http://www.mongodb.org/display/DOCS/Quickstart">working MongoDB server</a>, and that you have installed the <code>pymongo</code> and <code>scrapy</code> python packages, both installable with <a href="http://pypi.python.org/pypi/pip"><code>pip</code></a>.</p>
<p>If you have never toyed around with <a href="http://scrapy.org/">Scrapy</a>, you should first read this <a href="http://doc.scrapy.org/en/latest/intro/tutorial.html">short tutorial</a>.</p>
<h2 id="first-step-identify-the-url-patterns">First step, identify the URL pattern(s)</h2>
<p>In this example, we’ll see how to extract the following information from each <a href="http://isbullsh.it">isbullsh.it</a> blogpost :</p>
<ul>
<li>title</li>
<li>author</li>
<li>tag</li>
<li>release date</li>
<li>url</li>
</ul>
<p>We’re lucky, all posts have the same URL pattern: <code>http://isbullsh.it/YYYY/MM/title</code>. These links can be found in the different pages of the site homepage.</p>
<p>What we need is a spider which will follow all links following this pattern, scrape the required information from the target webpage, validate the data integrity, and populate a MongoDB collection.</p>
<h2 id="building-the-spider">Building the spider</h2>
<p>We create a Scrapy project, following the instructions from their <a href="http://doc.scrapy.org/en/latest/intro/tutorial.html">tutorial</a>. We obtain the following project structure:</p>
<div class="highlight"><pre><span></span><code>isbullshit_scraping/
├── isbullshit
│ ├── __init__.py
│ ├── items.py
│ ├── pipelines.py
│ ├── settings.py
│ └── spiders
│ ├── __init__.py
│ ├── isbullshit_spiders.py
└── scrapy.cfg
</code></pre></div>
<p>We begin by defining, in <code>items.py</code>, the item structure which will contain the extracted information:</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">scrapy.item</span> <span class="kn">import</span> <span class="n">Item</span><span class="p">,</span> <span class="n">Field</span>
<span class="k">class</span> <span class="nc">IsBullshitItem</span><span class="p">(</span><span class="n">Item</span><span class="p">):</span>
<span class="n">title</span> <span class="o">=</span> <span class="n">Field</span><span class="p">()</span>
<span class="n">author</span> <span class="o">=</span> <span class="n">Field</span><span class="p">()</span>
<span class="n">tag</span> <span class="o">=</span> <span class="n">Field</span><span class="p">()</span>
<span class="n">date</span> <span class="o">=</span> <span class="n">Field</span><span class="p">()</span>
<span class="n">link</span> <span class="o">=</span> <span class="n">Field</span><span class="p">()</span>
</code></pre></div>
<p>Now, let’s implement our spider, in <code>isbullshit_spiders.py</code>:</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">scrapy.contrib.spiders</span> <span class="kn">import</span> <span class="n">CrawlSpider</span><span class="p">,</span> <span class="n">Rule</span>
<span class="kn">from</span> <span class="nn">scrapy.contrib.linkextractors.sgml</span> <span class="kn">import</span> <span class="n">SgmlLinkExtractor</span>
<span class="kn">from</span> <span class="nn">scrapy.selector</span> <span class="kn">import</span> <span class="n">HtmlXPathSelector</span>
<span class="kn">from</span> <span class="nn">isbullshit.items</span> <span class="kn">import</span> <span class="n">IsBullshitItem</span>
<span class="k">class</span> <span class="nc">IsBullshitSpider</span><span class="p">(</span><span class="n">CrawlSpider</span><span class="p">):</span>
<span class="n">name</span> <span class="o">=</span> <span class="s1">'isbullshit'</span>
<span class="n">start_urls</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'http://isbullsh.it'</span><span class="p">]</span> <span class="c1"># urls from which the spider will start crawling</span>
<span class="n">rules</span> <span class="o">=</span> <span class="p">[</span><span class="n">Rule</span><span class="p">(</span><span class="n">SgmlLinkExtractor</span><span class="p">(</span><span class="n">allow</span><span class="o">=</span><span class="p">[</span><span class="sa">r</span><span class="s1">'page/\d+'</span><span class="p">]),</span> <span class="n">follow</span><span class="o">=</span><span class="kc">True</span><span class="p">),</span>
<span class="c1"># r'page/\d+' : regular expression for http://isbullsh.it/page/X URLs</span>
<span class="n">Rule</span><span class="p">(</span><span class="n">SgmlLinkExtractor</span><span class="p">(</span><span class="n">allow</span><span class="o">=</span><span class="p">[</span><span class="sa">r</span><span class="s1">'\d</span><span class="si">{4}</span><span class="s1">/\d</span><span class="si">{2}</span><span class="s1">/\w+'</span><span class="p">]),</span> <span class="n">callback</span><span class="o">=</span><span class="s1">'parse_blogpost'</span><span class="p">)]</span>
<span class="c1"># r'\d{4}/\d{2}/\w+' : regular expression for http://isbullsh.it/YYYY/MM/title URLs</span>
<span class="k">def</span> <span class="nf">parse_blogpost</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
<span class="o">...</span>
</code></pre></div>
<p>Our spider inherits from <code>CrawlSpider</code>, which “provides a convenient mechanism for following links by defining a set of rules”. More info <a href="http://isbullsh.it/2012/04/Web-crawling-with-scrapy/readthedocs.org/docs/scrapy/en/0.14/topics/spiders.html#crawlspider">here</a>.</p>
<p>We then define two simple rules:</p>
<ul>
<li>Follow links pointing to <code>http://isbullsh.it/page/X</code></li>
<li>Extract information from pages defined by a URL of pattern <code>http://isbullsh.it/YYYY/MM/title</code>, using the callback method <code>parse_blogpost</code>.</li>
</ul>
<h2 id="extracting-the-data">Extracting the data</h2>
<p>To extract the title, author, etc, from the HTML code, we’ll use the <code>scrapy.selector.HtmlXPathSelector object</code>, which uses the <code>libxml2</code> HTML parser. If you’re not familiar with this object, you should read the <code>XPathSelector</code> <a href="http://readthedocs.org/docs/scrapy/en/0.14/topics/selectors.html#using-selectors-with-xpaths">documentation</a>.</p>
<p>We’ll now define the extraction logic in the <code>parse_blogpost</code> method (I’ll only define it for the title and tag(s), it’s pretty much always the same logic):</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">parse_blogpost</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
<span class="n">hxs</span> <span class="o">=</span> <span class="n">HtmlXPathSelector</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
<span class="n">item</span> <span class="o">=</span> <span class="n">IsBullshitItem</span><span class="p">()</span>
<span class="c1"># Extract title</span>
<span class="n">item</span><span class="p">[</span><span class="s1">'title'</span><span class="p">]</span> <span class="o">=</span> <span class="n">hxs</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s1">'//header/h1/text()'</span><span class="p">)</span><span class="o">.</span><span class="n">extract</span><span class="p">()</span> <span class="c1"># XPath selector for title</span>
<span class="c1"># Extract author</span>
<span class="n">item</span><span class="p">[</span><span class="s1">'tag'</span><span class="p">]</span> <span class="o">=</span> <span class="n">hxs</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s2">"//header/div[@class='post-data']/p/a/text()"</span><span class="p">)</span><span class="o">.</span><span class="n">extract</span><span class="p">()</span> <span class="c1"># Xpath selector for tag(s)</span>
<span class="o">...</span>
<span class="k">return</span> <span class="n">item</span>
</code></pre></div>
<p><strong>Note</strong>: To be sure of the XPath selectors you define, I’d advise you to use Firebug, Firefox Inspect, or equivalent, to inspect the HTML code of a page, and then test the selector in a <a href="http://doc.scrapy.org/en/latest/intro/tutorial.html#trying-selectors-in-the-shell">Scrapy shell</a>. That only works if the data position is coherent throughout all the pages you crawl.</p>
<h2 id="store-the-results-in-mongodb">Store the results in MongoDB</h2>
<p>Each time that the <code>parse_blogspot</code> method returns an item, we want it to be sent to a pipeline which will validate the data, and store everything in our Mongo collection.</p>
<p>First, we need to add a couple of things to <code>settings.py</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">ITEM_PIPELINES</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'isbullshit.pipelines.MongoDBPipeline'</span><span class="p">,]</span>
<span class="n">MONGODB_SERVER</span> <span class="o">=</span> <span class="s2">"localhost"</span>
<span class="n">MONGODB_PORT</span> <span class="o">=</span> <span class="mi">27017</span>
<span class="n">MONGODB_DB</span> <span class="o">=</span> <span class="s2">"isbullshit"</span>
<span class="n">MONGODB_COLLECTION</span> <span class="o">=</span> <span class="s2">"blogposts"</span>
</code></pre></div>
<p>Now that we’ve defined our pipeline, our MongoDB database and collection, we’re just left with the pipeline implementation. We just want to be sure that we do not have any missing data (ex: a blogpost without a title, author, etc).</p>
<p>Here is our <code>pipelines.py</code> file :</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">pymongo</span>
<span class="kn">from</span> <span class="nn">scrapy.exceptions</span> <span class="kn">import</span> <span class="n">DropItem</span>
<span class="kn">from</span> <span class="nn">scrapy.conf</span> <span class="kn">import</span> <span class="n">settings</span>
<span class="kn">from</span> <span class="nn">scrapy</span> <span class="kn">import</span> <span class="n">log</span>
<span class="k">class</span> <span class="nc">MongoDBPipeline</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">connection</span> <span class="o">=</span> <span class="n">pymongo</span><span class="o">.</span><span class="n">Connection</span><span class="p">(</span><span class="n">settings</span><span class="p">[</span><span class="s1">'MONGODB_SERVER'</span><span class="p">],</span> <span class="n">settings</span><span class="p">[</span><span class="s1">'MONGODB_PORT'</span><span class="p">])</span>
<span class="n">db</span> <span class="o">=</span> <span class="n">connection</span><span class="p">[</span><span class="n">settings</span><span class="p">[</span><span class="s1">'MONGODB_DB'</span><span class="p">]]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">collection</span> <span class="o">=</span> <span class="n">db</span><span class="p">[</span><span class="n">settings</span><span class="p">[</span><span class="s1">'MONGODB_COLLECTION'</span><span class="p">]]</span>
<span class="k">def</span> <span class="nf">process_item</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="n">spider</span><span class="p">):</span>
<span class="n">valid</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">for</span> <span class="n">data</span> <span class="ow">in</span> <span class="n">item</span><span class="p">:</span>
<span class="c1"># here we only check if the data is not null</span>
<span class="c1"># but we could do any crazy validation we want</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">data</span><span class="p">:</span>
<span class="n">valid</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">raise</span> <span class="n">DropItem</span><span class="p">(</span><span class="s2">"Missing </span><span class="si">%s</span><span class="s2"> of blogpost from </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">item</span><span class="p">[</span><span class="s1">'url'</span><span class="p">]))</span>
<span class="k">if</span> <span class="n">valid</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">collection</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="n">log</span><span class="o">.</span><span class="n">msg</span><span class="p">(</span><span class="s2">"Item wrote to MongoDB database </span><span class="si">%s</span><span class="s2">/</span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span>
<span class="p">(</span><span class="n">settings</span><span class="p">[</span><span class="s1">'MONGODB_DB'</span><span class="p">],</span> <span class="n">settings</span><span class="p">[</span><span class="s1">'MONGODB_COLLECTION'</span><span class="p">]),</span>
<span class="n">level</span><span class="o">=</span><span class="n">log</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">,</span> <span class="n">spider</span><span class="o">=</span><span class="n">spider</span><span class="p">)</span>
<span class="k">return</span> <span class="n">item</span>
</code></pre></div>
<h2 id="release-the-spider">Release the spider!</h2>
<p>Now, all we have to do is change directory to the root of our project and execute</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>scrapy<span class="w"> </span>crawl<span class="w"> </span>isbullshit
</code></pre></div>
<p>The spider will then follow all links pointing to a blogpost, retrieve the post title, author name, date, etc, validate the extracted data, and store all that in a MongoDB collection if validation went well.</p>
<p>Pretty neat, hm?</p>
<h2 id="conclusion">Conclusion</h2>
<p>This case is pretty simplistic: all URLs have a similar pattern and all links are hard written in the HTML code: there is no JS involved. In the case were the links you want to reach are generated by JS, you’d probably want to check out <a href="http://pypi.python.org/pypi/selenium">Selenium</a>. You could complexify the spider by adding new rules, or more complicated regular expressions, but I just wanted to demo how Scrapy worked, not getting into crazy regex explanations.</p>
<p>Also, be aware that sometimes, there’s a thin line bewteen playing with web-scraping and <a href="https://en.wikipedia.org/wiki/Web_scraping#Legal_issues">getting into trouble</a>.</p>
<p>Finally, when toying with web-crawling, keep in mind that you might just flood the server with requests, which can sometimes get you IP-blocked :)</p>
<p>The entire code of this project is hosted on <a href="https://github.com/BaltoRouberol/isbullshit-crawler">Github</a>. Help yourselves!</p>Create a webcam manager using pyGTK and Gstreamer2012-02-29T00:00:00+01:002012-02-29T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2012-02-29:/create-a-webcam-manager-using-pygtk-and-gstreamer<h2 id="introduction">Introduction</h2>
<p>I recently joined the <a href="http://www.strongsteam.com">Strongsteam</a> project for a 6 month internship. Our main goal is to provide some <em>"artificial intelligence and
data mining APIs to let you pull interesting information out of images, video and audio."</em>
We will be doing a presentation at <a href="https://us.pycon.org/2012/">Pycon 2012</a>, the 9th of March …</p><h2 id="introduction">Introduction</h2>
<p>I recently joined the <a href="http://www.strongsteam.com">Strongsteam</a> project for a 6 month internship. Our main goal is to provide some <em>"artificial intelligence and
data mining APIs to let you pull interesting information out of images, video and audio."</em>
We will be doing a presentation at <a href="https://us.pycon.org/2012/">Pycon 2012</a>, the 9th of March, during the <a href="https://us.pycon.org/2012/community/startuprow/">Startup Row weekend</a>.
On this occasion, I had to implement a desktop GUI allowing to display a webcam video stream and to capture snapshots, with the following constraints:</p>
<ul>
<li>GUI written with <a href="http://wxpython.org/">wxPython</a> or <a href="http://www.pygtk.org/">pyGTK</a></li>
<li>the webcam stream must be integrated in the wxPython/pyGTK window</li>
<li>the webcam must not be handled with the <a href="http://opencv.willowgarage.com/wiki/PythonInterface">OpenCV</a> python module (the installation can be painful on Mac OS X)</li>
<li>the snapshots default format and resolution must be JPG and 640x480px</li>
</ul>
<h2 id="how-to-handle-the-webcam">How to handle the webcam ?</h2>
<p>My initial research led me to consider two different solutions:</p>
<ul>
<li>using <a href="http://www.pygame.org">PyGame</a>, a set of python modules adding functionality on top of the <a href="http://www.libsdl.org/">SDL</a> library</li>
<li>using <a href="http://gstreamer.freedesktop.org/">Gstreamer</a>, a pipeline-based multimedia framework allowing <em>"to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing"</em> (quote: wikipedia article). Gstreamer is used by a bunch of multimedia applications, like <a href="https://live.gnome.org/Cheese">Cheese</a>, <a href="http://amarok.kde.org/">Amarok</a>, <a href="http://pitivi.sourceforge.net/">Pitivi</a>, ...</li>
</ul>
<p>I quickly turned to PyGame, because of the simplicify of the snapshot operation : all we have to do is to use the <a href="http://www.pygame.org/docs/ref/camera.html#pygame.camera.Camera"><code>pygame.camera.Camera.get_image()</code></a> function. However, the integration of the PyGame surface into a pyGTK interface turned out to be pretty complicated. I found a couple of <a href="http://stackoverflow.com/questions/25661/pygame-within-a-pygtk-application">StackOverflow posts</a> stating that even though this integration was possible, it was not advised. Indeed, some erratic behaviours seem to be observed when using different OS.</p>
<p>I thus considered Gstreamer, and quicky found this <a href="http://pygstdocs.berlios.de/pygst-tutorial/webcam-viewer.html">encouraging project</a>. This code allowed to start and stop a webcam video stream embedded in a pyGTK interface : I was definitlely in the right place !</p>
<h2 id="why-doesnt-it-work-with-my-webcam">Why doesn't it work with my webcam ?</h2>
<p>If you experience some problems testing the project introduced into the previous part (black screen, first run successful and following run leading to black screen, ...) check if your webcam is UVC (USB Video Class) Linux compliant. To do that, type in</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>lsusb
</code></pre></div>
<p>in a terminal and locate the line describing your webcam.</p>
<p>My laptop integrated webcam was described as <code>Bus 001 Device 003: ID 05ca:1814 Ricoh Co., Ltd HD Webcam</code>. The reference <code>05ca:1814</code> doesn't appear on the <a href="http://www.ideasonboard.org/uvc/">UVC</a> website. That could explain why I experienced so many problems with it (it appears that Ricoh webcams are poorly UVC compliant).</p>
<p>I hence bought a Logitech QuickCam Pro 9000, known for being well supported. Everything ran smoothly with this one.</p>
<h2 id="how-to-use-gstreamer">How to use Gstreamer ?</h2>
<p>If you don't know how to use Gstreamer, I'd advise you to have a look these pages :</p>
<ul>
<li><a href="http://wiki.oz9aec.net/index.php/Gstreamer_Cheat_Sheet">Gstreamer cheat sheet</a></li>
<li><a href="http://www.oz9aec.net/index.php/gstreamer/345-a-weekend-with-gstreamer">A weekend with Gstreamer</a></li>
</ul>
<p>The main idea is to construct a <strong>pipeline</strong>, by connecting various data sources, sinks and processing blocks (bins) in a data flow graph.</p>
<p>In our case, we are going to use the following pipeline to display the webcam stream:</p>
<blockquote>
<p><code>v4l2src ! video/x-raw-yuv,width=640,height=480,framerate=30/1 ! xvimagesink</code></p>
</blockquote>
<ul>
<li><code>v4l2src</code> : Video for Linux input : your webcam (the default device is <code>/dev/video0</code>, but if you are using an external webcam, use <code>v4l2src device=/dev/video1</code>)</li>
<li><code>video/x-raw-yuv</code> : video colorspace specific to webcam</li>
<li><code>width=640,height=480</code> : your webcam resolution (check that it is compatible with your webcam)</li>
<li><code>framerate=30/1</code> : number of frames per second</li>
<li><code>xvimagesink</code> : video sink</li>
</ul>
<p>Let's see how to do that in Python:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">create_video_pipeline</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Set up the video pipeline and the communication bus bewteen the video stream and gtk DrawingArea """</span>
<span class="n">video_pipeline</span> <span class="o">=</span> <span class="s1">'v4l2src device=/dev/video1 ! video/x-raw-yuv,width=640,height=480,framerate=30/1 ! xvimagesink'</span>
<span class="bp">self</span><span class="o">.</span><span class="n">video_player</span> <span class="o">=</span> <span class="n">gst</span><span class="o">.</span><span class="n">parse_launch</span><span class="p">(</span><span class="n">video_pipeline</span><span class="p">)</span> <span class="c1"># create pipeline</span>
<span class="bp">self</span><span class="o">.</span><span class="n">video_player</span><span class="o">.</span><span class="n">set_state</span><span class="p">(</span><span class="n">gst</span><span class="o">.</span><span class="n">STATE_PLAYING</span><span class="p">)</span> <span class="c1"># start video stream</span>
<span class="n">bus</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">video_player</span><span class="o">.</span><span class="n">get_bus</span><span class="p">()</span>
<span class="n">bus</span><span class="o">.</span><span class="n">add_signal_watch</span><span class="p">()</span>
<span class="n">bus</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">"message"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">on_message</span><span class="p">)</span>
<span class="n">bus</span><span class="o">.</span><span class="n">enable_sync_message_emission</span><span class="p">()</span>
<span class="n">bus</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">"sync-message::element"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">on_sync_message</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">on_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bus</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
<span class="w"> </span><span class="sd">""" Gst message bus. Closes the pipeline in case of error or end of stream message """</span>
<span class="n">t</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">type</span>
<span class="k">if</span> <span class="n">t</span> <span class="o">==</span> <span class="n">gst</span><span class="o">.</span><span class="n">MESSAGE_EOS</span><span class="p">:</span>
<span class="nb">print</span> <span class="s2">"MESSAGE EOS"</span>
<span class="bp">self</span><span class="o">.</span><span class="n">video_player</span><span class="o">.</span><span class="n">set_state</span><span class="p">(</span><span class="n">gst</span><span class="o">.</span><span class="n">STATE_NULL</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">t</span> <span class="o">==</span> <span class="n">gst</span><span class="o">.</span><span class="n">MESSAGE_ERROR</span><span class="p">:</span>
<span class="nb">print</span> <span class="s2">"MESSAGE ERROR"</span>
<span class="n">err</span><span class="p">,</span> <span class="n">debug</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">parse_error</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"Error: </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="n">err</span><span class="p">,</span> <span class="n">debug</span>
<span class="bp">self</span><span class="o">.</span><span class="n">video_player</span><span class="o">.</span><span class="n">set_state</span><span class="p">(</span><span class="n">gst</span><span class="o">.</span><span class="n">STATE_NULL</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">on_sync_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bus</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
<span class="w"> </span><span class="sd">""" Set up the Webcam <--> GUI messages bus """</span>
<span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">structure</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">message_name</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">structure</span><span class="o">.</span><span class="n">get_name</span><span class="p">()</span>
<span class="k">if</span> <span class="n">message_name</span> <span class="o">==</span> <span class="s2">"prepare-xwindow-id"</span><span class="p">:</span>
<span class="c1"># Assign the viewport</span>
<span class="n">imagesink</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">src</span>
<span class="n">imagesink</span><span class="o">.</span><span class="n">set_property</span><span class="p">(</span><span class="s2">"force-aspect-ratio"</span><span class="p">,</span> <span class="kc">True</span><span class="p">)</span>
<span class="c1"># Sending video stream to gtk DrawingArea</span>
<span class="n">imagesink</span><span class="o">.</span><span class="n">set_xwindow_id</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">movie_window</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">xid</span><span class="p">)</span>
</code></pre></div>
<p>Now, we have a live video stream displayed into a pyGTK interface, but still no way of capturing a snapshot.</p>
<h2 id="how-do-we-capture-a-snapshot">How do we capture a snapshot ?</h2>
<p>I encountered many StackOverflow open questions about this part, but no satisfactory answer...</p>
<p>At first, I wanted to use Gstreamer for that too, but I couldn't find any way to dynamically modify the pipeline to add a frame extraction, jpg encoding and a filesink (to save the snapshot). I thus tried this ugly hack : when the <em>'take snapshot'</em> button is clicked</p>
<ul>
<li>stop the video stream</li>
<li>start the following pipeline: <code>v4l2src device=/dev/video1 ! video/x-raw-yuv,width=640,height=480,framerate=30/1 ! ffmpegcolorspace ! video/x-raw-rgb,framerate=1/1 ! ffmpegcolorspace ! jpegenc snapshot=true ! filesink location=snap.jpeg</code>, which will extract a single frame, encode it to jpg and save it to a file.</li>
<li>stop this image pipeline</li>
<li>re-start the video stream</li>
</ul>
<p>That was of course ugly, and resulted into a ~2s flicker when taking the snapshot... Back to square one.</p>
<p>I'll save you the suspens, the right solution is to use the <code>gtk.DrawingArea.window.get_colormap()</code> method, as shown here:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">take_snapshot</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">""" Capture a snapshot from DrawingArea and save it into a image file """</span>
<span class="n">drawable</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">movie_window</span><span class="o">.</span><span class="n">window</span>
<span class="c1"># self.movie_window is of type gtk.DrawingArea()</span>
<span class="n">colormap</span> <span class="o">=</span> <span class="n">drawable</span><span class="o">.</span><span class="n">get_colormap</span><span class="p">()</span>
<span class="n">pixbuf</span> <span class="o">=</span> <span class="n">gtk</span><span class="o">.</span><span class="n">gdk</span><span class="o">.</span><span class="n">Pixbuf</span><span class="p">(</span><span class="n">gtk</span><span class="o">.</span><span class="n">gdk</span><span class="o">.</span><span class="n">COLORSPACE_RGB</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="o">*</span><span class="n">drawable</span><span class="o">.</span><span class="n">get_size</span><span class="p">())</span>
<span class="n">pixbuf</span> <span class="o">=</span> <span class="n">pixbuf</span><span class="o">.</span><span class="n">get_from_drawable</span><span class="p">(</span><span class="n">drawable</span><span class="p">,</span> <span class="n">colormap</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span> <span class="o">*</span><span class="n">drawable</span><span class="o">.</span><span class="n">get_size</span><span class="p">())</span>
<span class="n">pixbuf</span> <span class="o">=</span> <span class="n">pixbuf</span><span class="o">.</span><span class="n">scale_simple</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">W</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">H</span><span class="p">,</span> <span class="n">gtk</span><span class="o">.</span><span class="n">gdk</span><span class="o">.</span><span class="n">INTERP_HYPER</span><span class="p">)</span> <span class="c1"># resize</span>
<span class="c1"># We resize from actual window size to wanted resolution</span>
<span class="c1"># gtk.gdk.INTER_HYPER is the slowest and highest quality reconstruction function</span>
<span class="c1"># More info here : http://developer.gnome.org/pygtk/stable/class-gdkpixbuf.html#method-gdkpixbuf--scale-simple</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'snap.jpg'</span>
<span class="n">filepath</span> <span class="o">=</span> <span class="n">relpath</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">pixbuf</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">snap_format</span><span class="p">)</span>
</code></pre></div>
<p>This snippet does the following operations:</p>
<ul>
<li>extract the last frame from the <code>gtk.DrawingArea</code></li>
<li>encode it to RGB</li>
<li>resize it to 640x480px</li>
<li>save it to <code>snap.jpg</code></li>
</ul>
<p>And that's done, without even a teeny-tiny flicker! Yay! We now have a perfecly functional snapshot operation.</p>
<h2 id="project-source-code-git-repository">Project source code & Git repository</h2>
<p>All the code can be encountered on my <a href="https://github.com/BaltoRouberol/Gstreamer-webcam-tool">GitHub</a>.</p>How to randomly generate a Monty Python parody2011-11-16T00:00:00+01:002011-11-16T00:00:00+01:00Balthazar Rouberoltag:blog.balthazar-rouberol.com,2011-11-16:/how-to-randomly-generate-a-monty-python-parody<p>If you always wanted to write texts in the way of Monty Python, I have what you need !
In this post, I am going to show you mathematical techniques to analyse a text, in order to randomly generate look-alike texts.</p>
<h2 id="introduction-to-basic-concepts">Introduction to basic concepts</h2>
<p>First essential question: what is a …</p><p>If you always wanted to write texts in the way of Monty Python, I have what you need !
In this post, I am going to show you mathematical techniques to analyse a text, in order to randomly generate look-alike texts.</p>
<h2 id="introduction-to-basic-concepts">Introduction to basic concepts</h2>
<p>First essential question: what is a text?</p>
<p>From a mathematical point of view, a text of length <em>n</em> simply is the concatenation of <em>n</em> symbols, all taken from a finite alphabet <em>A</em>.
In our context, the alphabet is generally composed of all lowercase and uppercase letters, punctuation signs, etc.</p>
<p>In a real-life situation, <strong>the symbols sucession is not random, but depends of the previous symbols</strong>. Indeed, if the 3 last symbols are <em>" "</em>, <em>"t"</em> and <em>"h"</em>, it is highly probable that the next one will be <em>"e"</em>, because the world <em>"the"</em> is fairly common.</p>
<p>The whole problem can thus be resumed to obtaining a transition probability matrix between strings of fixed length and all smbols of the alphabet.</p>
<p><em>Example</em> : Let's assume that the three last symbols are <em>" "</em>, <em>"t"</em>, and <em>"h"</em>, and that the probability of the next symbol being <em>"e"</em> (written $p("e" / " th")$ ) is 0.6, an <em>"a"</em> is 0.3 and <em>"u"</em> is 0.1.
We would then obtain a line of the matrix of transition probability between <em>" th"</em> and all alphabet symbols:</p>
<p><em>" th"</em> —> a: 0.3, b: 0, c: 0, ..., e: 0.6, ..., u: 0.1, ...</p>
<p>The probability $p("e" / " th")$ is called a conditional probability.</p>
<h2 id="markov-chain-of-order-k">Markov chain of order $k$</h2>
<p>We are going to model our data text (here, the "Monthy Python and the Holy Grail" script) with a Markov chain of order $k$. This barbarian name refers to :</p>
<blockquote>
<p>"a mathematical system that undergoes transitions from one state to another (from a finite or countable number of possible states) in a chain-like manner
--
<a href="http://en.wikipedia.org/wiki/Markov_chain" title="Wikipedia">Source : Wikipedia</a>"</p>
</blockquote>
<p>That means that the following state is conditioned by the $k$ previous ones.</p>
<p>If we deal with a Markov chain of order 3, the probability of occurence of the next symbol will only depends on the 3 previous symbols. From previous tests, I can say that <strong>$k=10$ is a good place to start</strong>. (More on that later)</p>
<h2 id="text-alphabet">Text Alphabet</h2>
<p>We've just fixed the value of k, which was the first step of the process. Now, we need to to create a list of all encountered symbols (ie: the alphabet).</p>
<p>First, we read the data file, and join all the lines in a single string.</p>
<div class="highlight"><pre><span></span><code><span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'../data/monty.txt'</span><span class="p">)</span>
<span class="n">f_lines</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">readlines</span><span class="p">())</span>
</code></pre></div>
<p>Then, we create the alphabet list:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">alphabet</span><span class="p">(</span><span class="n">datafile_lines</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Returns all used characters in a given text</span>
<span class="sd"> """</span>
<span class="n">alph</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">letter</span> <span class="ow">in</span> <span class="n">datafile_lines</span><span class="p">:</span>
<span class="k">if</span> <span class="n">letter</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">alph</span><span class="p">:</span>
<span class="n">alph</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">letter</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">alph</span><span class="p">)</span>
</code></pre></div>
<h2 id="finding-all-exiting-k-tuples-in-the-source-text">Finding all exiting K-tuples in the source text</h2>
<p>Now, <strong>we need to identify all distinct strings of length $k=10$ in the text</strong>.</p>
<p>This can seem a bit tedious, but list comprehensions and sets will do a lovely work.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># -- split text in ak chunks of length k</span>
<span class="n">ak_chunks</span> <span class="o">=</span> <span class="p">[</span><span class="n">datafile_lines</span><span class="p">[</span><span class="n">i</span><span class="p">:</span><span class="n">i</span><span class="o">+</span><span class="n">k</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">xrange</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">datafile_lines</span><span class="p">))]</span>
<span class="c1"># -- remove final chunk if not of size k</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">ak_chunks</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="o">!=</span> <span class="n">k</span><span class="p">:</span>
<span class="n">ak_chunks</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">ak_chunks</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="c1"># -- Extract unique values from list</span>
<span class="n">ak_chunks</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">ak_chunks</span><span class="p">))</span> <span class="c1">#set: reduce to unique values</span>
</code></pre></div>
<h2 id="empirical-probabilities-of-transition">Empirical probabilities of transition</h2>
<p>Now comes the hard work. So far, we have</p>
<ul>
<li>a text,</li>
<li>its alphabet,</li>
<li>a HUGE list of all distincts strings of length $k=10$ contained in the text</li>
</ul>
<p>What we then need is a way to calculate the empirical probability of transition between each string of length 10 and symbols of the alphabet ("empirical" in the way that these probabilities will only apply to the text we study).</p>
<p>Let's formalize a bit the problem:</p>
<ul>
<li>$a^k$ : string of length $k$ (here, 10)</li>
<li>$b$ : symbol located after $a^k$</li>
<li>$n_(a^k)$ : number of times that the string $a^k$ is encountered in the text</li>
<li>$n_(b/a^k)$ = number of times that the string $a^k$ is followed by the symbol $b$</li>
</ul>
<p>We can now express the empirical probability $p(b/a^k) = n_(b/a^k) / n_(a^k)$
(number of times that the string $a^k$ is followed by the symbol $b$ / number of times that the string $a^k$ is encountered in the text)</p>
<p><em>Example</em> : if our text is ABCABDABC, $a^k = AB$ and $b = C$:</p>
<ul>
<li>$n_(AB) = 3$</li>
<li>$n_(C/AB) = 2$</li>
<li>$p(C/AB) = 2/3 = 0.667$</li>
</ul>
<p>Let's write all that in Python:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">conditional_empirical_proba</span><span class="p">(</span><span class="n">chain</span><span class="p">,</span> <span class="n">ak</span><span class="p">,</span> <span class="n">symbol</span><span class="p">,</span> <span class="n">n_ak</span><span class="p">):</span> <span class="c1"># p(b/a^k)</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Returns the proportion of symbols after the ak string (contained</span>
<span class="sd"> in chain string and of length k) which are equal to the value</span>
<span class="sd"> of given parameter 'symbol'</span>
<span class="sd"> Ex:conditional_empirical_proba('ABCABD', 2, 'AB', 'C', n_ak)-> 0.5</span>
<span class="sd"> """</span>
<span class="n">nb_ak</span> <span class="o">=</span> <span class="n">n_b_ak</span><span class="p">(</span><span class="n">chain</span><span class="p">,</span> <span class="n">ak</span><span class="p">,</span> <span class="n">symbol</span><span class="p">)</span>
<span class="k">if</span> <span class="n">n_ak</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">float</span><span class="p">(</span><span class="n">nb_ak</span><span class="p">)</span><span class="o">/</span><span class="n">n_ak</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">n_b_ak</span><span class="p">(</span><span class="n">chain</span><span class="p">,</span> <span class="n">ak</span><span class="p">,</span> <span class="n">symbol</span><span class="p">):</span> <span class="c1"># n_(b/a^k)</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Given a string chain, returns the number of</span>
<span class="sd"> times that a given symbol is found</span>
<span class="sd"> right after a string ak inside the chain</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="n">chain</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="n">ak</span><span class="o">+</span><span class="n">symbol</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">n_ak</span><span class="p">(</span><span class="n">chain</span><span class="p">,</span> <span class="n">ak</span><span class="p">):</span> <span class="c1"># n_(a^k)</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Given a string chain and a string ak, returns</span>
<span class="sd"> the number of times ak is found in chain</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="n">chain</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="n">ak</span><span class="p">)</span>
</code></pre></div>
<p>Now, the only remaning thing to do is to calculate the empirical conditional probability for each k-tuple and for each symbol.</p>
<p>A few remarks are necessary:</p>
<ul>
<li>We will only store empirical conditional probabilities > 0 (more on that later)</li>
<li>We will store accumulative empirical conditional probabilities (more on that later)</li>
<li>The matrix will be created with a dictionnary of dictionnaries</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="c1"># Initialization of matrix</span>
<span class="n">prob</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">ak</span> <span class="ow">in</span> <span class="n">ak_chunks</span><span class="p">:</span>
<span class="c1"># New matrix line</span>
<span class="n">prob</span><span class="p">[</span><span class="n">ak</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
<span class="c1"># -- calculate p(b/a^k) for each symbol of alphabet</span>
<span class="n">pbak_cumul</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">symb</span> <span class="ow">in</span> <span class="n">alpha</span><span class="p">:</span>
<span class="n">pbak</span> <span class="o">=</span> <span class="n">conditional_empirical_proba</span><span class="p">(</span><span class="n">datafile_lines</span><span class="p">,</span> <span class="n">ak</span><span class="p">,</span> <span class="n">symb</span><span class="p">,</span> <span class="n">nak</span><span class="p">)</span>
<span class="c1"># cumulative probabilities</span>
<span class="n">pbak_cumul</span> <span class="o">+=</span> <span class="n">pbak</span>
<span class="c1"># if sucession ak+symb is encountered in text, add probability to matrix</span>
<span class="k">if</span> <span class="n">pbak</span> <span class="o">!=</span> <span class="mf">0.0</span><span class="p">:</span> <span class="c1"># Very important, if pbak = 0.0, the combination ak+symb will not be randomly generated</span>
<span class="n">prob</span><span class="p">[</span><span class="n">ak</span><span class="p">][</span><span class="n">symb</span><span class="p">]</span> <span class="o">=</span> <span class="n">pbak_cumul</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'../results/distribs/distrib_k</span><span class="si">%d</span><span class="s1">.txt'</span> <span class="o">%</span> <span class="p">(</span><span class="n">k</span><span class="p">),</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">proba_file</span>
<span class="n">pickle</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">prob</span><span class="p">,</span> <span class="n">proba_file</span><span class="p">)</span>
</code></pre></div>
<h2 id="random-text-generation">Random text generation</h2>
<p>Close your eyes for a second, and think about what we just did. <strong>We calculated empirical transition probabilities between all existing strings of length 10 and all symbols of the alphabet, and stored the non nil acumulative probabilities in a matrix</strong>. (The non-nil part has two main advatages : it implies less storage cost, and we only store combinations that occured in the text. This way, random generation becomes really easy !)</p>
<p>It is now extremely easy to generate a text using these accumulative probabilities! Let's consider a quick example.</p>
<p><em>Example</em> : $a^k = AB$, $p(A/AB)=0.2$, $p(B/AB)=0.5$, $p(C/AB)=0.5$. We then store these acumulative values in the matrix:</p>
<ul>
<li>$p(A/AB)=0.2$</li>
<li>$p(B/AB)=0.7$</li>
<li>$p(C/AB)=1$</li>
</ul>
<p>That way, we only have to pick a random float between 0 and 1 using a uniform distribution to match this float with a symbol. <code>random(0,1) = 0.678 --> symbol = B</code></p>
<p>For this technique to work, the first $k=10$ symbols of the generated text must directly come from the original text (and hence will be contained in the matrix). This will give us a valid initial condition.</p>
<p>Let's now generate the text :</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">random_text</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">k</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Given a result size and an integer k,</span>
<span class="sd"> returns a randomly generated text using</span>
<span class="sd"> probability distributions of markov chains</span>
<span class="sd"> of order k dumped in ../results/distribs/distrib_kX.txt</span>
<span class="sd"> files</span>
<span class="sd"> """</span>
<span class="c1"># -- Initial string</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'../data/monty.txt'</span><span class="p">,</span><span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span>
<span class="n">initial_string</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">readlines</span><span class="p">())[:</span><span class="n">k</span><span class="p">]</span>
<span class="n">out</span> <span class="o">=</span> <span class="n">initial_string</span>
<span class="c1"># -- Import probability distribution</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'../results/distribs/distrib_k</span><span class="si">%d</span><span class="s1">.txt'</span><span class="o">%</span><span class="p">(</span><span class="n">k</span><span class="p">),</span><span class="s1">'r'</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">IOError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
<span class="nb">print</span> <span class="n">err</span>
<span class="n">exit</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="n">distrib_matrix</span> <span class="o">=</span> <span class="n">pickle</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="c1"># -- Generate text following probability distribution</span>
<span class="n">kuple</span> <span class="o">=</span> <span class="n">initial_string</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">xrange</span><span class="p">(</span><span class="n">size</span><span class="p">):</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">char</span> <span class="o">=</span> <span class="s1">''</span>
<span class="c1"># read distribution specific to k-tuple string</span>
<span class="n">dist</span> <span class="o">=</span> <span class="n">distrib_matrix</span><span class="p">[</span><span class="n">kuple</span><span class="p">]</span>
<span class="k">for</span> <span class="n">symbol</span> <span class="ow">in</span> <span class="n">dist</span><span class="p">:</span>
<span class="n">char</span> <span class="o">=</span> <span class="n">symbol</span>
<span class="n">i</span> <span class="o">=</span> <span class="n">dist</span><span class="p">[</span><span class="n">symbol</span><span class="p">]</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">></span> <span class="n">p</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">out</span> <span class="o">+=</span> <span class="n">symbol</span>
<span class="n">kuple</span> <span class="o">=</span> <span class="n">kuple</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span><span class="o">+</span><span class="n">symbol</span> <span class="c1"># update k-tuple</span>
<span class="k">return</span> <span class="n">out</span>
</code></pre></div>
<p>Done ! Now, you only have to call the function <code>random_text(len_text, 10)</code> and BOOM !</p>
<h2 id="example-of-generated-text-with-k-10">Example of generated text with $k = 10$</h2>
<div class="highlight"><pre><span></span><code><span class="ss">"KING ARTHUR: Will you ask your master that we have been charged by God with a sacred quest. If he will give us food and shelter for the week.</span>
<span class="ss">ARTHUR: Will you ask your master if he wants to join my court at Camelot?!</span>
<span class="ss">SOLDIER #1: You're using coconuts!</span>
<span class="ss">ARTHUR: Ohh.</span>
<span class="ss">BEDEVERE: Uh, but you are wounded!</span>
<span class="ss">GALAHAD: What are you doing in England?</span>
<span class="ss">FRENCH GUARDS: [whispering] Forgive me that' and 'I'm not worth"</span>
</code></pre></div>
<h2 id="what-if-we-change-k">What if we change $k$ ?</h2>
<p>k can be interpreted as the quantity of context you take into account to calculate a symbol occurence probability. We chose $k = 10$, because a context of 10 symbols allows the program to generate a text with apparent sense (limited by the randomness of the process, and by the fact that THIS IS MONTY FREAKING PYTHON).</p>
<p>The more context you add, the more alike the generated and original texts will be, up to a point where they will be identical.</p>
<p>If you decrease k, you can find a interesting case where you generate words, but where the context is senseless.</p>
<p>Example, for $k=5$:</p>
<div class="highlight"><pre><span></span><code><span class="ss">"KING ARTHUR: Yes!</span>
<span class="ss">VILLAGER #3: A bit.</span>
<span class="ss">VILLAGER #1: You saw saw saw it, did you could</span>
<span class="ss">separate, and master that!</span>
<span class="ss">ARTHUR: Will you on Thursday.</span>
<span class="ss">CUSTOMER: What do you can you think kill your every</span>
<span class="ss">good people. It's one.)</span>
<span class="ss">OTHER FRENCH GUARDS: [whispering]"</span>
</code></pre></div>
<p>If you decrease $k$ even more, you will only generate rubbish.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We have seen a pretty simple text analysis technique which allows us to randomly generate a text, based on statistical analysis of the data text. This technique is based on the fact that the probability of occurence of a letter depends on its local "past".</p>
<p>Playing with the value of the "past length", you can generate text more or less alike to the original, and with more or less "sense".</p>
<p>This simple technique does not use the nltk python module, or a set of texts to generate "theoretical" rules on a language. Its is purely empirical.</p>
<p>All source code available on <a href="https://github.com/brouberol/Generate-Monty-Pyhon-Dialog" title="GitHub repository">GitHub</a>.</p>
<p><strong>EDIT</strong> : A nice comment from reddit:</p>
<blockquote>
<p>"This approach was first proposed by Claude Shannon in his landmark paper "A Mathematical Theory of Communication"… in 1948.
Gotta love how people keep reinventing the same things over and over again. But this time, in Python!"</p>
</blockquote>